Didn't both of them try their best?
Genesis 4:3 And in process of time it came to pass, that Cain brought of the fruit of the ground an offering unto the LORD. 4 And Abel, he also brought of the firstlings of his flock and of the fat thereof. And the LORD had respect unto Abel and to his offering: 5 But unto Cain and to his offering he had not respect. And Cain was very wroth, and his countenance fell. 6 And the LORD said unto Cain, Why art thou wroth? and why is thy countenance fallen? 7 If thou doest well, shalt thou not be accepted? and if thou doest not well, sin lieth at the door. And unto thee shall be his desire, and thou shalt rule over him.
‘Doest well’ – Hebrew ‘yatab’ – meaning: to do good or to do right
From what God says to Cain we understand that God wouldn’t accept Cain’s offering because he wasn’t doing that which was right.
He was not living according to God’s precepts.
The writer to the Hebrews explains that Cain lacked ‘faith’.
Hebrews 11:4 By faith Abel offered unto God a more excellent sacrifice than Cain, by which he obtained witness that he was righteous, God testifying of his gifts: and by it he being dead yet speaketh.
This means that he was intent on serving himself rather than God.
John tells us that Cain’s works were evil in contrast to Abel’s.
1 John 3:12 Not as Cain, who was of that wicked one, and slew his brother. And wherefore slew he him? Because his own works were evil, and his brother’s righteous.
God tells Cain that if he did that which was right, God would accept his offering.
From this we see that faith and repentance to seek God’s favour were necessary right from the beginning.
I hope you find this helpful.