Genesis 4:3 And in process of time it came to pass, that Cain brought of the fruit of the ground an offering unto the LORD. 4 And Abel, he also brought of the firstlings of his flock and of the fat thereof. And the LORD had respect unto Abel and to his offering: 5 But unto Cain and to his offering he had not respect. And Cain was very wroth, and his countenance fell.
Some think Cain should have made a blood sacrifice, but it doesn’t read like that to me.
God says to him “If thou doest well, shalt thou not be accepted?”
Genesis 4:7 If thou doest well, shalt thou not be accepted? and if thou doest not well, sin lieth at the door. And unto thee shall be his desire, and thou shalt rule over him.
This indicates to me that the problem wasn’t the offering but rather Cain’s behaviour and attitude of mind.
The writer to the Hebrews says:
Hebrews 11:4 By faith Abel offered unto God a more excellent sacrifice than Cain, by which he obtained witness that he was righteous, God testifying of his gifts: and by it he being dead yet speaketh.
And John writes
1 John 3:12 Not as Cain, who was of that wicked one, and slew his brother. And wherefore slew he him? Because his own works were evil, and his brother’s righteous.
From this we understand that Abel had faith, whereas Cain did not; and consequently, Cain’s works were evil.
On this basis, no kind of sacrifice would have been acceptable because Cain’s attitude to God was wrong.
Hence "if thou doest well" - he needed to change his behaviour.
To offer a sacrifice for ‘outward show’ only is hypocrisy.
I hope you find this helpful.