Ishmael was Abraham's son too. Hagar was his wife, wasn't she?
Thanks for your question.
Hagar and Ismael were cast out because Hagar, when she became pregnant, despised her mistress. Hagar was a "secondary wife" and was still Sarah's handmaid. It was the custom of the time for the primary wife, if she were barren, to give her handmaiden to her husband to wife so that she could bear children as a kind of surrogate. Hagar made the mistake of flaunting her fruitfulness and looking in contempt upon her mistress thus earning her mistress's deep displeasure. Also, when Isaac was born, Ishmael, then a teenager, mocked his younger brother - the word "mocking" as it is translated in the KJV is vague and perhaps it was something more abusive that Ishamael was engaged in. In any case, it was the last straw for Sarah and she had Hagar and Ishmael exiled from the family.
God chose Isaac because He had decreed that the promised seed would come through Abraham and Sarah and not Hagar. See Genesis 17:21; 21:12; Romans 9:7; Hebrews 11:18. Paul uses the metaphor of the son of the freewoman, Isaac as a type of Christ and his subsequent spiritual offspring while the son of the bondwoman is a type of those lost to the bondage of sin. See Galatians 4:23-31
I hope you have found this helpful.