Correct Stk! The estate goes though the males...Matthew's son would inherit regardless of who the mother was, a Crawley female was irrelevant.
Matthew's son may inherit the estate; it depends on how Reggie Swire's money was invested into it. (Was the money added to the entail? Or did Matthew buy the estate from Robert?)
But Patrick is still a closer heir to the Earldom than Matthew's son because Matthew's patrilineal ancestor is further back than Patrick's.
The laws of inheritance were reformed in, I think, 1923, which may make all of this moot.