Others have answered from the point of view of the connection being identified by the IP+Port pair. Let me talk a little about how it works in the layered structure.
TCP is a layer that provides the 'illusion' of a reliable stream on top of an unreliable packet layer (IP). For that, it has to keep account of several variables to manage the stream, and also have to provide the relevant parameters to the layer below.
So, when you ask TCP to open an stream, you give it the IP+Port of the destination. It keeps that IP number, and each time it has to transfer something, assembles an IP packet and tells the IP layer to send it to the intended machine, identified only by the original IP number.
When you changed the IP number of one machine, the other one's TCP layer doesn't have any mean to know what happened. It only sees that any IP packet sent to the original IP number is no longer answered (maybe it gets an ICMP message telling that there's no machine with that IP number). Also, it doesn't get any more packets with that IP number. Obviously the connection will be dropped after some timeout.
Even worse, it might start to get some unrelated packets from a different origin (the new IP number), but those assume a connection is already in place! Of course, the only answer that machine might get (if at all) is a RST packet to make it immediately cease and desist.