Fences don't appear out of nowhere. It often takes days to install. Where were you when it was being installed? Why didn't you put a stop to it on the spot? Or tear it down as soon as you saw it?
If you knew it was being installed and didn't object that's like giving permission.
Did you ever put your neighbor on written notice that he has to remove it by a certain deadline or you would have it removed and sue him for the cost?
You can still do that, you know.
But if you keep on doing what you are doing, it'll be meaningless and the fence will still be there long after you are gone.
By the way, how far on to your property is it? One foot? Two feet? Is it a nice looking, well constructed fence that might actually be enhancing your property for which you get that advantage for free? You might think about that.