Generally no. If a unit is used once, anywhere in the project, it doesn't matter how many more times it's used. Conversely, it doesn't matter how many places you remove a unit from if it's still used at least once somewhere. The compiled program will behave the same, and it will have roughly the same size.
The only difference would be in the order of unit initialization and finalization sections. Unit-usage order affects the order those sections are executed in, although the precise effect has never been documented (so try not to rely on initialization order).
But I still encourage you to clean up your unit lists, for the same reason you're encouraged to clean up your variable lists and your parameter lists. When you get rid of the stuff you don't need, it makes it easier to read the code you've kept because you can be reasonably confident that what you're reading gives an accurate picture of what the code does. If your code mentions a bunch of units but never really makes use of them, then the next time you or someone else looks at the code, there's a good change you're going to spend some time trying to find where your code uses the facilities of those units. (You'll say to yourself, "Hmm, this code includes
Graphics, but I can't see where it draws anything. I'd better take another look, because I didn't think this code had any responsibilities like that. Hey, co-worker — can you take some time out of your day to tell me where this unit draws things?")