The Amazon show is about an alternate world where Nazi Germany and Japan rule the US after winning World War ll.
The ads showed an American flag with a German eagle and cross in place of the stars and an Imperial Japanese flag.
New York Governor Andrew Cuomo intervened to get the advertising removed.
The city's Mayor Bill de Blasio had also called on Amazon to remove the advertising, which he called "irresponsible and offensive to World War Two and Holocaust survivors, their families, and countless other New Yorkers".
The advertising was wrapped around the seats, walls and ceilings of one train on the shuttle line that connects Times Square and Grand Central Terminal.
The Anti-Defamation League criticised the advertising for its lack of context.
"On the television program, which explains this is the notion of an America controlled by Hitler, you get that context. On the train, seeing the American flag paired with a Nazi symbol is viscerally offensive because there is no context as to what it means. The fact that the flag is spread across the seats only compounds the effect."
Amazon released a statement about the programme that did not refer to the advertising specifically, but said the show was "high-quality, provocative programming that spurs conversation".
The advertising was taken down after Governor Andrew Cuomo called the head of the Metropolitan Transportation Authority to ask him to ensure it was removed.
Posters for the TV show, which is based on the Philip K Dick novel, are still in the subway.
The show's creator and executive producer Frank Spotnitz told Entertainment Weekly that he understood why people were upset by the advertising but said Amazon had a tough job.
"It's very difficult with a show with subject matter like this to market it tastefully, so I understand they're walking a very difficult line."
"If they had asked me, I would have strongly advised them not to do it...
"Within the show, there is a context where you see why [they're used], but just to put them out like that without the context was unfortunate."
Nazi-inspired advertising for the TV show The Man in the High Castle has been removed from the subway in New York after complaints.