The company has been running services from Waterloo to the south coast, Salisbury and Reading since 1996.
Its contract expires in 2017 but it hoped to secure a further two years.
The Department of Transport confirmed it was unable to reach agreement with the company and said the franchise would go out to competition.
In a statement, Stagecoach said: "A significant difference has remained between both parties regarding the financial evaluation of the proposals.
"Nevertheless, as the incumbent operator with nearly 20 years' experience in growing and improving one of the most complex and busiest rail franchises in the country, we believe we are in a strong position to submit a powerful and attractive bid for a new South West Trains franchise."
Secretary of State for Transport Patrick McLoughlin said he believed a franchise competition would achieve passenger benefits and better value for taxpayers.
Basingstoke MP Maria Miller welcomed the announcement, saying it could mean a "more modern structure to cope with what has been a huge increase in passenger numbers".
She added: "In our region we've seen the effect of that increase in numbers come out in congestion, which means that some peak time trains can frankly feel like cattle trucks at the best of times."
But the RMT's general secretary Mick Cash called the decision a "shocking indictment of government policy and the privatised railway that a company that has been running the franchise for 20 years is not entrusted with the service going forward".
Train operator Stagecoach has failed to reach an agreement with the government over its renewal of the South West Trains rail franchise.