Colin Clark took the seat with 21,861 votes compared with Mr Salmond's 19,254.
The SNP have won the majority of the 59 Scottish constituencies.
Following the declaration, Mr Salmond said serving the local area had been the "privilege of my life". He added: "You've not seen the last of my bonnets and me."
The SNP's foreign affairs spokesman had swept to power in the Gordon seat with a 8,687 majority in 2015, overturning decades of Liberal Democrat rule.
In his victory speech, the newly-elected Tory MP Mr Clark said: "The silent majority have spoken. We are proud to be part of the United Kingdom."
Earlier, Douglas Ross of the Tories overturned a majority of 9,065 to defeat the SNP's deputy leader Angus Robertson by 4,200 votes in Moray.
In a short speech after the Gordon declaration, Mr Salmond said: "The SNP has lost a number of fine parliamentarians this evening, that is a grievous blow for us.
"But overall results show the SNP will win a majority of the seats in this country and a majority of the votes."
He said that while the SNP may have been reduced in numbers at Westminster, they will still have significant influence.
Mr Salmond bounced back to a prominent role in the SNP as foreign affairs spokesman after stepping down as first minister and party leader after defeat in the Scottish independence referendum in 2014.
He joined the party in 1973 while at St Andrews University and was elected MP for Banff and Buchan in 1987, gaining prominence when he was suspended for a week for disrupting the chancellor's budget speech the following year.
He later said the incident - in which business in the Commons was suspended while Mr Salmond was expelled from the chamber - took him from "total nonentity to notoriety".
He took over the leadership of the SNP for the first time in 1990 and remained at the helm in 1997 when the SNP increased its tally of MPs to six and played a key part in the successful campaign for a new Scottish Parliament.
The first Holyrood election took place in 1999, with Mr Salmond becoming MSP for Banff and Buchan.
He stood down as SNP leader in 2000 and left the Scottish Parliament the following year.
Despite refuting speculation he would return to the top job, he said: "If nominated, I'll decline; if drafted, I'll defer; and if elected, I'll resign."
He then stood on a joint ticket with Nicola Sturgeon marking the start of a decade-long partnership.
In 2007, he returned to Holyrood as MSP for Gordon and the new first minister, a position he held for seven years, making him the longest serving first minister.
Four years later, his party won an unprecedented overall majority at Holyrood and secured a referendum on Scottish independence.
Former SNP leader Alex Salmond has lost his Gordon constituency to the Scottish Conservatives.