A state of sheltered and unworldly intellectual isolation.
The first mention of ivory towers is in the Bible, Song of Solomon 7:4 (King James Version):
Thy neck is as a tower of ivory; thine eyes like the fishpools in Heshbon, by the gate of Bathrabbim: thy nose is as the tower of Lebanon which looketh toward Damascus.
That biblical allusion is to the notion of ivory towers as symbols of virginal purity.
The contemporary figurative meaning is of a place of unworldly isolation. This may be in allusion to the famous Hawksmoor Towers of Oxford University's All Souls' College, which are ivory in colour (or at least, they were when they were built in 1716). The relative lateness of the first uses of the phrase (below) tend to argue against that derivation.
There are citations of the term in French in the 19th century but the earliest work in English that specifically refers to the current meaning of the phrase is a collaborative work of Frederick Rothwell and the splendidly named Cloudesley Shovell Henry Brereton -H. L. Bergson's Laughter, 1911:
"Each member [of society] must be ever attentive to his social surroundings - he must avoid shutting himself up in his own peculiar character as a philosopher in his ivory tower."