Sure, people can develop a fascination for the most bizarre things - knitting, collecting Hard-Rock-Cafe shirts, or photoshopping their wifes into shape - and leave others puzzled about their obsession. But when it comes to mathematics, indifference and puzzlement is replaced with plain rejection. There's those for who mathematics is the essence of everything, it's the language in which the book of Nature is written and the secrets of the universe are encoded in. And then there's those who believe that the lover of mathematics is narrow-minded, and that the world is so much more, so vastly more complex than what mathematics can possibly capture, that anybody who thinks incomprehensible, abstract symbols capture elementary truths must be seriously disturbed.
And sure, people can argue furiously about politics or who has the best pizza in town, but in no case I can think of do you find a comparable utter lack of understanding for the other side than when it comes to the power of mathematics. The lack of understanding is probably so complete, one can't even argue about it. Over and over I have found people who reject the notion of mathematics being a universal language, and who discard it as insufficient for reality. They are dead wrong to do so of course, but since I've encountered this attitude over and over again, I want to dedicate some paragraphs to what I believe is the origin of this divide.
At the very beginning is, of course, school education. Unfortunately, what's called mathematics in school has little to do with mathematics. It should more aptly be called calculation. Don't get me wrong, it is essential knowledge to be able to multiply fractions and calculate percentage rates, but it has about as much to do with mathematics as spreading your arms has with being a pilot. Problem is, that's about all most people ever get to know of mathematics. The actual heart of math however is not number crunching or solving quadratic equations, it's the abstraction, the development of an entirely self-referential, logically consistent language, detached from the burden of reality.
Let me focus on an example that those of you with high school education will have met: vector spaces. A vector space is basically a set with elements that have a structure allowing for an operation called addition and a second operation that's multiplication with a scalar. These operations have to fulfill certain criteria which you can look up somewhere if you've forgotten, but it's not so relevant for the following. What's relevant is how abstract this notion of a vector space already is. The vector space really is that definition, and nothing else. And it's taught in school! Of course, at the time pupils come across a definition for a vector space most know examples already and have a mental picture. My math teacher used pens to visualize vectors. But nothing in the definition of a vectorspace tells you it ought to be three-dimensional, or the elements be coordinate-vectors (pens).
Given that vector spaces are such a simple concept that is introduced even in school, I was surprised to learn how late in the history of science it came along. The phase space in physics is essentially a higher dimensional vectorspace whose elements aren't only coordinate vectors but also momenta (and, by virtue of this, has some additional "symplectic" structure). Knowing what you know today this sounds hardly like a revolutionary concept. But in the middle of the 19th century it was. In his (highly readable) Physics Today article on the history of the Phase-space, David Nolte writes:
Today it is natural for us to assign each variable its own axis in a generalized multidimensional space. But in the 1700s it was not natural. [...] Cayley in his 1843 paper titled "Chapters in the Analytical Geometry of (n) Dimensions" was the first to take the bold step of referring to a geometry of more than 3 dimensions. After that, the stage was set for the "invention" of multiple dimensions when Grassmann developed the concept of an n-dimensional vectorspace in 1844.
The vector space that you've heard of in school is a result of many generations of abstractions. Yet, it is still an extremely special case of what the mathematician considers a "space."
If you go out on the street and ask random passers-by what they associate with "space," you might hear office space, or the space they don't have in their living room after they bought the drum set for the eldest son. You might hear parking space, or the space between two letters in a sentence, or maybe outer space. That's the real world, and at first sight it seems indeed like a selection of complex and very different notions of space. But in fact all these spaces that you encounter in the real world are highly specific. They are three or lower-dimensional. They are to excellent precision flat. They come with a distance measure that allows you to tell if the new couch will fit next to the drum set. The general space in mathematics however may do away with all of these properties that we are so used to. Imagine an infinite dimensional space. Imagine one without distances. Imagine what it would be to try to park your car in one.
In physics one does encounter more general spaces than the standard 3-dimensional vector space. The best known example is probably the Hilbertspace of quantum mechanics, which can easily be infinite dimensional. But also in physics, the realm that we deal with is only a tiny part of all that mathematics has to offer. The functions we deal with are typically nicely differentiable, so are the manifolds we put them on, blessing us with plenty of additional structure. The differential equations we have are typically not higher than 2nd order, spaces are hausdorffian and almost all of the pathological examples you come across in mathematics the physicist never has to bother with.
This of course then brings one to the question, if the world of mathematics contains so much more, then where is it? Does it exist, somewhere, that space without distances, that module, that left-invariant subgroup? I have some sympathy for Tegmark's Mathematical Universe, which posits that all of mathematics must exist somehow, somewhere in the multiverse. My central objection to Tegmark's idea just is that it's not insightful and plain useless.
If you've scrolled down to this last paragraph, shame on you. The point of all the words above was that during the history of science we have come to realize it is the world of mathematics that is vastly larger than what the real world has to offer, not the other way round.