Einstein was wrong. Still, understanding Einstein's struggles is a prerequisite for grasping the profound weirdness of quantum physics. I would argue that to successfully embrace quantum theory, you have to go yourself through Einstein's struggles, and... leave them behind. Einstein never succeeded at that, but you can!
You can because you have the advantage of living in later years. Would Einstein have lived just ten more years, he would have liberated himself from the shackles that bounded him to classical physics. A stupefying new insight derived by the UK physicist John Bell would probably have given him the shock of his life, but it would have helped him taking the mental hurdle to accept quantum physics as our deepest view on reality. Einstein would probably have felt his famous physics intuition had lost contact with reality, and he would certainly happily have admitted that Feynman's claim "nobody understands quantum physics" makes no exception for him. I would love to hear the words that the most quotable physicist would have uttered at the occasion. Probably something along the lines "Magical is the Lord, magical in subtle and deceitful ways bordering on maliciousness".
Alas, all of this was not meant to be. Einstein lived till 1955. John Bell was working on his PhD in nuclear physics at that time, and not yet involved in what is now referred to as Bell's theorem. As a result, Einstein died as the last of the great classical physicists, unaware of the insights into quantum magic that would soon unfold.
So what was it that this great mind struggled with, and what insights were obtained after his dead?
Lay persons interested in quantum mysteries are invariably presented a dumbed-down version of the arguments against quantum theory put forward by Einstein. They learn about Heisenberg's uncertainty principle and the probabilistic nature of quantum measurements, and are subsequently presented with Einstein's quote "God doesn't play dice". This presents you with no more than a caricature of Einstein's position on quantum mechanics. Einstein's concerns ran much deeper and addressed the question what constitutes reality in an entangled quantum world. Einstein talked about God and dice, but is also quoted to have asked the question "Do you really believe the moon is there only when you look at it?". And that quote brings us much closer to Einstein's skepticism towards quantum theory.
Once upon a time in a small university town, a natural philosopher called Albert filled his days contemplating life, the universe and everything. Like many of his colleagues, Albert struggled each morning to equip his feet with a matching pair of socks. Would you spot Albert on a number of days, chances are you would occasionally observe his shoes filled with a red left foot and a green right foot or any other combination of colors.
Although his absent-mindedness was strong enough to serve as explanation for any ill-fitting garments, Albert did have a valid excuse for his poor choice of outfit. His Danish housekeeper, Niela Bohr, kept his socks in a chest of drawers. Three rows, each consisting of three drawers, made up this piece of furniture. Whenever Albert pulled open a drawer in search for socks to wear, he would be presented either a pair of matching socks or a single sock. Every subsequent drawer he opened, would reveal socks of a color different from those in the drawers already opened. To make things worse, each drawer opened would block from opening all drawers not in the same row and all drawers not in the same column. This effectively limited Albert each morning to the opening of three drawers configured in a horizontal row or in a vertical column.
Each night Niela prepared the chest of drawers for the next morning. To Albert's frustration, he couldn't figure out what procedure Niela followed. Each morning when opening a line of three drawers, the outcome came to him as a compete surprise. Albert labeled the drawer rows A, B and C, and the columns X, Y and Z, and started recording his observations. Each morning he wrote down a line like B121, indicating the opening of the drawers in row B containing 1, 2, and 1 socks respectively.
Following a few weeks of observations, Albert has recorded the following set of data:
C112 B222 X212 Y111
A211 Z111 Y221 B121
Y221 X122 A112 A222
B112 C211 Z212 X221
Z122 Y212 Y122 Z221
A121 C112 C121 B211
When questioning Niela about the way she filled the chest of drawers each day, she responded that she didn't fill the drawers, rather she prepared them according to the laws of quantum physics. "What do you mean you don't fill the chest of drawers?" Albert asked, "surely you fill it as I have never encountered an empty drawer." Niela hesitated. "Sir, this is a quantum chest. There is no reality associated with the contents for each drawer." Albert looked puzzled. "You mean the unopened drawers don't contain any socks?" Albert focused at her face. Was she making a joke? She seemed perfectly serious. "Sir, an observation not made is a non-existent observation. Now if sir would please excuse me, I need to wash sir's socks for tomorrow and prepare sir's chest of drawers." And off she went.
Albert thought about Niela's puzzling remarks. It all didn't make sense. He knew about this weird quantum theory. A statistical theory that he was sure, could not represent the deepest truth of nature. He knew for a fact that each time all drawers are filled. If that was not the case, surely he would on occasions have hit an empty triplet of drawers. There must be some explanation. Probably she was playing a game with him, and filling the drawers according to some secret allocation algorithm.
Months go by, the list of drawer observations kept growing, but Albert didn't manage to work out the algorithm. One day, he explains the issue to his colleague, Jim Bell. Jim was a practical guy and an expert on quantum theory. "Can I have a look at the data?", he asked. Albert handed over a sheet of paper. It took Jim only a few seconds to remark "This is interesting, a horizontal line of drawers always contains an even number of socks, while a vertical line always contains an odd number of socks". He handed back the paper to Albert, who once more inspected the data. His mouth opened. With his eyes wide open and still fixed on the paper, he uttered "But this is impossible". Jim smiled, "Well, the results are puzzling indeed. But those are your own observations. If you doubt them, you have to redo them."
Albert was still staring at the paper, and didn't look up. "This really is impossible. If at any given morning I would open three rows, I would end up with an even number of socks. But would I open three columns I would end up with an odd number of socks. Yet in both cases I would have opened the same nine drawers. This is absolutely impossible."
"Right. Albert, can I remind you that you started by telling me that chest of yours contains quantum drawers and that on each given day you can open only one row or one column of drawers at a time?"
"Yes, but let's assume, just for sake of argument, that we can open all drawers."
"Albert, you have to make up your mind. Can you, or can you not open all drawers? If not, then you should realize it is not that you don't know the facts about the contents of the drawers that can not be opened, there simply aren't any such facts."
Hours later, back at home Albert was staring at his spooky drawers. He had checked the data many times. There was no doubt, Jim's observation on even and odd sock counts was correct. Jim had tried to convince him it is meaningless to discuss the contents of drawers that can not be opened. But still, a-priori there is no drawer that can not be opened. Each morning he can decide to open any of the nine drawers, it is just that already opened drawers limit the opening of subsequent drawers. So each drawer must contain either one or two socks. Or not? This quantum stuff was really driving him crazy.
Could it be that the chest contained a hidden mechanism that played tricks on him? Maybe the socks could move from one drawer into the other based on the drawers that he opened. The next few mornings Albert checked the drawers that he pulled open and inspected them for any hidden mechanics or other tricks. Nothing of that. There was no way for the socks to move from one drawer to the other.
Could it be that Niela knows in advance if he was going to select a row or a column of drawers? No, this is a crazy thought. Precognition is pseudoscientific nonsense. But physical reality not allowing him to talk about the contents of unopened drawers seemed even crazier. So what the heck. Albert took a die and marked it with the symbols A, B, C, X, Y and Z. Henceforth, each morning he threw the die and opened the row or column of the cupboard corresponding to the symbol on the die.
Basing the choice of the drawers to be opened on the throw of a die didn't change anything to the outcomes. Rows continued to come up with even numbers of socks, and columns with odd numbers of socks. Albert looked again at the chest of drawers. What a spooky device! A spooky and revealing cupboard that was telling him something deep about the nature of physical reality. His observations on drawer contents did not leave room for any other explanation than what Jim was telling him all along: we are living in an utterly strange quantum universe. A universe in which what could have happened but didn't has no meaning.
Emergent Quantum Mechanics
What is one to make of all this? Is it possible to prepare the chest of drawers in such a way that even rows and odd columns result? Like it or not, according to quantum theory, the answer is yes. Albert's chest of drawers can in principle be constructed. And what's more: similar such devices have recently been created and operated. This is done not with drawers filled with socks, but rather with entangled microscopic systems such as photons in quantum optical experiments. The principle is all the same.
Einstein considered quantum theory as nothing more than what in our current terminology would be called an emergent theory. An approximate theory that resulted from an underlying more fundamental truth. That fundamental truth, Einstein felt, must honor the existence of physical reality independent of the measurements one executes. Einstein was convinced that things one cannot know anything about (such as the number of socks in a drawer that can not be opened) do exist all the same.
Most of Einstein's contemporaries considered Einstein's thoughts about the existence of an objective reality as philosophical musings without any practical consequences. Wolfgang Pauli was very clear on this when he wrote: “One should no more rack one's brain about the problem of whether something one cannot know anything about exists all the same, than about the ancient question of how many angels are able to sit on the point of a needle. But it seems to me that Einstein's questions are ultimately always of this kind.”
Bell's theorem published in 1964 by John Bell, and particularly its extension published a few years later by Simon Kochen and Ernst Specker, establish that Einstein and Pauli were both wrong. The question if something one can not observe does exist, is not a meaningless philosophical musing, but a question that can be answered experimentally. And when these experiments are performed, that is: when devices like Albert's chest of drawers are build and operated, quantum entanglement effects as displayed by Albert's socks come to the fore. These spooky effects force us to answer the question 'does something exist if we can not know anything about it?' with a resounding 'no'. What can not be observed does not exist. This is not a crazy philosophical thought, but a hard experimental fact.
The inevitable conclusion is that if there is a more fundamental truth from which the known laws of quantum physics are emergent, this more fundamental truth must be at least as weird as quantum theory. More in particular, a classical physics theory capable of explaining all of quantum physics - Einstein's hope - can not exist. Any wish for a classical foundation of physics is death, and so is the last great physicist who believed in it. Each scientist is the product of a generation. Einstein's thoughts ran deep, but even he could not stay ahead of developments when a next generation of physicists started to unravel the spookiness of our quantum universe.
Follow-up post: Letting Go Of The Reality Ether.