Let me first answer your questions and then offer a comprehensive review of the experiment above.
Answers to your questions
There is a principle I may assure you is true: none of these experiments, and no other experiments, may send an actual information faster than light (and surely not backwards in time). "Faster than light" and "backwards in time" are really related; the vectors that go backwards in time but superluminally are equivalent, via the Lorentz transformations, to vectors that are "faster than light" but directed into the future.
So why does your idea technically fail to send signals faster than light? There are two aspects of the experiment that you may be neglecting, a general principle and a technicality which is not really a technicality because it's essential for the explanation why your superluminal device won't work:
If you're only observing what's measured in the upper part of the picture, e.g. the photographic plate D0, then everything that happens in the lower part of the picture is completely inconsequential for the measurements done in the upper part of the picture. So whether you replace the system of mirrors and detectors by another photographic plate E0 is totally irrelevant for the results in D0 only.
A related point is that there is no interference pattern of the signal photons observed in D0 itself, if you don't use any information about their partnered idler photons from the lower part of the picture. The interference pattern only occurs for subsets of the photons hitting D0, and you may only "see" such pattern(s) a posteriori, once you get the information about what was detected in the detectors such as D1,D2.
It's surely not the case that decisions of the experimenters in the lower part of the picture change the odds of various things happening in the upper part of the experiment. Even the replacement of the half-device by a totally new one makes no difference. That doesn't mean that there's no correlation (entanglement): correlation is there but correlation doesn't mean causation.
Once again, the critical point you seem to be neglecting is that even with the very picture you included above, the interference pattern in D0 only appears if you isolate some signal photons that were hitting D0, namely those whose partner idler photons were detected in D1. If you only isolate the signal (upper) photons partnered with idler (lower) photons detected in D1, you get an interference pattern. If you only isolate the signal (upper) photons partnered with idler (lower) photons detected in D2, you get an interference pattern, too (but a different one: the negated one). The union of these two sets neutralizes the interference pattern.
The signal photon partners of idler photons detected in D3 or D4 (or both, three possibilities) don't create any interference pattern at all.
A new review of the experiment
The fast answer above assumed some familiarity with the experiment. Here I describe what the experiment above actually does, why it's counterintuitive, and why it's completely rational and in agreement with the principles of physics such as locality and causality, too.
The diagram looks very complicated but it's not. I ask you for a little bit of patience:
In the left upper corner, there's a laser. The light instantly gets to a double slit; the two slits are called "a" (red) and "b" (light blue) on the picture. This part of the experiment is trivial. Everything that is convoluted happens after the light goes through the double slit.
The photon takes slit "a" or slit "b" (or "both" in a way that will be discussed). In the picture, it means that the photon is either red or light blue.
You see that the red or light blue photons immediately split into pairs. That's achieved by a "beta barium borate (BBO) crystal" which performs a "spontaneous parametric down conversion". If a photon goes through the BBO crystal, it splits into two entangled photons with 50% of the energy (frequency) each.
So each photon that has made it through the double slit - whether it's red ("a") or light blue ("b") or undetermined - splits into two "lighter", mutually entangled photons. One of them goes to the upper portion of the picture, one of them goes down. The photon going up is called the "signal photon" and it may draw an interference pattern on the photographic plate "D0". Or not.
The photon that went down is called the "idler" because it's interesting to force this photon to wait for a while; don't forget that the idler is entangled with the signal photon. The idler is going to be manipulated by further components of the apparatus that may either preserve of destroy the interference pattern that may be drawn by the signal photon.
There are some grey-green mirrors that only reflect photons and they don't change anything about the information carried by the photons. However, there are also light green beam splitters ("BS") that have a 50% probability to transmit and 50% probability to reflect an incoming photon.
The geometry - including the ordinary geometric prism (incorrectly labeled "Glen-Thomson prism on the picture"; the GT prism is a part of the BBO) - is such that if the photon goes to "BS_b", it had to arrive from the red "a" slit, and when it goes to "BS_a", it had to arrive from the light blue "b" slit. Not sure why they crossed the convention for "a" and "b" for the beam splitters on the picture but maybe the slits' "a" and "b" labels have already been erased. ;-)
One-half of the idler photons from "BS_a" and "BS_b" are sent to detectors "D3" and "D4", respectively. If you detect an idler in "D3" or "D4", it proves that it had to go through slit "b" or "a", respectively. Because the signal photon is entangled and Nature never forgets any correlations - one of the general rules - it follows that the upper, "signal photon" had to go through "b" or "a", too.
Such photons whose path is known couldn't have interfered. And indeed, the signal photons corresponding to idlers seen in "D3" or "D4" are found not to contribute to an interference pattern. If you reconstruct the sub-picture made by the signal photons that were linked to the "D3" and "D4" idlers, you will see no interference pattern at all.
However, one-half of the photons going through either "BS_a" or "BS_b" are manipulated in such a way that the "which slit" information will never be detected. We say that this "which slit" information is "erased" and this part of the gadget is the "quantum eraser".
How does it work? You see that the parts of the photons that went through the "BS_a" or "BS_b" splitters but didn't end in "D3" or "D4" continue, through mirrors "M_a" or "M_b", to another beam splitter "BS_c" where the beams from both mirrors "M_a" and "M_b" are re-united into a single beam. Well, more precisely, they're reunified into two beams going to "D1" or "D2" but "D1" or "D2" is not correlated with the "a", "b" slits in any way. So the information about the slit is no longer remembered by the position of the photon.
The beam splitter "BS_c" is designed in such a way that it sends the unified beam either to "D1" or "D2", whether it comes from the mirror "M_a" or "M_b".
Because the "which slit" information was erased by the eraser - by the re-unification of the two beams - the idler photons that ultimately end in "D1" or "D2" were allowed to interfere. Because they are entangled with some signal photons in the upper part of the picture, it follows that the corresponding signal photons could interfere as well.
And indeed, if you reconstruct the positions of the signal photons whose partner idlers were found in "D1" - which could occur much much later - you will find out that the positions of these signal photons did draw a sharp interference pattern. Also, if you reconstruct the positions of the signal photons whose partner idler photons were found in "D2", you also find a sharp interference pattern - but a different one than for "D1".
These two patterns are actually negations of one another - minima for "D1"-partnered signal photons overlap with the maxima for "D2"-partnered signal photons and vice versa. So if you add all the signal photons whose partners appeared either in "D1" or "D2", the interference pattern disappears. Again, recall that the signal photons associated with the idler photons detected in "D3" or "D4" didn't contribute to the interference pattern, either. So if you simply detect all signal photons and make no distinction between them, there will be no interference pattern on the screen.
Why do people find it counter-intuitive?
And why do they think that the experiment reveals retrocausality (i.e. the ability to influence the past of the Universe)?
Again, it is because these folks violate one of the rules 1,2,3,4 (see an appendix below) - or several of them. Most typically, they think that we are "retroactively forcing" the signal photons to contribute - or not to contribute - to one of the interference patterns or another by doing something to their partnered (and entangled) idler photons.
However, as the rule 2 emphasizes, it's not "us" who is deciding about the fate of the idler photons. It's the random generator of Nature itself. We have no credentials to influence it. Moreover, the manipulations with the idlers don't influence the predictions for the signal photons at all, as I will explain below where I confirm rule 4.
The logical reasoning above that was needed to determine which signal photons contributed to one of the two mutually negated interference patterns (partners of "D1" or "D2" idlers, respectively) and which signal photons didn't ("D3" and "D4") is respected by the laws of quantum mechanics, especially because of the rule 3 that Nature never forgets any correlations. You shouldn't forget them, either.
Nevertheless, as the rule 4 makes clear, correlation doesn't imply causation. So the later manipulations with the idler photons didn't "force" their partnered signal photons to behave in one way or another. In fact, it is always more sensible to imagine that the causal relationship goes in the other way. But even this is not needed.
Because some signal photons ended e.g. in the near vicinity of the interference minima of one of the interference patterns - which is the case that makes us maximally certain that these photons didn't contribute to the interference patterns - it followed that their idler partners couldn't have been seen in "D1" or "D2", respectively (depending on which pattern's minima we're talking about) which is appropriate for idler partners of non-interfering signal photons or the idler partners of the signal photons that had a maximum at the point.
Typical individual photons can't be sharply said to be "a part of one of the interference patterns" or "not a part of any interference pattern" before their idler twins are detected. As rule 2 tells you, the quantum predictions are probabilistic, and unless the predicted probabilities are 0% or 100%, both answers - Yes, No - may be true.
However, the 0% or 100% probabilities are exactly what we get once we know where the idler photons were detected. Once we know it, it is possible to reconstruct whether the corresponding signal photons did behave as the parts of one of the two interference patterns or parts of the non-interfering boring blob.
A brief chronological history of a photon pair
Let us now look at a photon - that becomes a pair of photons - chronologically, to see that no "retrocausality" (modification of our own history) or "faster than light signals" ever appear in such experiments if they are interpreted properly, according to the postulates of quantum mechanics.
A high-energy photon emerges from the laser and goes through the slit. We know that we should associate it with a wave function that has the potential to interfere - unless we're going to deal with the photon in a way that destroys the interference pattern.
The high-energy photon gets through the double slit and gets transformed into a pair of photons. The most general wave function of the high-energy photon is a complex superposition of the states "a" and "b".
The BBO crystal transforms the state "a" into a specific entangled state of two low-energy photons (signal photon and idler) near the "a" slit. In the same way, it transforms "b" into a specific entangled pair of low-energy photons (signal photon and idler) near the "b" slit. One of the two photons in the pair is x-polarized while the other is y-polarized, and Glen-Thomson prism is used to send them in different directions (signal goes up, idler goes down). (The diagram and Wikipedia incorrectly use the term "Glen-Thomson prism" for another prism to be discussed momentarily.)
Linearity of quantum mechanics also tells you what is the state of the two low-energy photons (signal photon and idler) that you get from a general linear superposition of "a" and "b": it's the linear superposition of the corresponding two-photon states, with the same complex coefficients.
Now, we know that the signal photon is going up, attempting to produce something like an interference pattern - or not. We want to measure its exact position on the photographic plate "D0". This signal photon is entangled with the idler that goes down to the prisms, beam-splitters, mirrors, and binary detectors.
Imagine that the signal photon reaches the photographic plate "D0" (well) before the idler gets to the prism. How does the signal photon decide whether it should help to paint an interference pattern or not? It's not yet known whether its partner's "which slit" information will be measured or not, is it? Well, it doesn't matter at all. It's always clear which probabilistic distribution should be used for the first particle. What is the rule?
The wave function is defined as a function of all the measurable variables (a maximum set of commuting observables) of both photons. It is
psi(signal photon properties, idler properties)
For each combination of signal photon properties - we will care about its position - and the idler's properties - we will care about the qubits that ultimate determine (probabilistically) which detector out of "D1,D2,D3,D4" it can take - there is a complex number.
So what is the probability that the signal photon is detected at a particular spot? Let's imagine that all the variables are discrete. (If they're continuous, and the signal photon location should be continous, the sum should be replaced by an integral and the probabilities are really probability densities.)
The probability that the signal photon chooses location L is given by
Prob(signal photon location is L) = Sum over basis vectors "i" of the idler's Hilbert space of |$\psi$(signal photon at L, idler has property "i")|^2.
This formula shouldn't be shocking if you have ever calculated any probability in quantum mechanics. The probability of a micro-outcome in quantum mechanics is given by the squared absolute value of the probability amplitude.
But because we are (and Nature is) only interested in a property of the signal photon right now (She has to urgently decide where the signal photon appears), we must sum the probabilities over all independent vectors of the Hilbert space of all other degrees of freedom - in this case, all basis vectors of the idler photon's Hilbert space.
Now, it is important to realize that it doesn't matter which (orthonormal) basis of the idler photon's Hilbert space we choose. The probability of location L of the signal photon described above is completely independent of the choice of the basis for the idler's Hilbert space! That's a simple consequence of the idler's basis' being orthonormal.
The probability that the signal photon's location is equal to L doesn't depend on any idler photon's degrees of freedom. So the formula above implies that it is a kind of "squared length" of a vector in the idler's Hilbert space - a vector that depends on L - and such a "squared length" is independent of the choice of the basis.
Why am I saying that? For a simple reason. James Gallagher who, together with Mephisto, has provoked me to write this text a year before it appeared on Stack Exchange, posted the following comment one year and 20 seconds ago:
[I] look forward to it [this blog entry], but you should say whether an inteference pattern could be retrieved (by a signal/idler match) even if the eraser is removed before the idler photons hit it.
(I agree orthodox QM predicts the correct result, but I'm puzzled as to why you think removing the eraser before the idlers hit them is not interesting or puzzling to contemplate)
He wanted me to say "whether an interference pattern could be retrieved even if the eraser is removed before the idler photons hit it".
The formula for the "Probability (signal photon location is L)" written above clearly and simply answers all such questions. The answer is, of course, that the pattern painted by the set of all the signal photons is totally independent of anything we later do with the idlers.
This is a simple consequence of the fact that the wave function for both photons has some values before we do - or we don't do - anything with the idlers. And the probabilities for the signal photon are completely independent of our choice of the idler's basis, because of the simple unitarity rule. And we're summing the probability over all possible idler's outcomes.
In practice - and in this experiment, the interference pattern never exists if you just collect all the signal photons - because they're entangled with the idler photons that we "trace over" which destroys the interference. However, in more general experiments, a weak interference pattern could be predicted even with all the photons.
Obviously, whatever we do with the idlers (and the whole "eraser") has absolutely no impact on the picture that is created by all the signal photons. All the signal photons will create a "diluted" interference pattern in general - which can be interpreted as a juxtaposition of an interference pattern and a non-interference pattern. In this particular experiment, there is no interference pattern visible at all if you just collect all the signal photons.
More generally, the relative representation of the interference pattern vs non-interference blob in the mixture is only determined by the properties of the experiment "before the eraser" - e.g. only by the laser, double slit, and BBO - and in the particular example discussed above, the relative weight of the interference and non-interference patterns in the mixture - in the "diluted" interference pattern - is 50%:50%. However, the interfering "D1,D2" signal photons induce opposite interference pictures that cancel if you just add them up - so that the picture drawn by all the signal photons shows no interference at all.
Obviously, without the eraser - or without its measurement of the idlers (whether they are of the "D1/D2" type or "D3/D4" type), we can't exactly say which particular signal photons were contributing to one of the interference pictures. We can only calculate the probability for each signal photon that it was a participant in the "D1" or "D2" interference pattern project. Only for the signal photons that end up "exactly" in the interference minima, we can be sure that they couldn't have been paired with "D1"-idlers, or that they couldn't have been paired with the "D2"-idlers, because the interference pattern has a vanishing probability for them to land in the minima. Only when the probabilities are 0% or 100%, we can be certain.
James and others clearly believe that there has to be some "retrocausality" in the predictions of quantum mechanics but there is absolutely no retrocausality. If you decide to ignore any conceivable (future) measurements done with the idler photons, and you only look at the pattern created by the signal photons, this pattern (or any measurements of the signal photons) will be completely independent of any torture that the idler photons may be exposed to! In this particular experiment, there will be no trace of interference on the photographic plate.
This is a very important point but it is not specific to this experiment, either. It could have already appeared in the ordinary EPR entanglement experiment, too. If you only measure the left-moving photon, it will be left-handed and right-handed in 50%:50% of cases, and if you measure its linear polarization, it will also be 50%:50% x-polarized or y-polarized, respectively. Whatever dramatic or complicated torture the other guy decides to do to the right-moving photon has absolutely no impact on the separately measured properties of the left-moving photon. The "other experimenter" has no tool to influence what you measure, not even statistically. He's just correlated with you, the "first experimenter", but his correlation with you doesn't mean his control over your lab.
However, if you measure both photons, you will also find the correlations. In the EPR case, the correlation encoded in the two-photon state will guarantee that the two photons either have the same circular polarization or the opposite linear polarizations - in the cases (in the pairs of measurements) when the same "category" of polarization is measured with both photons.
In the complicated case of the eraser, it's analogous. What will happen in our chronological story of the wave function after we have measured the location of the signal photon? It's simple and you must surely know what's the answer by now.
We have already measured the location of the signal photon to be "L". As rule 3 says, Nature never forgets about any correlations. So all the probabilities for states that contradict the assumption that the signal photon landed at location "L" - which is already a historical fact at this moment - become irrelevant. If we want to predict the fate of the idler photon, we may simply imagine that the wave function has "collapsed" to
$\psi$[AFTER] (signal photon location K, idler basis vector "h") = $\psi$[BEFORE] (signal location L, idler vector "h") $\times \delta(K,L)$
I have simply multiplied the wave function by the Kronecker delta that is only non-vanishing for locations "K" that are consistent with - e.g. (in this case) equal to - the measured location "L". (The new wave function should be normalized so that its norm is 1 again, but I hope you understand similar trivial points.) The rest of the wave function becomes irrelevant for any further predictions because Nature doesn't forget about any correlations and we already know the location of the signal photon.
So after you have measured the signal photon's location, you have a simple wave function that effectively depends on the degrees of freedom "h" of the idler photon only. The whole system is now again reduced to quantum mechanics of a single photon. Whatever you do with the erasers is just choosing a basis of the Hilbert space of the idler photon. A single-particle quantum mechanics predicts the probabilities of different outcomes for the idler photon in the simplest way that we know for one-particle quantum mechanics. The signal photon that has already been fully measured becomes totally inconsequential for the predictions of the idler photon's fate.
It's useful to appreciate the rule 4 - which means that the reduction of the wave function is not real. And it doesn't matter at all when and in which order we "reduce" the wave function. The measurements of the two photons may be spatially separated - like in the EPR experiment - but no real signal has to propagate in between the two photons so there is no violation of relativity whatsoever.
There is one extra subtlety that leads people to incorrectly think that there is some retrocausality in the experiment: the measurements of the idler photons are discrete (D1/D2/D3/D4) while the measurement of the signal photon is continuous (location on the would-be interference pattern). People often think that discrete data can "cause" or "select" continuous pictures and interference patterns, but not the other way around. However, that's a wrong assumption: there is absolutely nothing wrong about "collapsing" the wave function of the signal photon and determining the likelihoods of its idler partner's discrete results out of the reduced wave function.
Quantum mechanics simply predicts the probability for any combination of properties of the two (or more) photons (or any other objects) we decide to measure. A psychological issue could be that some people find it harder to "imagine" a probabilistic distribution for many variables instead of just one variable but if you don't find it hard, you should have no problem with quantum mechanics in these experiments.
There is absolutely no retrocausality or superluminality of the influences in similar experimental setups. Quantum mechanics makes predictions that make complete sense, that are fully self-consistent, causal, local (in the case of quantum field theory) and that agree and will always agree with any experiment with as many slits, prisms, mirrors, beam splitters, photographic plates, detectors, and counters as many you want.
Appendix: rules 1,2,3,4
Key principles of quantum mechanics
People make several basic mistakes before they incorrectly determine that there is something paradoxical about quantum mechanics. Most of the mistakes arise because people fail to appreciate some (or all) of the following four fundamental and universal points about quantum mechanics:
Quantum mechanics only predicts the final results of experiments: it is not possible to say what the "real properties of the system" were prior to the measurement; and it's unsurprising that everything that occurs before a measurement may influence its outcome
All the predictions of quantum mechanics for the outcomes of the measurements are probabilistic and we can't influence the random generator: there can't be any hidden variables that decide about the exact fate of individual particles, not even in principle, and whenever some phenomena are random in the quantum sense, the data is genuinely random and produced by Nature at the given moment; we should never think that it was "us" who made the decision
Nature never forgets about any correlations: if the equations of quantum mechanics predict two objects "A, B" or their properties to be correlated - i.e. the probabilities of combinations of outcomes "P(A1, B3)" can't be universally written as "P(A1) P(B3)", then Nature never forgets about this correlation, not even after a long time (or at a very different place); so Nature's random generator that decides about the outcome of the measurement of "B" uses the conditional probabilities in which all quantities that have already been measured - e.g. "A1" - are already assumed; after "A" is measured to be "A1", the relevant probabilities for outcomes in "B" are the conditional probabilities assuming "A1"; this prescription may be visualized as a "collapse of the wave function" but this collapse is not a real physical process in any sense, it is just a rule for us to know which amplitudes are relevant
Correlation is not causation: the fact that two spatially or temporarily separated measurements are correlated doesn't mean that one of them has physically influenced the other; instead, in all EPR-like experiments above, the correlation between the two measurements appears because both measurements share a common past - or a common "cause", if you wish; this comment is the typical and nearly universal reason why there's never any propagation of "faster than light" signals in any similar experiments as many people incorrectly say