24 August 2011—When the science and engineering team that operates the Chandra X-Ray Observatory celebrated the space telescope’s 12th year in orbit on 23 July, it was with a sense of real accomplishment—and also a collective sigh of relief. Just a few weeks earlier, the NASA satellite suffered the kind of mysterious failure that had the team worried that its long-lived instrument was about to sign off. It also had the astrophysics community concerned that its chief tool for taking X-ray images of the universe might be lost.
The nearly 4.8-metric-ton telescope is capable of imaging the hottest objects in the universe. Notably, it has provided astonishing insights into the anatomy and physiology of black holes and their crucial role in galaxy formation and destruction.
"It lets us see the freshly made elements in stars that have just exploded," says Jonathan McDowell, an astrophysicist with the Chandra X-Ray Center , in Cambridge, Mass. "We can look at supernova remnants and see, oh, here’s a lump of iron the mass of the sun that’s just been created." (Project scientist Martin Weisskopf acknowledged the telescope’s accomplishments with a "top 10" list of Chandra’s scientific achievements.) Though more specialized X-ray telescopes have been fielded over the years, Chandra remains the most sensitive.
The first sign of trouble came on the afternoon of 6 July, when Chandra failed to respond to a routine call. Running through a checklist for such events, the team reduced the data rate used to download telemetry data from Chandra. "Once telemetry was flowing, we found the critical onboard components had switched to their backup units," explains Michael Juda, Chandra’s mission operations manager. (As with most big satellites, Chandra carries a nearly complete set of redundant components, including gyroscopes, heaters, and the power electronics for its solar array.) The switch meant that Chandra was now in "safe mode," a kind of holding pattern it enters when it detects a possible failure. In safe mode, the satellite shuts down its scientific instruments, points its solar panels toward the sun so that it continues to collect power, transfers control from its main onboard computer to a backup unit called the control processing electronics, or CPE, and awaits further instructions. It was the first time the satellite had done so in 11 years.
"We had three safe modes during our first seven months in orbit [in 1999 and 2000] due to operational errors—we were still learning to drive the satellite. But this was the first time the safe-mode information indicated there was a problem onboard the satellite itself," explains Harvey Tananbaum, who with astrophysicist Riccardo Giacconi proposed the telescope back in 1976. Tananbaum now serves as director of the Chandra X-Ray Center, based at the Smithsonian Astrophysical Observatory, which operates the telescope for NASA.
The next two and a half days were "nerve-wracking," Tananbaum says. Under normal conditions, the operations center communicates with Chandra once every 8 hours or so via the Deep Space Network, the NASA system of large ground antennas that allows communication with space probes throughout the solar system. Realizing they’d need to stay in closer contact with the satellite, the operations center staff asked other users of the network to turn over some bandwidth to Chandra. "Other projects were really good about giving up time," Tananbaum says. That generosity allowed his team to remain in near continuous contact with Chandra over the next several days.
Working through the night, staff members discovered that the safe mode had been triggered at 8:28 eastern daylight time on 6 July, when Chandra’s computer detected an unusually large change in its angular momentum. Tiny changes are okay: Based on readings from the gyroscopes and the aspect camera (a visible-light star tracker used to help point the satellite toward its scientific targets), Chandra’s flywheels routinely make small corrections to maintain its alignment, which can be perturbed by things like the solar wind, Earth’s magnetic field, and the difference in the pull of gravity across the satellite’s body. But what the computer showed was a momentum change more than 25 times as great as the telemetry indicated by the gyroscopes and the flywheels, also known as reaction wheels.
Over the next two days, Tananbaum says, there was "a lot of analysis going on, a lot of scratching of heads." There didn’t seem to be anything wrong with Chandra’s hardware, so the working hypothesis was that the computer’s data had become corrupted.