Dodo has blamed a faulty router for bringing down the internet for millions of people across Australia yesterday, but did Telstra have the power to stop it?
(Credit: Josh Taylor/ZDNet Australia)
International traffic was completely cut off for millions of mobile and fixed-line customers for a number of the major telcos, including Telstra, iiNet, Dodo and Optus, between 1.40pm AEDST and 2.25pm AEDST yesterday. The issue also brought down the Commonwealth Bank's online services.
Late yesterday, Dodo took the blame for the national outage, saying that it had "experienced a hardware issue with a Cisco border router" that had caused Dodo to broadcast network routes to Telstra. These were accepted by Telstra, and propagated to downstream customers.
This afternoon, Telstra said in a statement that the outage had been caused by a configuration issue between Telstra's wholesale-customer network and Telstra's core network.
"At this stage, we understand [Dodo] began to present an excessive number of IP routes, and the configuration of Telstra's core network (Telstra Internet Direct) allowed this to overload the Telstra network," Telstra said. "Service was restored by removing the impact of excessive routes."
It has been speculated on network-operators group AusNOG that the border gateway-protocol interface between Telstra Australia and "Reach", or Telstra Worldwide, went down because Dodo advertised close to 40,000 prefixed routing tables to Telstra, which were accepted and then advertised to Reach. However, Reach apparently had a filter that realised there were too many routes for Telstra to have, and it shut down the border gateway protocol to Telstra.
Had Telstra had a similar filter in place for Dodo, yesterday's outage would only have affected the budget internet service provider, rather than causing a chain reaction that brought down the internet for millions of Australians.
Telstra will be working hard to ensure that this kind of thing never happens again. The company said that steps were taken yesterday to add additional protection to Telstra's core network, and a full review of the protection mechanisms is underway.
I imagine that whoever let this happen at Dodo hasn't been let off lightly, either, but it was an interesting insight into just how wide an impact a relatively short-lived internet outage can have across so many businesses in Australia.