A Level 3 configuration error rendered significant parts of the United States disconnected from the Internet.
One of the most alarming scenarios that could happen to anyone in today’s digital world is to be shut off from the Internet. However, that’s precisely what happened to many parts of the U.S. yesterday (including in my area, ugh). Monday saw the country in a concerning situation when a series of outages broke out nationwide. The culprit? A Level 3 error.
Primary internet service providers (ISPs) such as Comcast, Verizon, and AT&T all appeared to have issues, and that’s no surprise as Level 3 is a network “backbone” of sorts to these ISPs. These events are uncommon but not rare, as it should be remembered that over a year ago on October 2016, a critical DDOS attack on Dyn, an Internet performance management company, crippled the U.S. internet for a day.
Early Monday, people started speculating that the Internet disruption was due to some sort of attack. DownDetector, a company that monitors issues and outages, confirmed that majority of Comcast users affected by the connectivity issues came from Mountain View, Denver, Portland, Chicago, Seattle, New York, San Francisco, Houston, Minneapolis, and Boston.
So, how could an alleged Level 3 error disconnect a whole nation from the Internet?#Level3Error caused major Internet disruption in United StatesClick To Tweet
The Level 3 Error Crippled America’s ISPs
Level 3 is a Colorado-based multinational telecommunications and Internet service provider. In 2016, the telco was acquired by CenturyLink, another telecommunications company based in Louisiana. Level 3 operates a vast Internet network that spans across 46 states in the U.S.A., major cities in Western Europe, and in Asia.
Level 3, as we mentioned, serves as a backbone company that underpins other big networks. A report from Wired confirmed that a certain Level 3 error, a misconfiguration, caused a routing issue that allegedly created a ripple effect which in turns affected major telcos such as Comcast, Spectrum, Verizon, Cox, and RCN.
In a statement to Wired, Level 3 said:
“Our network experienced a service disruption affecting some customers with IP-based services. The disruption was caused by a configuration error.”
On Nov. 6, our network experienced a disruption affecting some IP customers due to a configuration error. All are restored.
— Level 3 Network Ops (@Level3NOC) November 6, 2017
Comcast confirmed yesterday about outage reports received from its users. However, a Twitter post from the company claimed that it was monitoring an external network issue and not a problem with its own infrastructure.
UPDATE: our teams continue to monitor an external network issue. We apologize for the inconvenience & will provide updates as we learn more.
— ComcastCares (@comcastcares) November 6, 2017
RCN also confirmed having problems with Level 3 internet backbone which it temporarily resolved by rerouting traffic to a different backbone.
Level 3 internet backbone currently has disruptions affecting U.S. RCN immediately rerouted to alternate backbone. RCN service normal.
— RCN (@RCNconnects) November 6, 2017
Roland Dobbins, a principal engineer at the DDOS and network-security firm Arbor Networks, said that the Level 3 error was called a ‘route leak.’
Apparently, ISPs make use of Autonomous Systems (AS) to track what IP addresses are on which networks, routing packets of data between them. Then, the ISPs utilize the Border Gateway Protocol (BGP) to establish and communicate via those routes.
For instance, packets can route between networks A and B. However, network A can use network B to route packets to network C. This is how ISPs inter-operate to allow users to browse the WHOLE internet, not just the IP addresses on their own networks.
Now, if there’s a route leak, an AS or multiple ASes would issue incorrect information about the IP addresses on their network, which compromises the routing process. A route leak could render inefficient routing and failures not just on the originating ISP but on other ISPs trying to route traffic as well.
At times, route leaks are perpetrated with malicious intent. In these cases, they’re called route hijacks or BGP hijacks, typically caused by hackers and unknown attackers. Fortunately, Monday’s Level 3 error was just due to a simple misconfiguration that ballooned to impact major ISPs.
According to data released by Network Research Lab (NRL), route leaks happen occasionally but consistently in recent history. From 2003 to 2010 alone, NRL was able to detect over 60 major route leaks across the globe.
ISPs are trying to minimize the impact of route leaks by using route filters that check the IP routes their peers and customers could potentially use to send and receive packets and attempt to catch problematic IPs.