What forced the US to ground all planes? A router switch, we now know

Airlines are still scrambling to return operations to normal after the longest and only second nationwide stop of flights in the United States since the attacks of 9/11 in 2001.

More than 5,000 flights were cancelled in the US on Saturday after over 9,000 were scrapped a day earlier as the knock-on effects continued from Friday’s abrupt ground stop. The stoppage was halted after more than four hours on Friday night, but airports and carriers were still grappling with large backlogs and resulting delays.

The decision to suddenly ground all planes on Friday initially triggered speculation of a cyberattack.

But officials have since ruled that out and named the reason for the weekend disruptions as a routine replacement of the communications routers that connect the US air traffic control system.

Here’s everything we know about what went wrong:

The Federal Aviation Administration (FAA) ordered a nationwide ground stop of flights at 3:54pm East Coast time (20:54 GMT) on Friday, citing a telecommunications outage affecting air route traffic control centres.

The outage caused thousands of flights to be cancelled or delayed across the US.

The ground stop was removed at 8:15pm on Friday (01:15 GMT on Saturday), but hundreds of flights were still delayed or cancelled that night, according to the aviation analytics firm Cirium.

Aircraft already in the air were allowed to continue to their destinations and land during the ground stop while international flights were handled on a case-by-case basis with some allowed to depart despite the restrictions.

A routine upgrade was being carried out – maintenance work that ultimately overloaded antiquated equipment and caused “intermittent” communications and radar gaps, leading to the ground stop, US officials said.

L3Harris, a contractor, was replacing a section of the system’s old and discontinued copper wire with new fibre optic lines in a remote location on Friday afternoon, Transportation Secretary Sean Duffy said on Saturday.

The router in that location then started overloading the air traffic control system, causing communications disruptions. When officials couldn’t locate the cause of the overload, they grounded flights nationwide.

“When we see that, your safety is paramount. We shut down the airspace,” Duffy said.

The transportation secretary said “contractor error or a faulty piece of equipment” caused the router to start flooding other ones.

“We had 1,000 connection attempts every four to five seconds,” he told reporters.

The issue affected communications at 17 of the FAA’s 22 en route centres.

The Professional Aviation Safety Specialists (PASS), a union representing FAA safety employees, said the issue was isolated to two routers that were updated on Friday. An official told the Reuters news agency it was believed the routers were in Arkansas and Texas.

An FAA official said the agency conducted a safety assessment before the work. The union said PASS-represented technicians were not involved in the upgrades that led to the ground stop.

Ageing technology and equipment alongside a shortage of several thousand controllers have long been concerns plaguing the US air traffic control system.

Duffy on Friday called for $50bn in funding to upgrade these systems, up from his earlier request of $30bn after Congress approved $12.5bn last year.

White House officials had convened an emergency meeting in the Situation Room on Friday amid concerns that the incident was due to a cyberattack. But the US government has since said it did not believe that was the cause.

“Once we were able to isolate the work being done at this location, all of our systems came back online,” Duffy said. “So we don’t believe this was cyber at all.”

The FAA chief reiterated that view at Saturday’s news conference.

“We see zero evidence of anything nefarious and zero evidence of anything AI-related,” FAA Administrator Bryan Bedford said.

L3Harris, a US-based aerospace and defence company, is the contractor behind the systems upgrade, the US Department of Transportation said.

Three-fourths of the company’s revenue comes from US government contracts, largely from the military, The New York Times reported.

“What we’re working to determine was whether we had incompetence, meaning was it … installator error from L3Harris? Or did they install a faulty router?” Duffy said. “We haven’t determined what the fault was.”

Duffy added that L3Harris, which he said had performed decades of work for the FAA, “didn’t meet the moment”.

“They own this network. They’re our prime contractor. I can’t go somewhere else. They own it,” he said. “Frankly, we’re going to hold people accountable.”

In response to Duffy’s allegations, L3Harris told Al Jazeera it has been working “in close coordination with the FAA” and safety and reliability are its “highest priorities”.

“We are supporting the FAA in its investigation to understand the conditions that led to the November 13 ground stop of our nation’s air travel,” a spokesperson for the company said.

“L3Harris takes the nationwide ground stop and its impact on travellers, airlines and air traffic operations seriously.”

L3Harris said it has been “modernising” the communications infrastructure supporting the National Airspace System in “close coordination” with the FAA.

It added that it has completed more than 70 percent of the planned airspace communications conversion from legacy copper wires to a fibre optic system.

The FAA said it was doing additional testing with L3Harris to see if there are any further risks.

Bedford, the head of the FAA, cautioned that it could take time.

“We have crews and aircraft that are out of position for 12,000 flights that were lost,” Bedford said. “These crews, pilots and flight attendants need to get to their points of reconnection with their schedules.”

He stated that he thinks “the worst is behind us”.

Duffy said airlines were expected to restore their full schedules by Sunday night.

Airlines have issued travel waivers for passengers affected to rebook flights for later dates at no cost.

Tens of thousands of people remain stranded in airports across the country while others have been left to pay surge prices for hotels and rental cars as airlines scramble to resume flights.