I wonder if the corruption could have been caused by a cosmic ray bit flip. It would be funny to think that one single tiny subatomic particle knocked down our entire flight system.
Cosmic rays are a cop out but a good plot device. They're so rare, but one could write a bit flip into a script for a novel or a tv show as the solution to any computer mystery.
Cosmic rays are more common than you think. Google's early infrastructure was impacted by a supernova (because their nodes were so cheap). But something like NOTAM can handle these single bit flips without a problem.
You can expect 250 or so cosmic ray events per second in a 42 litre sodium iodide crystal pack at 100m above sea level.
Source: 10 years airbourne geophysics, radiometric calibrations.
Addendum:
In-flight upset 154 km west of Learmonth, WA 7 October 2008 VH-QPA Airbus A330-303 [1] was a probable (but uncertain) example of cosmic ray events causing multiple spikes in one of three air data inertial reference units (ADIRUs) that also went on to cause a failure mode of the "best of three" reporting system leading to a pitch down in which [2]
> 110 of the 303 passengers and nine of the 12 crew members were injured; 12 of the occupants were seriously injured and another 39 received hospital medical treatment.
HOWEVER .. despite 250 events per second in a 42 litre volume, it took 128 million hours of unit operation to see a failure mode.
It's a lot of billiard balls going through a lot of space and a high bar for "something bad" ( just the right bit flip ) to happen.
One cosmic-ray bit-flip per 256MB per month. Significantly more for computers on aircrafts at higher altitudes. I kinda wish I hadn't learned this fact, how fragile everything is. On the other hand, it makes me appreciate the importance of error tolerance and recovery.
> Cosmic ray flux depends on altitude. Computers operated on top of mountains experience an order of magnitude higher rate of soft errors compared to sea level. The rate of upsets in aircraft may be more than 300 times the sea level upset rate.
The reason we all now wear our seatbelts through the entire flight is there was a cross-Pacific flight that had a sudden altitude drop where several people were injured from rebounding off the ceiling. When the news reported on it, they reported the cause was believed to have been an undetected massive downdraft that shoved the plane down.
I lost the thread on that story before the investigation concluded the actual cause was autopilot error. It is, in some ways, more comforting to me to know that the issue wasn't novel atmospheric phenomena, but instead relatively-mundane cosmic radiation flipping one packet of data from the sensors to the avionics that the avionics lacked sufficient redundancy to detect or discard. As a result, the autopilot believed the plane had suddenly pitched 90 degrees and drastically corrected to escape stall.
Surprising that only two people in all of the comments here suggest to use ZFS. It could have detected any broken data before it was moved to the backup system.
