Yeah, this is a preliminary report, they only report direct factual findings that they have collected so far, and as they mention in this report, they haven't yet examined the electronic manufacturing records that the insider was referring to.
The NTSB is not quick to jump to conclusions; they are very thorough, and they collect and sort through a lot of evidence. It's not surprising that there's information out sooner that says more for such a high profile incident; but once the NTSB final report comes out, it will be quite detailed and thorough, and cover all of this and more.
Yep, biggest new thing here is discussion of witness marks showing no evidence of the bolts, and photo before the rivet repair showing that at least two of the bolts were present, and photo after the rivet repair during installation of the insulation showing at least 3 bolts missing.
So this all just serves to confirm that report, and what people suspected for a while; the bolts were just missing, removed for removing the door plug for the rivet rework and never reinstalled.
there was apparently two (identical) procedures -- one for "removing" the door plug and one for "opening" the door plug -- but only one of them actually called for paper-work and quality inspection after it is done.
It seems they decided to "open" the door plug instead of "remove" it to work on the sealing issue? -- and thereefore documentary record of the work performed and inspection thereafter is lacking.
It would be useful IMO to explore what led the team to prefer "opening" versus "removing" -- and if the subtle difference in documentation requirement was a consideration in preferring one over the other -- and if that points to deeper pervasive culture issues that leans towards less work rather than give safety and quality the primary importance.
I doubt this was intentional? It sounds more like mistakenly applying the procedure for opening a normal door, which requires no followup, to opening a door plug, which does.
what the Spirit mechanics had to do was a RMV, they knew this, because the Boeing eng rep told them so, but they didn’t like this, so they marked it in the system as an OPEN but performed the work of a RMV, to avoid the additional quality checks a RMV would require. And then they fucked up the RMV but didn’t know and no one else did because it was marked as an OPEN so no one checked the bolts.
this was very intentional. the Spirit mechanics thought they were hot shit and didn’t need to follow the stupid, slow process. and they abused the system to allow them to file the work as more innocuous than it was. because they didn’t need anyone checking their work, or whatever they were thinking.
My experience in across a range of technical and consulting roles has been that junior folk understand the need gor their work to be reviewed and checked, because they know that they are inexperienced. Good senior folk want their work to be checked because they are experienced enough to know that they are fallible. It is the mid level folks (especially those who are promoted above their competence to senior roles) who think that they are good enough not to need hheir work reviewing.
This is my experience as well, but it’s usually not those promoted above their capability but those who were passed over and then grow resentful that end up causing the issue. An otherwise competent, but now bitter, middle aged guy on the third shift (or otherwise with less supervision than usual) who is going through a divorce or some other major life stress is the scenario I see most often. They fall behind, they come in to catch up, they have to leave unexpectedly for a family medical crisis, they fall further behind, and so on and so on and before you know it, a perfectly rational person is doing stuff like half-assing the QA checks and it seems almost normal.
But you’re right. It’s not the black belts you have to watch out for it, it’s the browns.
It's amazing, your experience with Git is so different than my own.
I routinely open a file in my editor, hit "Ctrl-c v B" for Git Blame mode, go to the line I'm interested in, and hit "Enter". Bam, there's the full commit message. From there I can can continue to trace backwards, blaming lines and reading full commit messages.
But, you know, not everyone uses Emacs and Magit, fair. How about just using "git gui blame file"? Click on a blame line, see the full commit message. This is a tool included with Git (available in a separate package in some installations).
OK, rather use an IDE? Install GitLens in VSCode. Easily accessible blame in your editor, where you can hover or click in various places to see full commit messages.
I mean, I agree in part; there are some tools which make good commit messages hard to write or find. The tiny little commit message edit box in VScode is not ideal. Lots of people use a workflow of "commit lots of crappy commits with one liner commit messages, let GitHub/GitLab squash them on merge."
But as an expert Git user who has managed to convince some teams to have a good commit message culture, if you do get people used to writing good commit messages, they can be very easy to find and read later on, there are tons of tools that make them easy to browse.
I regularly use the git command line, and "git show (pasted SHA)" in my second terminal doesn't really feel like the road block to understanding the grandparent seems to make it out to be. It takes me many orders of magnitude more time understanding what is output rather than searching for it, and like you mentioned there are any number of UIs (third party, editor integration, or even shipped with git like gitk) that wire everything up into a nice UI.
And I also disagree with the GP's complaint that "Most people only read the shortlog" being any kind of disadvantage. The commit message isn't for everyone, it's for the one time someone needs to figure out exactly what it did and why that commit was made, and why a change in X causes a behavior change in Y, and can save hours of work. It's like code comments, 99 times out of 100 you don't need them as you're just interacting with a documented API, but that 1 other time they are a godsend.
Aviation isn't perfect; nothing implemented by large groups of fallible humans with budget constraints will be. But it has one of the best safety track records of any industry.
Now, why won't aviation style engineering be applied in other fields, like databases? Well, because no one really cares enough. No one dies if some random database used by some ad platform loses the occasional transaction. Yeah, it's frustrating to engineers who are trying to build reliable systems, but in the grand scheme of things losing a few percent of transactions isn't the end of the world for most businesses.
You get safety cultures like that in aviation because there are real, substantial risks, so you need to have thorough engineering discipline, properly designed redundancy, etc.
For databases that are used for the majority of the business world, efficiency is generally a bigger concern than correctness; they'd rather have cheap and fast databases that lose a few transactions occasionally than something that actually provides consistency. But of course everyone thinks they need consistency, so it's advertised as a selling point while not actually being provided in practice.
> For databases that are used for the majority of the business world, efficiency is generally a bigger concern than correctness; they'd rather have cheap and fast databases that lose a few transactions occasionally than something that actually provides consistency
'majority' is a strong claim. Any figures to back it up?
No, no figures to back it up. This is just based on anecdotal experience, having never seen very many businesses that actually prioritize consistency or testing or validating it. The fact that Kyle keeps on finding these massive problems in popular databases is part of that anecdotal evidence.
Obviously people want correctness; they would like their databases to not randomly lose data. Hence the fact that it's a highly advertised feature.
But when it actually comes down to selecting a database, convenience and performance seem to be what people actually compare on, there are very few places that actually hire someone like Kyle to dig in and verify the the consistency claims about a database.
I'm positive people want their databases to be correct. There are databases which promise great speed in return for occasionally losing bits of your data, and they get very little use outside of special uses.
Exactly; people would like their databases to be correct and consistent, but generally don't care enough to actually do something like hiring someone like Kyle to verify it before buying a database. They just find something with the right features and performance, and go ahead and use it. You see a lot of them in this thread; people who used RavenDB for a while, and then had to migrate away because of issues. If the business actually cared enough about consistency, there would have been some kind of validation and verification before selecting a database.
"Failed Jepsen tests" is relative; Kyle tends to be quite thorough in his Jepsen tests, so can usually find some issue in even the best of databases.
The big difference is in how severe the difference between claims and reality are, and how the maintainers or vendors react to such bug reports.
In some cases, the maintainers or vendors will fix bugs, or update docs to be more clear. The different transaction isolated levels are complicated and nuanced, and there are standards which disagree with the general consensus in the literature, so there can easily be ambiguities that need to be cleared up or bugs that need to be fixed.
But then there are things like RavenDB; where they make clearly impossible claims like "ACID across multiple documents and multiple cluster nodes in an AP database." There is simply no way to achieve this.
And then there's how they respond to his findings. He filed bugs and had had responses a month ago, about things like "this thing that is documented to have transactional semantics does not have transactional semantics", and their response was just to say "oh, yeah, that's expected", and not fix anything or update any of their documentation to reflect that.
So, there's a big gulf between "this is a complex topic, and even some of the best systems have some issues if you test it thoroughly enough", and "the documentation is blatantly lying about transactions and consistency, and the CEO of the company doesn't think it's a problem."
The batched operations that are sent in the SaveChanges()
will complete transactionally. In other words, either all
changes are saved as a Single Atomic Transaction or none
of them are. So once SaveChanges returns successfully, it
is guaranteed that all changes are persisted to the database.
But the tests in this post show that no, there is no single atomic transaction for a session saved with SaveChanges, even in a single node database this will lose writes.
I dunno. If I paid for an ACID database, I'd expect, well, some ACID features, like the ability to run two transactions concurrently and have them be isolated. It looks like RavenDB is fundamentally not implementing even the most basic of its claims. This isn't some "oh, yeah, this is a complex problem, and there are a few bugs lurking in obscure corner cases", this is "it fundamentally doesn't support what they claim to support as the major front-page selling point."
Their front page selling point is "Fully transactional NoSQL database" which links to a page that says "A database without transactions is… well, not much of a database, and as far as transactions are concerned – ACID is the gold standard."
But then in response to these findings, they say that the thing that is documented to have transactional semantics doesn't actually have transactional semantics.
No, something being under github.com/google means the person who started it was paid by Google, not paid by Google to code this. Google contracts (like most tech contracts in the US) have ridiculously broad IP assignment clauses, so unless you go through a lengthy process to request Google disown something, they own anything you code, and they insist you open source your things under github.com/google.
You decide your own definitions, but that's very different from "Gmail by Google" or even "Go by Google" in my book. Note how the main author has "Ex-Google" in their bio, too.
Not necessarily. You can write open source code in your own time and publish under Google org on GitHub. This is the recommended process if you don’t care about retaining the copyright to your code.
If someone does want to retain copyright, there’s another process for getting approval.
To me its the same thing, they are paid my Google to code stuff the is put in their org and not their private accounts/orgs so to me this IS in fact "by Google".
Who are these Americans? I have never once been happy with my health insurance; I mean, sure, it beats not having it in this country, but I constantly get billed the wrong amount because someone at the hospital didn't code it exactly the way the insurance company requires, or suddenly my health insurance decides that my doctor or dentist isn't covered so now I have to pay up front and do a song and dance to get reimbursed for some smaller fraction of the price, or whatnot. If I go to the ER (even accidentally, when I asked for urgent care but got directed to the ER instead), I've got a $500 dollar bill waiting for me regardless of what the treatment I needed was. Oh, or I'm on a medication that's prescribed by my doctor, but suddenly the insurance company decides it's not covered and I have to pay hundreds of dollars out of pocked while I chase my doctor down to send them more documentation that yes, they tried other alternatives first and they weren't effective, yes I really do need this medication.
And I pay more for this, just in terms of my monthly premiums, not to mention all of the out of pocket and deductible costs, than I would in taxes for healthcare in just about any other developed country.
I don't know what metric people are using when saying that they are "happy" with their health insurance, other than comparing it to being uninsured in this country, in which case, sure, it beats that.
Kaiser in California is pretty great. It's like an all-in-one thing[1] with the general doctors, pharmacies, specialists etc all under one roof. It's bureaucracy heavy, but sensible, sort of thing.
[1]: this arrangement is also technically illegal in California, but worked around with a Three Corporations In a Nonprofit Trenchcoat sort of deal. The seams mostly don't show.
Mostly you still have to use their tools, Chisel just compiles to Verilog and then you use their tools for the rest of the process.
There are open source tools based on reverse engineering the bitstream for some smaller FPGAs like the Lattice family, some preliminary work on Xilinx, and one vendor actually supporting the open source toolchain (QuickLogic), but for anything serious on the major FPGA platforms, you still need to use the vendor toolchain.
Yeah, I'm working on a fly by wire eVTOL project, we are using the CAN bus as our primary bus, but there are a number of off the shelf components like ADAHRS that we use that talk ARINC-429 so our FCCs will have a number of ARINC-429 interfaces.
But at least for the components we're developing, we have basically standardized on ARM, the TMS570 specifically since it offers a number of features for safety critical systems, and simplifies our tooling and safety analysis to use the same processor everywhere.
Z80 is pretty retro, and while I'm sure there may be some vendors who still use it, it's got to be getting pretty rare for new designs, between all the PowerPC, Arm, and now RISC-V processors available that allow you to use modern toolchains and so on, I'd be surprised if many people were doing new designs with the Z80
The NTSB is not quick to jump to conclusions; they are very thorough, and they collect and sort through a lot of evidence. It's not surprising that there's information out sooner that says more for such a high profile incident; but once the NTSB final report comes out, it will be quite detailed and thorough, and cover all of this and more.