My first programming role was on the back of work by an engineer in this article. The core of the solver was a FORTRAN implementation of a paper on p-convergence. It was really amazing seeing our software predict how a small crack in a part of an aircraft would propagate. The 3D model produced matched the photograph shared later.
The lead developer (at the time) once said that the biggest software failure we can have is not incorrect results, but incorrect results without the user knowing. This is probably why I am so bothered by silent failures in my big company role now.
For physical processes, you can sometimes lean on conservation laws. Financial, no-arbitrage "laws". Sometimes it can help to run two different models, or different numerical methods, and compare their results. More generally, it's very, very hard. I am fairly sure there are multiple published results in computational fluid dynamics that are subtly wrong.
All manner of tests are done in fluid-dynamical modelling. Test suites are developed along with models. Simulations are done with a range of alternative models, or individual models with different conditions are applied, and the differences between the simulations are studied in detail. Models must rely on a fairly large set of parameterizations of processes that cannot be resolved directly, and tests are done to see which parameterizations make the most sense.
Models stem from the academic environment, not the business environment. This means that the models are open-source, so everyone can see what everyone else is doing. And 'everyone' is a pretty large set.
Note that there is no commercial element to any of this, which means there is no incentive to hide problems. It's the reverse: reputation is earned by finding problems, not by hiding them.
A large part of the challenge of climate modelling is the leveraging of increasing computer power to resolve progressively smaller scales of motion, and this uncovers the need to understand those scales in isolation. Modelling is often used at this level also.
None of this is to say that climate models are perfect. They obviously are not. But the system of academic science is very good at improving models and, importantly, exposing their limits. An indication of the latter is the pairing of uncertainties with predictions: a hallmark of this scientific community.
They are all wrong. Historically they've overpredicted warming, and then the models are updated to fit the (new) data better. They perform worse in areas with less data, like polar regions.
To forestall knee-jerk downvotes: I'm not saying climate change isn't real or that anthropogenic global warming doesn't exist. I'm saying the models are not yet developed enough to predict very accurately. Early heliocentric models made poor predictions too because they assumed circular rather than elliptical orbits, they were still more "right" than geocentric models.
It depends on how you define "quite well". They are definitely directionally correct and in the right ballpark. This is a politically sensitive research area so the researchers are incentivized to make strong claims that you wouldn't make about, say, predicting sports outcomes or the stock market.
The first link you posted is kind of punting on the hard part by saying that the reason the models overpredict warming is because CO2 didn't rise as much as they expected so if you put the actual observed CO2 concentration in then the temperature prediction comes out closer to what was observed. But the CO2 concentration is a parameter of the model, so they didn't capture its dynamics properly and then had to retroactively change it to get the observed data.
Again, I want to reiterate that I'm not disputing the process of climate change or saying it's not a problem. I'm saying that modeling it is hard and historically the models have overestimated warming.
That doesn’t seem like a fair criticism. Most climate models are physics models. They can’t know how much CO2 people are going to add to the atmosphere. But if you tell them how we altered the composition of the atmosphere, they predict the temperature change correctly. That’s what I mean by good performance.
Part of the job of the modeler is to make good decisions about the parameters and their uncertainties. If they are systematically overestimating CO2 and getting high predictions as a result then they aren't modeling properly, unless there are exogenous reasons why CO2 is lower than expected. Which might be possible for all I know (Global financial crisis maybe? China growth lower than expected?) but the dynamics of CO2 are part of the model they are using for prediction so they can't just punt on it by saying they didn't know what CO2 would be. Prediction's hard, especially about the future, as the saying goes.
No sarcasm intended. The science of climate change rests on many legs. It was already established long before massive models on the scale we have today were possible. Computer simulations attempts to understand how it will affect us, but does not contribute much evidence.
> I know it's not a simple answer, but how would you embed checks to flag or highlight potentially incorrect results to the user?
For starters, some problems do have analytical solutions. Yo can compare a FEM model of these problems with the known analytical solution and see if it's close enough or not. One of them is the elasto-plastic plate with a hole.
Revising contracts with binding arbitration clauses has some recent precedent in the Supreme Court of the United States. Epic Systems Corp. (not the game company) sent employees a memo which can be paraphrased as, “you can no longer class action us. If you keep coming to work, you show you agree to this change.”
The lead developer (at the time) once said that the biggest software failure we can have is not incorrect results, but incorrect results without the user knowing. This is probably why I am so bothered by silent failures in my big company role now.