The project I work on, Mesa3D, routinely ignores malloc failures for a couple of reasons: no one knows how to reasonably test the additional code, and often no one has any idea what sort of graceful failure is even possible when malloc fails.
We've had some cargo culted patches to call _mesa_error_no_memory() when malloc fails, but it was recently noted that this can't possibly work because it internally calls fprintf, and there's no guarantee that fprintf doesn't call malloc! In fact, glibc's implementation does.
The proper way to deal with a memory allocation failure, when a graceful return isn't possible, is to call abort() or similar, instead of trying to continue with a potentially inconsistent state.
Of course, it would be better to avoid getting into a situation where either a graceful return isn't possible, or poorly-tested recovery code has to be run to deal with the failure. I read somewhere that seL4's strategy is to divide the work in two phases. The first phase allocates all the resources it will need and checks all preconditions, but changes no state; the second phase has no allocations, does all the work, and will never fail. That way, any error recovery is a simple release of the resources it had already allocated.
Check out the concept of fault injection -- you can wrap all of the malloc() calls and the systematically cause each one to fail. You might want to combine that with valgrind to make sure you don't end up doing undefined things after a malloc failure.
You can also use that wrapping technique to test _mesa_error_no_memory(), by setting the state to "all mallocs fail" and then calling it. Hint: Don't call fprintf, don't use stdio, call write() directly.
Fault injection can't cover all of the combinations of all inputs and all places to fail, but it's a lot better than nothing.
If you don't have anything better to do, just abort().
One (naughty) option is to malloc a decent sized buffer at startup (50MB say), and then when malloc fails free it, and immediately display a warning box.
One most modern 64-bit systems, malloc will never fail anyway, you'll get killed due to memory overcommit.
The alternative, if you have no good way of testing the codepaths, or if you're running on a system with overcommitted VM, is basically segfaulting. And you're running on a system with overcommitted VM.
Overcommit is not actually relevant. An overcommit OOM-kill is a manifestation of the administrator killing the process for being a memory hog (it's just that the administrator has delegated this decision to some less-than-scrutable kernel code).
Library code should in general punt out-of-memory conditions back to the caller.
It is if you ever want to exercise the code paths. Overcommit means you never get an out of memory error. Your process just dies. Kind of like it would if you just aborted.
When you're running tests in development you can easily set up a test environment with and without overcommit enabled (and in fact there are other ways to limit allocated memory on a per-process basis that work even with overcommit, such as with rlimits).
...but that doesn't even really matter, because in order to fully exercise those code paths you need precise control over exactly which call to malloc() returns NULL, which in practice means you need to interpose malloc() with a debugging framework to force a NULL return.
(these kinds of) Crashes can be manipulated, though. If you can't restore a stable state, you have created instability. Failing gracefully in this context means failing without creating instability or insecurity.
The failure mode is the one where you terminate in the middle of an operation that was expected to be atomic or ordered and then the program restarts and does step two without finishing step one.
And operating system crashes, and being killed by the OOM handler (which is probably likely if you're out of heap), and all of the other things that can kill a process without warning.
Sadly it doesn't on linux due to overcommit. You can actually get the failure much later when you try to use the memory. The exception is when you try to allocate absolutely huge buffers. So malloc returning a valid pointer doesn't indicate success (though it returning NULL does indicate failure).
Yes, but it becomes harder to tell where in the code that happened. If you run abort() or the like, you've got more information as to the circumstances of the crash.
It is quite possible to write to files, stdout, etc without using any heap memory. It can be achieved with the bare bones read/write syscalls. I'm not sure I would consider this a "best practice", but it's worth noting. Boehm GC actively uses read when it needs to read files without allocating any heap memory.
> ... no one knows how to reasonably test the additional code ...
wouldn't a simple 'ulimit -m' set appropriate limits on memory for the process ? and then use that to simulate failures, and hopefully see what breaks, fix it, and rinse-lather-repeat ?
edit-1: this is ofcourse predicated on the fact that you are using a unix like system.
We've had some cargo culted patches to call _mesa_error_no_memory() when malloc fails, but it was recently noted that this can't possibly work because it internally calls fprintf, and there's no guarantee that fprintf doesn't call malloc! In fact, glibc's implementation does.
Suggestions for best practices welcome.