I use SQLite to store all of my simulation data (~10s of GBs). It's remarkably versatile and the fact that there are good libraries for Python and C++ to interface with and query SQLite dbs makes it a synch to use for data analysis.
I've seen so many people struggle with custom binary formats; I imagine there are countless research hours lost in figuring out how to work with these obscure formats. I've advocated to all students I work with to make use of SQLite to store simulation data for their thesis projects and my experience is that they're quick to pick it up and figure out how to do some pretty complex querying.
It's one of those things that I don't understand about academia: there are so many standards and well-established tools in the tech/IT sector that we don't take advantage of. SQLite and JSON are the two that I constantly advocate to everyone I work with.
I've seen so many people struggle with custom binary formats; I imagine there are countless research hours lost in figuring out how to work with these obscure formats. I've advocated to all students I work with to make use of SQLite to store simulation data for their thesis projects and my experience is that they're quick to pick it up and figure out how to do some pretty complex querying.
It's one of those things that I don't understand about academia: there are so many standards and well-established tools in the tech/IT sector that we don't take advantage of. SQLite and JSON are the two that I constantly advocate to everyone I work with.