I'm looking for a database that can store dense N-dimensional numpy arrays. Query performance is not as important as storage efficiency, the arrays are dense but mostly 0's and 1's and compress well.
I tried pg's array support (very slow) and storing it in pg's JSON field (also quite slow, but more acceptable)
I should also add, we're going to be writing a lot more than reading -- potentially for many processes, so anything with ACID-like support would be great.
But there must be dedicated databases out there for dense numerical data like this?
https://tech.blue-yonder.com/efficient-dataframe-storage-wit...
https://www.slideshare.net/julienledem/strata-ny-2017-parque...
If ACID guarantees are a must then use Kafka as a message broker between memory and file, but this has the cost of added complexity. For more info:
https://eng-staging.verizondigitalmedia.com/2017/04/28/Kafka...
http://activisiongamescience.github.io/2016/06/15/Kafka-Clie...