Yours is the first example where I a decent knowledge of the field, so can understand the needs accurately. In most cases I see people using NoSQL in places where MySQL could handle it easily.
Maybe I am just too set in my relational database ways of thinking (having used them for 13 years), but there are few cases where I see NoSQL solutions being beneficial, other than for ease of use(most people are not running Facebook or Google).
I a bit sceptical in most cases (though I would certainly like to know where they are appropriate).
First, I should note that my needs are fairly specific, and not typical of the rest of the NGS world. The datasets are essentially the same though.
The rate at which we are acquiring new data has been accelerating, but each of our Illumina datasets is only 30GB or so. The total accumulated data is still just a few TB. The real imperative for using MR is more about the processing of that data. Integrating HMMER, for instance, into Postgres wouldn't be impossible, but I don't know of anything that's available now.
Edit: A FDW for PostgreSQL around HMMER just made my to do list.
Is that the same as an 'impossible to do otherwise' case?
Edit: I should say 'currently impossible' since as I noted, I can imagine being able to build SQL queries around PSSM comparisons and the like. I just can't build a system to last 5+ years around something that might be available at some point.
That comment was based on your "into Postgres wouldn't be impossible" phrase.
No fair enough of its the only way you can get things to work. I still see lots of people jumping on the "big data" bandwagon with very moderate sized data.
