For anyone who can't immediately see the significance..this is Elasticsearch's entry into real-time log analytics. There is plenty of room for innovation and financial opportunity in this area, given the success of the $5 billion valued Splunk along with companies like SumoLogic and LogLogic.
What's most interesting is that Elasticsearch seems like a completely open source (and widely used) offering of a product that Splunk charges close to oracle pricing for.
Shameless plug: If you're looking for an opportunity at a well-funded true real-time analytics company in silicon valley...feel free to ping me. There's lots of exciting and fun work to do in this area.
The one thing Splunk has going for it over ES is the amount of resources it requires to work at scale.
I needed 12 ES boxes for every one Splunk box to handle the 100MB/day log load of my system, and even then they ran at a high load and searches often failed, and in some cases it took hours for the indexer to catch up.
This experience sounds especially bad. Sorry about that.
As mentioned in another comment in this post, I was doing 300gigs of data per day with an elasticsearch cluster size of 7 elasticsearch nodes (16 cores & 16gb ram per node) and load was around 5-10% cpu utilization.
100MB/day is pretty small in terms of log data, I think. If you attempt this again, please invoke the community (elasticsearch's is great!) and see if we can assist you in figuring out what's busted.
We built a similar system log-searching system using SenseiDB at LinkedIn. Splunk was outrageously expensive.
It turns out that lucene based systems are pretty good at information retrieval and aren't shackled with all the OLTP requirements most databases have.
What's most interesting is that Elasticsearch seems like a completely open source (and widely used) offering of a product that Splunk charges close to oracle pricing for.
Shameless plug: If you're looking for an opportunity at a well-funded true real-time analytics company in silicon valley...feel free to ping me. There's lots of exciting and fun work to do in this area.