Ok, thank you and others very much for your detailed info. I did use OpenSolaris for a brief time, but kind of lost track after Oracle came and took it.
In regards of the free tier, now I found it, but its mention is placed as the last line, after all the paid plans and the part about "Contact sales if your requirements exceed everything we've planned a price for" ;)
About Manta, I'd like to know about what kind of data objects it supports? Like having a matrix of time series data and making quick selections and sorts, near real time?
Docs talked about hierarchical storage and search, which sounds really useful, too, but what is search in this context?
* Log into a Manta zone using bbc.png as my input file. This creates a zone, and maps in my file which is stored on the same machine (you are always operating on the same host as your data is stored). mlogin(1) is a nice way to prototype jobs in an interactive session, and once you have it working correctly you can use mjob(1) to run it automatically.
$ mlogin /jperkin/public/tmp/bbc.png
* created interactive job -- f1a2e579-34f8-4dd4-da19-db33954a0772
* waiting for session... | established
jperkin@manta #
* At this point it's just Unix, so I can run any command on the file (which has been mapped in under /manta) I like:
jperkin@manta # uname -a
SunOS 0ae1c6ec-d47a-455c-9dd6-97eec16da31b 5.11 joyent_20140628T000418Z i86pc i386 i86pc Solaris
jperkin@manta # ls -l /manta/jperkin/public/tmp/bbc.png
-rw-r--r-- 1 root root 540751 Nov 7 11:41 /manta/jperkin/public/tmp/bbc.png
jperkin@manta # file /manta/jperkin/public/tmp/bbc.png
/manta/jperkin/public/tmp/bbc.png: PNG image data, 1680 x 940, 8-bit/color RGBA, non-interlaced
* Note that only the file I chose has been mapped in:
Of course this is a simple and contrived example, the real power of Manta comes when you have say 1,000,000 log files stored under a particular path and want to grep them all for a particular string. To do that you'd do something like:
This will scale to whatever size your Manta cluster is, e.g. if you have 10 hosts then the log files will be split up across those hosts and they each will spin up multiple zones to run "gzcat | grep" on the local data, before a final "cat" reduce job is used to collate the results from each map job.
Wow, thank you, again. I have to definitely take a look this. My use cases tend to vary so much that creating a Hadoop like system would require too much custom coding.
I wonder if it is possible to have compression and de-duplication, so that there could be a one big base dataset and lots of containers that only add what new data they generate.
Anyhow, looking at this it feels really approachable. What I have in mind are quick-and-dirty data-sciency scripts for ad hoc use cases, like diffing structured files and combing over matrix data.
In regards of the free tier, now I found it, but its mention is placed as the last line, after all the paid plans and the part about "Contact sales if your requirements exceed everything we've planned a price for" ;)
About Manta, I'd like to know about what kind of data objects it supports? Like having a matrix of time series data and making quick selections and sorts, near real time?
Docs talked about hierarchical storage and search, which sounds really useful, too, but what is search in this context?