I think there may be a space for super-large multi-GB files served from static storage being accessible from SQlite as well. Another one would be this full-text search of a 43GB SQLite database of Wikipedia's full text search: http://static.wiki/ . Hearing there's official support for this is awesome and I hope they also might add some nice stuff for those sticking with POSIX/Emscripten as well; maybe some optimizations to access patterns or other stuff like indexing or split DBs?
There are some amazing things for SQLite in the browser especially if you're looking for ways to host queryable data for cheap. The example I have below costs $0.42 cents a month to host. For 28GB. Insane.
I have a hacked up POC experimental version of the datasette-lite Python UI which runs in Emscripten to be able to look at multi-GB databases at https://github.com/simonw/datasette-lite/pull/49. It uses a hacked up chunk'd lazyFile implementation from emscripten and others to grab pages from Cloudflare R2.
That demo link is so impressive: I just watched my browser DevTools and it loaded 23MB of data in order to run that query against that 28GB database.
I really need to dig in and figure out how Datasette (and Datasette Lite) can work better for this. I think this issue might help - the ability to turn off row counts entirely: https://github.com/simonw/datasette/issues/1818
(I noticed that trying to access the table directly seems to suck in a LOT of data, presumably because it's trying to calculate a count across the whole table?)
> I think there may be a space for super-large multi-GB files served from static storage being accessible from SQlite as well. Another one would be this full-text search of a 43GB SQLite database of Wikipedia's full text search
To the best of my knowledge, OPFS's current quota is about 256mb (per origin).
The browsers currently have no way of viewing/managing the content of OPFS, so it's sort of a storage black hole. i can't even tell you how many sqlite3 database files have been orphaned in my local OPFS since development of the new sqlite wasm support started, with no reasonable way of me being able to find them without using the OPFS-specific JS API to fish through the storage (which i haven't yet been willing to do).
OPFS storage cannot sensibly be exposed at the system filesystem level (i.e. browseable with a file manager) because that would open not only security holes (the ability to "side load" data into any origin) but also huge file locking headaches, especially on platforms which use virus scanners.
There are some amazing things for SQLite in the browser especially if you're looking for ways to host queryable data for cheap. The example I have below costs $0.42 cents a month to host. For 28GB. Insane.
I have a hacked up POC experimental version of the datasette-lite Python UI which runs in Emscripten to be able to look at multi-GB databases at https://github.com/simonw/datasette-lite/pull/49. It uses a hacked up chunk'd lazyFile implementation from emscripten and others to grab pages from Cloudflare R2.
Here's a test/demo with california's unclaimed property records (https://www.sco.ca.gov/upd_download_property_records.html) of a 28GB searching up that guy who owns Twitter:
https://datasette-lite-lab.mindflakes.com/index.html?url=htt...