Recent SQLite builds include the memdb [1] VFS (and the associated sqlite3_serialize and sqlite3_deserialize APIs) by default. How does this compare?
I'm also curious what use cases you had in mind. My SQLite wrapper reimplements the memdb [2] VFS mostly as an exercise of the VFS API, but it provides a few additional niceties. The backing memory doesn't need to be contiguous, nor is it copied when the DB grows/shrinks. This makes it possible to have significantly larger in memory DBs. I also make it easy to bootstrap the database from (e.g.) a file on disk, or data embed in the binary. It's not as easy, however, to access the DB bytes after the DB goes “live.”
It looks like memory is also not contiguous in your implementation.
You don't support concurrency at all (xLock/xUnlock/xCheckReservedLock are noops).
I'm not sure how APSW handles journaling (you usually want the journal_mode to be memory in these cases, but to force that, you must return SQLITE_OPEN_MEMORY as an OutFlag from xOpen). Otherwise, you need to deal with files (like journals, etc) for which reads/writes are not page aligned.
There is one caveat, though: if you want to “deserialize” a WAL database with journal_mode memory, you must convert it to rollback before opening (but you can do that by editing bytes 18 and 19).
> and the associated sqlite3_serialize and sqlite3_deserialize APIs
So... I have to admit I missed the existence of these. So for many cases, using sqlite3_deserialize is probably a very reasonable choice over sqlite-memory-vfs
Still, as you say, sqlite-memory-vfs doesn't require the memory to be contiguous, unlike sqlite3_deserialize
> You don't support concurrency at all (xLock/xUnlock/xCheckReservedLock are noops).
Pending exists to prevent writer starvation: if there's always at least one reader, no one can ever write.
You take pending before exclusive (which should always be possible because you have reserved by then), and you should check pending before taking a shared lock (no need to check exclusive, because they got pending before exclusive).
Shared forces writers to wait;
Reserved prevents new readers from upgrading to write;
Pending prevents new readers from starting;
and exclusive forces everyone else to wait.
- If doesn't go via PENDING if it doesn't need to. Not sure what's best really - but since RESERVED can already be skipped in some cases (recovering from a hot-journal) then so far happy with "skipping is a thing", and PENDING is private to the VFS anyway from what I can tell, so nothing else should care I think
- It doesn't do the blocking in the lock function, but depends on SQLite's core code doing it via the busy handler/timeout
No benchmarks right now. And in fact, I'm not even really sure what would make good ones, so suggestions welcome.
In terms of the difference from pointing SQLite to /dev/shm, my answer is somewhere between "I don't know", and "I suspect the VFS is more portable because not all systems have /dev/shm?"
While SQLite supports the special filename :memory: that allows the creation of databases in memory, there is no built-in way to populate such a database using raw bytes without hitting disk as an intermediate step. This virtual filesystem overcomes that limitation.
I'm also curious what use cases you had in mind. My SQLite wrapper reimplements the memdb [2] VFS mostly as an exercise of the VFS API, but it provides a few additional niceties. The backing memory doesn't need to be contiguous, nor is it copied when the DB grows/shrinks. This makes it possible to have significantly larger in memory DBs. I also make it easy to bootstrap the database from (e.g.) a file on disk, or data embed in the binary. It's not as easy, however, to access the DB bytes after the DB goes “live.”
[1] https://www3.sqlite.org/src/file?name=src/memdb.c
[2] https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/memdb#s...