I think this could be a single pipeline?

ssh username@server "sqlite3 my_remote_database.db .dump | gzip -c" | gunzip -c | sqlite3 my_local_database.db

gzip/gunzip might also be redundant if using ssh compression with -oCompression=on or -C on the ssh call

My first thought, too. It also seems somewhat glaringly obvious that it needs a `pv` in there, as well.

This looks like a very good opportunity for explaining how fast computers are and how spinning up infra doesn't always make sense.

I'd try to show how much a single process could scale by writing a simple benchmark, and comparing the performance, cost and latency of the alternative.

The results should speak for themselves at that point.

(But also, spinning up a lambda to wait for a db request? Wat? Must be missing some more context/details)

I did this. But they said lambdas are so cheap it doesn't even matter.

I think there's a more pervasive problem of all engineers, especially younger ones not realizing that spinning up new serverless infra is harder then firing a coroutine.

>(But also, spinning up a lambda to wait for a db request? Wat? Must be missing some more context/details)

there's this crud server that serves as an admin view for a bunch of IOT devices. Sometimes we want to run a task that doesn't block this server. So they use lambdas for those external task runners.

Except those tasks are mostly database operations. It is a crud app after all. But I'm trying to tell them that you don't need to spin up a new CPU when the database is doing all the computation. Have the server wait for the database. Don't have the server wait for a lambda that waits for the Database. If you do that you're just spinning up a CPU and telling it to wait. It's more complicated.

At my company they think doing this is normal. It's pervasive how normal people feel this type of thing is. I'm just baffled. I'm wondering maybe it's just pervasive across the board. I don't think they realized that query calls on lambdas just make the lambdas wait on the database. The abstraction is so heavy they likely aren't even thinking about how they are provisioning an entirely new CPU to wait for a database.

I'm replying to this one, but I read all your other replies in this post, honestly I rarely laugh when reading here but I laughed more and more as I went through the discussion, and I think that maybe you were dealing with a troll, but I liked your explanations anyway, very informative, thanks for that.

Only a troll would defend that two CPUs are cheaper than one. Especially if you already have this one.

I'm a web developer, not even specialized in backend and I could follow and agree with what you said.

Hope you get better people to interact with next time :P

That's for deleting a tag, but is it possible to remove a tag from a note? (aka untag a note)

I just noticed today that that isn't working as intended. Will be releasing a revamped tags UI very soon that should make all this easier.

Just writing #tag would be nice... :)

Don't forget that rsyslog can also parse and generate structured data (json with mmjsonparse for input + templates with json escaping for output).

It can also queue up messages in memory and/or to disk if your remote data sink is having a hiccup.

And for those wondering how to send multiline data, well you don't. If you need to write out a big blurb, you write it out on a single line from the application. If using structured data, you can output lines as separate array items. The current built-in limit for a syslog line is 8096 bytes, but that's tunable. Just make sure the thing that writes to syslog doesn't have a low hardcoded limit like older versions of logger from util-linux (1024 bytes)

edit: the version of rsyslog shipped with the distros might be a bit dated. They're providing packages for their latest stable version, we're using that and it works pretty well.

6M! that's great. But that's not comparing apples to apples, the best test they did was using a 40 core machine with a 10gbe NIC. These cards usually have as many receive/send queues as the number of cores the machine has, while the ec2 instances only have 2 queues. (and most performance gains in packet receive usually come from handling interrupts on multiple CPUs...)

Would be great to see results for ulib on c3.8xlarge machines.