So what's the current status of the upstreaming work? It sounds like there's a good start discussed in the paper, in removing a lot of conditional and duplicated code and using some more cleanly separated layers, but that was two years ago, what's happened since?
I don't know about Windows, but on Linux, I would love to have a replacement for the entirety of NSS (and PAM, and other similar parts of the system) that ran in separate daemons rather than by loading shared objects into my process which could arbitrarily corrupt my process if they were buggy (which I have encountered before before).
To achieve that compatibly with existing systems, it would probably need to be able to act as an NSS module itself in order to be compatible with existing clients, and also be able to load NSS modules (hopefully in sandboxed, lower privileged processes) so it could support existing setups transparently, though it would be great to also have a better asynchronous interface so that you don't have to spawn thread or process pools for things like hostname resolution or fetching groups from LDAP.
That's a really neat idea. I like the idea of potentially building something that could be plugged directly into Linux as a PAM module, etc. I've never written something like that, so it would be fun learning how to do it.
> it would be great to also have a better asynchronous interface so that you don't have to spawn thread or process pools for things like hostname resolution
The library is built atop of Tokio, so is already async capable.
There is the UDP source port, which is where the reply will be sent, and a 16 bit ID number associated with each request. Randomizing both of these gives you 32 bits of entropy, which would makes spoofing attacks a lot harder. There were a lot of systems vulnerable to such attacks before source port randomization and ID randomization became common to mitigate such attacks.
Of course, for this attack, you may only need to get the source port to match, since if it's the DNS resolver that has the bug, it may parse the whole response before noticing that the ID doesn't match. And some NATs may break source port randomizaiton, since they allocate their own source ports to keep a table of their source port to the internal IP and source port; if they do so in a more predictable manner, it may be relatively easy to spam them with packets, or you could just spam the whole 65536 bit range with bad packets.
And calls create_mtg_gif, which calls get_giphy_gif, which requests GIFs from the giphy API by searching for the card name, and then finds one that has an appropriate aspect ratio.
While I mostly use Emacs and have only actually written a little bit of Rust, from what I've gathered the two best IDE experiences are either VSCode with RLS (Rust language server, which uses an adaptation of the compiler for reporting syntax errors and various other information like jump to definition, coupled with a simpler but faster engine for things like autocompletion so that you can start doing that before the server finishes a full compile) and the Rust IntelliJ Plugin which uses IntelliJ's native tooling and adapts it to understand Rust, which has the advantage of all being originally written for fast and robust autocompletion, highlighting, and the like, rather than adapting something written as a compiler to provide that information, but the disadvantage that you need to implement a lot of Rust's syntax and type inference so there are some more complicated cases that it can't necessarily handle yet.
If you're interested in trying out a Rust IDE, those are the two I'd try and see how you find them.
Looking at their commit history, it's not a fork in the sense of "we're actively developing our own new features and diverging from upstream", it's a fork in the sense of "we've taken a stable release from upstream, applied some of our own patches, some backported patches, and some patches from other forks."
It's a fork as much as the stable packages that RedHat or Debian ships are "forks" since they have a few patches applied to customize for their environment and fix a few issues that hadn't made it upstream by the time they picked a stable release to base on.
MariaDB is an active, independently developed project. They are probably funding MariaDB because working with upstream developers of an actively developed fork can be more efficient than striking out on their own with another actual fork. I wouldn't be surprised if they eventually start using MariaDB, and either rebase their AliSQL patches on top of it or drop AliSQL altogether (much of it just looks like backports of bugs or features).
> However having something available as part of DB or Hadoop ecosystem enables reproducible execution while taking care of compliance (e.g. privacy), security, access control etc.
You are conflating a couple of things here.
Of course for real analyses, you want to run them on standardized environments with appropriate access controls.
But nothing says that "standardized environment" has to use a distributed database that, from all accounts, seems to harm more than help.
You could just have a RDBMS or even filesystem, which is appropriately backed up or replicated, on a single beefy VM or physical machine.
> The real opportunity lies in ability to add arbitrary compute/memory capacity to DB query execution flows.
Only if the tools used to provide that ability actually allow you to take advantage of it. If scaling up to 160 nodes doesn't allow you to be any faster than 1 single laptop, you probably could have spent a lot less money on just adding more RAM to a single beefy server than a 160 node cluster.
You run into different, but similar, issues cloning Linux on macOS as well, since HFS+ is case-insensitive by default (and looks like the same is true of APFS), but the Linux source has filenames that differ only in case (don't ask me why).
Note that the author himself is working on a framework for distributed differential dataflow computations (dataflow computations in which it is very cheap to make small updates to the inputs and see the differences in the outputs), and he has plans to apply the same metrics (comparing to a single-threaded implementation) to his own work once he gets a little further:
I think that answering the "fastest way to do X in a cluster" question is only particularly interesting if X is something that you need to do in a cluster. But clearly they are testing X that is capable of being done on a mid-range laptop. There's a real question of whether the X that they are testing will actually scale to larger datasets than the laptop (or a decently beefy single node) could handle; it's entirely possible that the cluster solution will blow up at that point. And of course, if the single-threaded implementation is faster, then you could presumably just do "X in a cluster" by doing "X on a single node" and using the rest of the nodes to mine your CPU bound cryptocurrency of choice.
> And of course, if the single-threaded implementation is faster, then you could presumably just do "X in a cluster" by doing "X on a single node" and using the rest of the nodes to mine your CPU bound cryptocurrency of choice.
As the author hi,self points out, amusingly.
These numbers are for sure more interesting. SEED does better than any number I can actually achieve on my laptop. That demonstrates non-triviality, in that their system isn't just a laptop computing the answer and 156 cores mining bitcoin.
I really enjoyed the snark in this, and the series of blog posts he links in the beginning. I tend to be nice to a fault myself unless someone has shown themselves to not be acting in kind (and even then I confirm), but I admit I take quite a bit of voyeuristic pleasure in people being assholes in smart and interesting ways. Vicariously living and all that, etc...