The fact that tensorflow takes up 12.9TiB is truly horrifying, and most of that because they use pypi's storage as a dumping ground for their pre-release packages. What a nightmare they've put on other people's shoulders.
I think pypi should require larger packages, like tensorflow, to self-host their releases.
There is all support for that already - the pypi index file contains arbitrary URL for data file and a sha256 hash. Let pypi store the hashes, so there is no shenanigans with versions being secretly overridden, but point the actual data URLs to other servers.
(There must obviously be a balance for availability vs pypi's cost, so maybe pypi hosts only smaller files, and larger files must be self-hosted? Or pypi hosts "major releases" while pre-releases are self-hosted? And there should be manual exceptions for "projects with funding from huge corporations" and "super popular projects from solo developers"...)
I believe tensorflow does remove old pre-releases (I know other projects do), so that number I think might be fairly static?
That tensorflow is that big isn't surprising, given the install of it plus its dependencies is many gigabytes (you can see the compressed sizes of wheels on the release pages e.g. https://pypi.org/project/tensorflow/#files), and the "tensorflow" package (as opposed to the affiliated packages) based on https://py-code.org/stats is 965.7 GiB, which really only includes a relatively small number of pre-releases.
Why tenserflow is that big comes down to needing to support many different kinds of GPUs with different ecosystem versions, and I suspect the build time of them with zig cc (assuming it works, and doesn't instead require pulling in a different compiler/toolchain) would be so excessive (especially on IoT/weaker devices) that it would make the point of the exercise moot.
Is it though? If it saves one engineer one afternoon that storage has paid for itself, and this thing has hundreds of thousands of downloads a day.
Wouldn’t it be more horrifying to force everybody who wants to use a prerelease to waste an afternoon getting it to build just to save half a hard drive?
That's besides the point though. Yes, having prebuilt binaries is very helpful. But what happens if Fastly decides against renewing next time and there is nobody else willing to sponsor? The cost is through the sky for the PSF to handle. Where does PyPI go?
The issue there is how much gets downloaded from PyPI, not how much storage space it takes up. Making an archive copy of all of PyPI would only require a handful of ordinary drives. But the most popular packages get on the order of ten million daily downloads each.
When someone downloads and installs a package, it's generally a single wheel or sdist, out of a potentially massive version/platform matrix for that project. That inflates the storage cost, but it isn't why the bandwidth requirements are so high. A ton of CI systems are apparently poorly designed, individual wheels are bloated, and we can't use the best available compression. Those are the biggest issues.
The open-source version has no pricing. The base pricing is the cloud version. This is nothing like the other names in the SSO tax list. The whole point is exclusion of SSO in the lesser of their two paid offerings, not OSS vs paid.
There's a lot more checks and balances built into rust as a language, so the foot-gun opportunities are less likely - which makes it rather suitable for reliable systems programming.
And getting back to this particular case... kanidm is fast AND reliable. There's a lot of testing going on comparing it to 389 DS.
Yes, I write a fair bit of Rust. I'm just saying: between Python and Rust, there isn't that much of a security difference (you could nitpick things like deserialization in Python, but really the significant security win of Rust is not having memory corruption flaws, which Python has never really had.)
