I'm the first one to hate on JS, but the whole integrated event loop, with auto scheduled async tasks and Promise system that works both with callbacks or async is one of the easiest system for http calls I ever used.
Concurrency has a lot to do with sharing one resource, and Python has dedicated tools to deal with that depending on the resource you must share.
If you have to share one CPU while waiting on the network, then the specialized tools for this are asyncio, twisted, trio, gevent, etc.
Asyncio is the current standard to do this, but tornado, gevent and twisted solved this problem more than a decade ago. While trio and curio are showing us what the future could look like.
> If you need to get a few URLS fast, a ThreadPoolExecutor is likely the Pareto solution, 99% of the time.
I agree with this. ThreadPoolExecutor is easy to use. In general, using threads in Python to handle concurrent I/O is pretty simple and plenty performant.
It’s a waste if you need coordination between the tasks though. Putting an async task aside and sending only dumb synchronous function calls that never wait is more efficient than blocking a thread while it waits on another.
asyncio with an executor is a good way to run complex tasks in parallel.
> So, no event-oriented programming in Python, or by some other mechanism, or…?
My guess he's saying that anything that introduces concurrency is hard, and best avoided if you can do so. He's not wrong. True event driven code (without these libraries) introduces a lot of unneeded concurrency. You see it at it's worst in older UI's. There a central event loop receives mouse, keyboard and io events from the OS. The event handlers it calls transforms those events into higher level events like "field changed" and "focus change", and re-injects them into the event loop, where they get turned into still higher level events like "modal dialogue closed" until the job is done. It's a prick of a way to write a program.
If you just want parallelism (as he defines it in the article), threads with their conventional stack model are far easier to use. The problem with conventional threads is that you also get unwanted concurrency and the heisenbugs that go along with that. But green threads (aka fibres, aka cooperative multitasking) eliminate that sort of non-determinism. Green threads are what effectively gevent provides. That's why he's so positive about it.
The other libraries he discusses aren't about event oriented programming either. Rather, they tame the event loop, allowing you to write code in a way that looks similar to threaded code. And they do it without introducing unwanted non-determinism. But they come at the cost of coloured methods, so you have 2 ways of doing everything. That is why asyncio has no ftp / smtp libraries - because they have to be written in the different colour. With green threads (and gevent) all that old code continues to work. That is why these libraries suck compared to green threads. But if you have a language like javascript that doesn't have threads, they are a huge leap forward over the raw event loop processing so you'd take it any day.
What has me scratching my head is why Python introduced these higher level event libraries at all. They own the language - they could have gone the green thread route. As for Rust introducing colored code - words fail me.
They do half of what profiles do: they isolate cache, cookies, sessions, etc. But they do so in a very light manner and fast manner. The UI is better too. So for things that don't need getting different settings or extensions and so on, containers are the way to go.
The article is well written, and a good history of the all ordeal. But please note it gives more weight to the "against the GIL" side of the story, and all that can go wrong.
It doesn't highlight enough the other side of the coin:
- Sam's work is very high quality, and he brought with the no-gil some unrelated perf improvements so that people don't feel like they loose perf too much.
- Sam played the open source game perfectly, and is incredibly patient given what he is bringing on the able and how slow and flaccid the steering council reaction was (without the community pushing on it, it would still be collecting dust).
- Sub interpreters have yet to demonstrate any usefulness at all in Python. In fact, any serious metrics at all. This is the first attempt to be that well defined, and measured.
- The community feedback shown a great interest in this particular project.
- The steering council did conclude "We intend to accept PEP 703, although we’re still working on the acceptance details."
I'm not a no-gil enthusiast. I would be fine with have it never be removed, and I think we should try sub-interpreter first.
For now there is nothing impressive with it. The perfs they show off don't include data sharing, which is very primitive.
Which means there is nothing subinterpreters do that can't be done with multiprocessing.
They show progress, and it's good. But I would wait until we can see gunicorn spawning WSGI interpreters workers and get similar performances to the setup with regular workers to get enthusiastic.
to further your point: I've used the technique launching a separate python process for performance critical operations for years on different projects. I can run it in a completely different priority set as well.
I have a use case for sandboxing (without a secure-sandboxing requirement) that it would help solve. But at last read, they discourage/disallow sharing modules across the subinterpreters and reloading all modules in each subinterpreter is not acceptable for my use case.
They were always usable using the C-API but not exposed to the Python library (and until 3.12 you were blocked by the shared GIL so there was little point anyway).
I'd be interested in introducing appropriate abstractions (ABI) and in a module for parallelism. Whereby one can swap out different implementations for sub-interpreters, green threads, GIL or even hardware or cloud vendor-optimised implementations. Not that dissimilar to Project Loom in Java.
PEP582 - “Python local packages directory” is my personal bugbear when it comes to Python annoyance:
https://peps.python.org/pep-0582/
I believe PDM still supports it. I just hate virtualenvs for builds and deployments and wish Python could just do what JS does. It’s been proven that it can. https://discuss.python.org/t/pep-582-python-local-packages-d... Is the discussion thread. Also frustrating that “exactly one correct way to do something” seems to be one of the justifications thrown around for rejecting this when I’ve never found that to be true in Python.
> They use unmaintained C extensions that won't be updated.
Well, that's on them.
> They're concerned about fragmenting the ecosystem into GIL and No GIL.
Same way we have fragmentated ecosystem into ASYNC vs SYNC now? /s
> They think that it will make single-threaded Python programs slower.
Python is already horribly slow. A little bit slower single thread programs shouldn't be a problem. But the inability to run multi-threaded programs effectively in this day and age is very much a problem.
I haven't fully confirmed this but my understanding is the proposal forks the ABI into two versions. which IIUC means any package with native extensions needs to release 2x as many wheel files (not to mention if I need to have two versions of the interpreter installed to build them, not sure though). seems like a mess
