Python is a really bad example of cold blooded software. There is constant breaking changes with it (both runtime and tooling). So much so that the author still has to use python2 which has been EOL'd for quite a while.
A much better example would be something like go or java where 10 year old code still runs fine with their modern tooling. Or an even better example, perl, where 30 year old code still runs fine to this day
As an author of software, sometimes you make mistakes, and those mistakes are often of the form, "I permitted the user to do something which I didn't intend." How do you correct something like that? In the Java world, the answer is "add newer & safer & more intentional capabilities, and encourage the user to migrate."
In the Python world, this answer is the same, but it also goes further to add, "... and turn off the old capability, SOON," which is something that Java doesn't do. In the Java world, you continue to support the old thing, basically forever, or until you have no other choice. See, for example, the equals method of java.net.URL: known to be broken, strongly discouraged, but still supported after 20+ years.
Here's an example of the difference which I'm talking about: Python Airflow has an operator which does nothing -- an empty operator. Up through a certain version, they supported calling this the DummyOperator, after an ordinary definition for "dummy." But also -- the word "dummy" has been used, historically & in certain cultures & situations, as a slur. So the Airflow maintainers said, "that's it! No more users of our software are permitted to call their operators DummyOperator -- they now must call it EmptyOperator instead!" So if you tried to upgrade, you would get an error at the time your code was loaded, until you renamed your references.
This decision has its own internal consistency, I suppose. (I personally would not break my users in this way). But in the Java world it wouldn't even be a question -- you'd support the thing until you couldn't. If the breakage in question is basically just a renaming, well, the one thing computers are good at is text substitution.
So overall & in my opinion anyway, yes, it's very much true that you can upgrade Java & Java library dependencies much more freely than you can do the same sorts of things with Python.
> So the Airflow maintainers said, "that's it! No more users of our software are permitted to call their operators DummyOperator -- they now must call it EmptyOperator instead!"
Man, some companies and people have far too much time to waste.
Not to detract from your point (which I agree with), but rather as a side note, Airflow's developers publish top-notch migration and upgrade documentation and tools which hold your hand through the process of updating your DAGs when upgrading Airflow. Which IMO is the next best thing to do when you break backwards compatibility.
English isn't my first language but I haven't seen "dummy" being used as a slur, in any conversations I've engaged in or any books I've read. For me its connotation is more of a playful nature. When I think of slur I don't think of "dummy", I think of the r word and the like.
At least I can get the reasons for github's change to "main" for the default branch in a git repo. Maybe I don't agree with it but I can at least see how some people would interpet the word "master" in a negative way. I can't say the same for the word "dummy" though.
Yes your understanding is how almost everyone treats the word today. The slur is from a different meaning & different context. You'd never come across it in regular life, unless you were, like, into the history of baseball:
Maven is fantastic. As long as you stick to an LTS Java version and pick good dependencies you can always get things up and running. With Python I remember a ML class I took where one of the dependencies had introduced breaking API changes overnight and the lecturer hadn’t noticed because he was just using whatever version was available a few weeks ago when he first started prepping for the class.
I assumed you could specify versions in python. It’s not the first time I’ve run into issues building other people’s python projects, so I hope someone makes it a requirement to specify a version number when adding dependencies.
And I really wish ML and AI was as popular in Java as it is with Python, but I don’t see that happening anytime soon :(
> There is constant breaking changes with it (both runtime and tooling).
I'm not sure what you mean. Python 2 to 3 was a breaking change, but that was just one change, not "constant breaking changes".
If you stick with one major version no old code breaks with a new minor version (e.g., you can run old 2.x code under 2.7 just fine, and you can run old 3.x code under 3.12 just fine). The minor version changes can add new features that your old code won't make use of (for example, old 3.x code won't use the "async" keyword or type annotations), but that doesn't make the old code break.
The Python 3.11 release notes have a pretty lengthy list of removed or changed Python and C APIs followed by guidance on porting to 3.11, which implies potentially breaking changes to me.
It's a fair point that Python minor version changes can and do involve removal of previously deprecated APIs, which would break old code that used those APIs.
That said, when I look through the 3.11 release notes you refer to, I see basically three categories of such changes:
- Items that were deprecated very early in Python 3 development (3.2, for example). Since 3.3 was the first basically usable Python 3 version, I doubt there is much legacy Python 3 code that will be broken by these changes.
- Items related to early versions of new APIs introduced in Python 3 (for example, deprecating early versions of the async APIs now that async development has settled on later ones that were found to work better). These sorts of breaking changes can be avoided by not using APIs that are basically experimental and are declared to be so (as the early async APIs were).
- Items related to supporting old OSs or data formats that nobody really uses any more.
So, while these are, strictly speaking, breaking changes, I still don't think that "constant breaking changes" is a good description of the general state of Python development.
Python's changes between releases are not limited to removing deprecated APIs. Sometimes semantics changes in breaking ways, or new reserved words crop up, etc. etc. It certainly is Russian roulette trying to run python code on any version other than the one it was written for.
I know this specific example because it was one of a handful of reasons that delayed my workplace from upgrading past Python 3.5 (I think) for quite a while because of the addition of the reserved word `async`.
For me switching to Python 3.11 was really tough because of various legacy stuff removals (like coroutine decorators etc). While my code did not use these, the dependencies did. For some dependencies I had to switch to different libraries altogether - and that required rewriting my code to work with them.
There was also some time in the past when async became a keyword. It turned out many packages had variables named async and that caused quite a bit of pain too.
The problem is `requirements.txt` doesn't do anything with downstream dependencies. There's nothing like a shinkwrap/lockfile in python. Even if you pin dependencies to exact versions, if you check your project out in a new environment and run pip install -r requirements.txt, you can end up with different, broken downstream dependencies.
If you want to stick with using `pip` over any of the newer tools that build on top of it (Poetry - my favourite, pdm, pipenv, rye, ...) the simplest way I used in the past was to use a `requirements.human.txt` to set my dependencies, then install them in a venv and do `pip freeze > requirements.txt` to lock all of the transitive dependencies.
That's an awareness problem. requirements.txt was invented... a long time ago, I think before the much more sane (but still not perfect) dependencies/lockfile split got popular. requirements.txt tries to be both - and it can be both, just not at the same time.
In short, you want your deployed software to use pip freeze > requirements.txt and libraries to only specify dependencies with minimal version conditions.
I did not know about pip freeze, doh. Thanks will check that out!
Edit: so if I understand it, this is just listing all packages in the current python env and writing them to a file. Hm, requires more discipline than the npm equivalent. But thats a natural consequence of pip defaulting to installing packages globally (vs npm which installs in local node_modules by default). Better but still not awesome IMO
Why would you bluntly assume my comment lacks any foresight? I was simply recommending you a tool that I used, albeit briefly, that solves the exact the same problem for which you are claiming no solution exists.
Nobody is denying that it would be ideal if there is one best solution to every problem in the ecosystem. But at the end of the day all software, including core and third party libs is just code written by people, and it is too much to expect that any person (or a group of them) gets everything right the first time. Change, breaking or otherwise, is inevitable as people learn from their mistakes - its not like the core is guaranteed to never have any breaking changes either.
Just like you can pin the version of libraries, you can pin the versions of your tools too, as long as they are not depending on external services with no versioning. The point of the post is not absolute avoidance of change. It is to opt into a workflow and tooling setup so you can deal with the upstream changes at your own time and convenience.
And BTW, looking at their versioning, poetry hasn't yet had any breaking changes in its 4+ years of existence.
That said, I remember all three of those transitions (2.3 to 2.4, 2.4 to 2.5, and 2.5 to 2.6), and I remember changing Python code to make use of new features introduced in those transitions (for example, using with statements and context managers in 2.5), but those aren't breaking changes; the old code still worked, it just wasn't as robust as using the new features.
Something called onnx (all iirc) requires Python 3.8-3.9 but not 3.10+ in which it doesn’t work. So for my various AI needs I have three versions of Python 3 installed through different channels. And of course they all have their own multi-gigabyte caches of base libraries and models.
I know it may be more complex or trivial than I think, or tied to very few specific packages, but that’s the point – I have to figure it out where I shouldn’t need to. In contrast, I’m sure that no matter which latest version of Node I have, it will work.
I mean I was, up until Node 19/20, where they broke the loader, so ts-node doesn’t work anymore and the suggestion is to re-learn something called tsx. F that nonsense.
Agreed. This is one of the reasons why I avoid using Python whenever possible. Python code I write today is unlikely to be functional years from now, and I consider that a pretty huge problem.
This really depends on your environment. I've been running legacy Python servers continuously for 4+ years without breaking them or extensively modifying them because I invested in the environment and tooling around it (which I would do for any app I deploy). I can't say I want to bring all of them entirely up to date with dependencies, but they're still perfectly functional. Python is pretty great, honestly. I rarely need to venture into anything else (for the kind of work I do).
> I've been running legacy Python servers continuously for 4+ years
That seems like a large amount of effort to make up for a large language deficiency. My (heartfelt) kudos to you!
I might have been willing to do the same if I used Python heavily (I don't because there are a number of other things that makes it very much "not for me") -- but it would still represent effort that shouldn't need to be engaged in.
I think it depends on which bits of the Python ecosystem you're interacting with. The numerical/scientific parts have been quite stable for at least the past 10 years (new features have been added, but only small amounts of removal), compared with the more "AI" focused parts where I wouldn't trust the code to be working in 6 months. Similarly, some web frameworks are more stable than others. I think also over the last 5 or so years, there's been a change in maintainers of some larger projects, and the new maintainers have introduced more breaking changes than their predecessors.
None of this is implied by the language, I think it's much more driven by culture (though I think the final dropping of support for Python 2 did give some maintainers an excuse to do more breaking changes than was maybe required).
I'm not following. I've put in a total of a couple hours of maintenance over four years for the entire app stack. I think the maintenance issues and processes I use for Python would be the same as any other language. I remember my Java experience a decade back being essentially the same, the JS apps I am responsible have perpetual churn, and my .NET friends have said they feel behind if they're not keeping up with core changes every 6-12 months. Every asset, whether physical or digital, needs regular maintenance. What is different in your experience?
There are plenty of other removals, deprecated stuff like Thread.stop(), security providers, JDBC interfaces changes, finalizers no longer do anything (better not depend on them ever running) and might be removed in some future version.
Java does not have perfect backwards compatibility, but it's pretty good.
Thread.stop was deprecated very long time ago. It was deprecated at least in Java 6 (2006 year) and I'm too lazy to check for earlier versions.
Security providers have very limited usefulness for ordinary applications. And AFAIK it's not even removed for Java 21 yet, so formally old apps should work for now.
I'm not aware of incompatible JDBC interface changes and I used to use very old Oracle driver without that much problem. Yes, there are some new methods in JDBC interfaces which are not implemented with old drivers, but you can just not call those methods. It's not breaking change.
Finalizers were never reliable. But you're not correct about them not running right now, I just checked and with Java 21 they're called.
So while there are breaking changes, they're always some application issues. If you call Thread.stop, you should not do that, that's bad application. If you're using security providers, you should not do that, they're not safe enough, you should use containers or VM or other means of code isolation. I'm not completely sure about JDBC issues, but I had experience running very old JDBC driver and it works just fine. If you rely on finalizer, you should not do that, that's wrong approach to resource management and it was always wrong.
I maintain Eclipse RDF4J and I noticed this too between Java 9 and 11, after that there haven’t been any breaking things except for having to bump a maven plugin dependency. We make sure to stay compatible with both Java 11 and whatever is the newest version by running our CI on both Java versions.
All the stuff that got removed since Java 9, as the new policy is to actully remove deprecations after a couple of warning releases, instead of supporting them forever.
Additionally, being more strict regarding internal APIs security, not allowing access to JVM internals by naughty 3rd party libraries.
Regarding Python: Really? Obviously v2-to-v3 was an absolute fiasco, but since then, it's been great in my personal experience.
Don't get me wrong: Python hasn't overcome its tooling problem, so there's still that barrier. But once your team agrees on a standardized tool set, you should be able to coast A-OK.
Every time there's a 0.1 Python version increase, it takes months for other libraries to catch up. I still have to install conda envs with Python=3.9 because they are warm-blooded software.
Go is not a good example either. Some times ago we tried compiling a code a few years after it was made, it did not work. Someone who actually knew the language and tooling tried and said there was a migration to be done and it was complicated. I have not followed the subject up close but in the end they just abandoned IIRC.
I don't think I've ever had that problem -- particularly once they introduced Go modules, which specified a specific version of a library dependency. My experience is like the author's: Even old random packages I wrote 5 years ago continue to Just Work when used by new code.
There are a handful of functions that they've deprecated, which will produce warnings from `go vet`; but that doesn't stop `go build` from producing a usable binary.
Depends if you mean python the interpreter or python the language. e.g. pypy still supports python2 and has "indefinite support" or something along those lines.
Even the cpython2 interpreter is no longer supported by the original authors, but that doesn't stop someone else from supporting it.
The worst example perhaps. I have the unfortunate honor to work on our python projects from time to time, but rarely and every time that I do, something is broken. No other software is as unreliable. Only Ruby comes close and probably for the same reason.
I've had good luck with sveltekit (a framework for js sites). They'll break something with a new version but provide you with very helpful compile errors pointing to a migration script to re-write any old code.
C# has been pretty good as well.
But at some point you're going to need data for your app and that's where you'll get surprised. That Yahoo currency data you used to get for free or Wikipedia's mobile API? Gone ten years later.
What? You can just update EF Core without ever having to do a migration of the schema. It just works. Also, the versions that are EoL today are a really poor choice for Lambda anyway because you really do want to be using Native AOT + Dapper AOT with it instead.
Note that since Java 9, this isn't exactly true, after modules, removal of private apis being misused by packages and effectly removing deprecated APIs instead of having it forever on the platform.
I have yet to have a python versioning issue, but with java I've had tons.
Worst of all, it's always a clear "use the latest version and it will work". With python using the latest version almost always works, and you can import the previous functions if you really want to use the new interpreter on old code.
Maybe this is because most of the time with python you barely have external libraries. Similar to Java, but in Node.js it's like asking for trouble.
A much better example would be something like go or java where 10 year old code still runs fine with their modern tooling. Or an even better example, perl, where 30 year old code still runs fine to this day