Honestly, pip freeze includes the whole content of a venv site-packages, and the exact versions. For most projects, that's equivalent to all the dependancies recursively pins with peotry, although you don't have the clean pyproject-dev-prod/lock file separation.
So a huge number of cases can be handled with just that. It will be "reproducible" enought for a lot of people.
> although you don't have the clean pyproject-dev-prod/lock file separation
That's why I use "poetry export -f requirements.txt > requirements.txt" instead of pip freeze. It only exports prod requirements from the poettry lock file.
Most of the time, you don't need all that, since Python has zipapps. You defined deps, you zip it, you ship to any same os with the same python version. It embeds everything, and just run.
We even how have a nice tool to automate the bundling for you:
I agree, but it's still way easier than the original story, which is the one you also have with PHP, Ruby, JS, etc.
Using an interpretted language always leads to this.
I know no popular interpretted language with a seamless experience to ship a standalone exe.
In fact, Python is probably the one with the best story here, since it has nuitka (https://nuitka.net/), which allows to compile Python code into a fully standalone exe.
But then you need to install a compiler, headers, etc. And no cross compilation of course. Not to mention on Linux, you have to ensure you target the lowest version of libc you can.
You are still very far from Go or Rust, and I'm hoping one day that RustPython will succeed because that would mean an amazing deployment story.
Meanwhile, you trade the ease of deployment of compiled languages for the ease of development of interpretted ones.
I think it's a fare trade for most people: you dev the program much more often that you deploy it.
That doesn't mean we shouldn't work, as a community, to improve the deployment story. It's a serious hindrance.
Rust has a fantastic deployment story: compiling a rust program is super easy, and you can cross compile. Using cargo and rustc is a breath of fresh air compared to any similar experience with C compiling.
So if one day RustPython gets compatible enought with CPython that you can use it as a drop in replacement, you can start creating a tool that compiles any Python VM for any target, and bring along your program with it. Making a standalone version of it would become much easier.
Right now, doing so either requires you to bring in a pre-compile version of cpython for your target (which is what briefcase does) or compile the thing yourself with gcc + headers + deps(which is what nuitka does).
> So if one day RustPython gets compatible enought with CPython that you can use it as a drop in replacement
I don't think this will ever happen unless the community converges on a standard C-extension interface. Presently Python leans so hard on C-extensions, but there is no standard interface--if you're writing a C-extension library, you just depend on whatever obscure corner of CPython that suits your purpose. If you're writing an alternative Python interpreter, you have to implement the entire surface area of CPython, which generally means you must implement CPython exactly and you are severely restricted on the improvements you can make. At that point, why even bother?
Fortunately, I think there are emerging candidate interfaces, but the community needs to either update C-extension packages to use those interfaces or support packages (and maintainers) who already do. https://github.com/pyhandle/hpy.
There are probably only a dozen of c popular extensions that needs to support HPY reach the tipping point of mass adoption: numpy, scipy, pycuda, tensorflow, matplotlib, uvloop, etc. and some db drivers.
The rest is not popular enought to be a blocker. You will hear them scream a lot, but they will be like 0.00001% of the user base, and we can just tell them to stay on CPython with its limitations. They don't lose anything, just not gain anything either.
Those C extensions authors are directly in communication with Python core devs, when they are not core devs themselves, so if HPY is adopted, we can expect a total adoption under 5 years.
Numpy authors already said it would take 1 year to adopt it.
Give the huge number of benefits of HPY, I deeply hope it will be a success.
I'm not sure. I would certainly add psycopg2 to that list, since it's really the only well-supported way to speak to a Postgres database via Python. I imagine other database dialects will have similar issues. And there's probably a whole host of other prominent libraries that we're just not thinking about because we only run into them when we're trying to use something like Pypy, and even then we only run into one or two at a time before giving up and going back to CPython.
youtube-dl for example is distributed as a zipapp and it seems to be distributed just fine. It only requires you to have Python installed on your system, which isn't too burdensome of a requirement on macOS/Linux. On Windows they do actually distribute a Python interpreter.
As usual with extensions, you are not using Python anymore, but a compiled language. To get 100% certainty, you'd need to compile the whole thing.
That being said, a lot of extensions are pre-compiled and provided as wheel, which is the case for tensorflow (I don't know for CUDA, I can't test on a laptop without a GPU).
Let's see what this means:
$ py -m venv test
$ test\Scripts\activate
$ pip install tensorflow
$ code hello_tensor.py
# import tensorflow as tf
# def main():
# with tf.compat.v1.Session() as sess:
# a = tf.constant(3.0)
# b = tf.constant(4.0)
# c = a+b
# print(sess.run(c))
- it will only run on the system this particular wheel has been designed to run on. In my case cp38-win_amd64.
- it will come bundled with tensorflow, which is a behemot, meaning your hello world pyz will around 500 Mo.
- it needs to unzip, so the first run will be REALLY slow
For something like this, I would advice a more generic deployment tool, like fabric 2 if it's remote, or a make-like tool such as doit if it's local only.
Zipapps are an order of magnitude improvement in the Python world, but there are still lots of other major pain points like dependency management and performance which still leaves Python several orders of magnitude behind its competition. Hopefully these things change going forward.
It looks like zipapps built with “shiv” need to extract the contents of the zip file to disk before they can run? Does it delete the extracted files on exit?
If so, the extraction is going to make startup very slow. If not, that’s just messy. Either way, it’s not ideal.
But it beats shipping your entire dev env to the server.
I find it a good compromise. The extraction is done is $HOME/.shiv/{zipappname}_{zipapphash} so it's not a horrible mess. But if your project is big, you do have to clean up the old install because it can eat a significant amount of space.
Those tools have been existing for years, and as always with Python, they serve only a subset of the Python users.
Not that it's a bad article. If you don't know about them, give it a read, it's good to know it exists and the post is well written.
But don't think it's the ultimate stack or whatever.
And if you learn Python, you should always strive to master the basics first:
- installing python.org for windows and mac, or apt/yum on Linux (those ones are tricky, it's more than just the "python" package to get pip and venv)
- don't try to get the latest and greatest version (which is 3.8, or 3.9 alpha today). 3.6 is great already and is wildly available. I personnally strive for 3.7 now, but I'm happy with 3.6 if I don't have to use asyncio. So get the most modern version that is easy to install for you.
- be comfortable with "-m", the "py" command on windows, "pip" (-m, --user, install, install -r, freeze), and "-m venv".
- make sure you understand what the PYTHON PATH is and how it works regarding the import system
- know how to configure your favorite editor to work with those
Once you have those basics, you can go and try whatever have you: peotry, pyenv, pew, virtualenvwrapper, pipenv and so on. It will be easy because you will have a solid understanding of where it comes from and how it plugs in into the ecosystem. You will be able to chose if they are worth it or not for you. And more importantly, when you will have to move away from them (because of work, because they don't suit you, because they are abandonned...), you will always be able to fall back to basics.
But franckly, there are so many things that you should be learning before those extras: pdb, black, pylint, zipapps... Or even concepts like generators, decorators, unpacking, etc. They all have a better value for the time you spend on them than learning a new "modern" stack.
Not that it's not useful to have a modern stack. I update my stack all the time.
I'm currently using pip + pew + doit + nox + pytest, and I'm experimenting with dephell.
But it's my full time job. I'm an expert.
Most Python devs have a much limited amount of resource to spent in learning stuff. They have deadlines, and other stuff to care about than Python.
I know because I go from company to administrations to students to train them, and it's always the same: they chose Python because it's an efficient use of their time.
Context is important. Developing applications vs libraries vs scripts is different, developing internal company software vs external software is different.
If you're shipping a package on pypi, you will want to develop and automatically test using multiple Python versions and maybe a range of versions of your dependencies. Having good tools to manage multiple sets of Python versions and dependencies and parallel automated testing is a godsend.
If you're writing an internal or web application that will run only in a single well-known environment, pip + venv + pytest might well be good enough.
Sure, but again, there is nothing modern about this. And certainly not, hyper modern.
Besides, if you want to develop and automatically test using multiple Python versions and a range of versions of your dependencies, none of the tools from the article will do it.
At best they are one of the ways to get dependancies before you use a tool to do it. You can do that with raw pip and venv as well.
What you need though, is something like tox, which has existed for years.
As mentioned in my comment, I personnally use nox for this, as it is, ironically, more modern to my taste.
Nox doesn't care if you use poetry, pew, virtualenv wrapper or venv manually.
I would certainly advice to learn something like nox or doit before learning poetry, for example.
Now I don't advice against learning poetry, mind you. It's a good tool, well written.
Pyenv is another matter entirely. The day one really need what pyenv gives you to the point it's worth investing in it, they can discard all my advices entirely, as they would have the skill and experience to not need my explanations.
Not sure if you've read just the first article, it's a whole series, they cover nox later on.
I don't see how this stack is "hypermodern" either, mind you.
My point was just that these articles and other like it tend to omit context completely. What is this stack good for? Who should use it? Who shouldn't? Without answering these questions the articles are not all that useful, or even could be harmful, if a dev who doesn't know better apes a complicated stack for no good reason.
If you see "quantum" in a general public health product, you know it's going to be a scam, but probably a successful one.
My mother loves those kind of things and no amount of discussion with her will create any doubt in her mind.
I can't blame people falling for this though. They have been disrespected by the traditional health institutions for so long any escape seems good to them.
You can see the same effect in politics. To an extend, the current authorities have a lot of responsability in people listening to crooks.
If the app is free, I would just politely ask such users to eff off. On the other hand I am probably as far away from iPad-damaged demographic that you can come: Emacs isn't enough of an OS, I have to leave it a couple of times a day and I hate every single minute of it.
Everything I do on a computer, apart from the odd movie I watch, is centered around information in text form, yet literally every program except decent text environments try their best to ruin working with text. The only place where I think a non-text-based ui actually adds something is in calendar applications, yet most of the suck as well.
So I sit here alone with the marvel that is 1980s technology, thinking most things regarded as "modern" in computing are a farce.
The only new general purpose tool I can stand is VSCode, where my o my critique is that it isn't emacs (and that multiple cursors are a decent but not a great replacement for macros)
Git is not for kids, it should be learned only by people that will need to code regularly in the future.
But, if do need to learn git, it's not hard at all.
The problem is that most people teaching git suck at it.
Git has been poorly explained since the begining of time, and because it has a terrible UI, plenty of gotchas and untold prerequisite, you can't really fiddle with it and learn it by yourself without putting a lot of efforts.
It's a shame, because it really is a simple system to use... provided you understand it, which a lot of teachers don't.
I use a toy for 5 years old, that let you construct a 3D structure with magnets, to represent the history, and I stick colored post its on it to represent references.
Important points:
- git log and checkout must be explained in details, as they are a huge source of confusion.
- don't use the famous graphs with revert arrows, they confuse the heck of students
- explain push/pull/clone as late as you can. Most of the course should be on a local repo.
- help them setup diff tools for their particular env. It's harder than you think given the diversity of them.
- you will often have to explain ssh and key management as well. It's not git, but git is close to useless without it. For windows users, you may have to spend an hour of terminal tutorial, putty or not. Use cmder or windows terminal, not cmd.exe.
- do give them a cheat sheet, but only incrementally once they understand the concepts behind them, otherwise they will copy/paste blindly.
- teach them to have a clean working copy before some key operations, and show them why.
- make a big final exercice where they all work on the same code base and push/pull/merge from each others. Twice. Once where you help them. Once to check they can do it now without your help.
- rebase, squash and cherry pick are optional. Do them only if you have plenty of time at the end, and that the rest is well understood.
So a huge number of cases can be handled with just that. It will be "reproducible" enought for a lot of people.