That's by design, border control was a selling point of Brexit, as brexiters saw only the bad side of immigration. Which to be fair, is no seamless nor without challenges or cost.
I suppose the rational is:
• Diminish the friction that multi-culturalism and diversity integration can bring.
• Free jobs for the local population.
• Having to share less of lands, resources, and infrastructures.
And I see the point, but it also fails to take in consideration that:
• The UK needs the import/export flow it was used to pre-brexit to keep their current life style. They don't produce much good internally, and they need to export their service. Brexit added huge barriers to that, not just for people.
• The current British population doesn't seem very motivated to take back the jobs they delegated to immigration.
• Even if it were, it can't be instant, since that requires training an entire workforce and sometimes even require permits or certification. Trucks, bus and cabs come to mind. And the inertia will do some damage. The supply chain is already seen glitches, although not as much as I would have expected. They are surprisingly pretty resilient.
• The Northern Ireland situation is a ticking bomb. Yeah, immigrates are not just brown people. People have a short memory though, so I suppose IRA stands for nothing nowadays. They may get a nasty reminder that ignored history is doomed to be repeated.
• Scotland nationalism has been exacerbated since they were pretty anti-brexit. Also, as an immigrant, scots were pretty welcoming, they need people there.
• Big players that needed to stay in Europe are sometimes moving to France or Ireland. So the promised jobs have to take that in consideration.
• The UK has an aging population, with a 0.53% growth rate, and that was when compensated by immigration.
• The UK education system sucks. People think Oxford and Cambridge, but the reality is more Leeds Metropolitan. I've studied there, and it's not going to be easy to find qualified workers only in the island.
• The UK managed to negotiate a very privileged deal inside the UE. It had most of the advantages, and way less of the down sides that other members. This deal is dead now, they have the worst of all worlds. So even if they get some benefits, the cost may be huge.
• France and the UK never really liked each other. The UE brings peace. But now, all bets are off. Also, no more bonding program such as Erasmus to bring future generation together. And good luck to make you BSC valued outside of the UK. And I guarantee the already little patience we had for the retired brits in the South of France that never try to learn proper France will not hold forever now that we are not bros.
• All that in the middle of covid. Not their fault, but damn, that's a terrible timing.
A couple of counterpoints (offered in good faith):
> - the UK needs the import/export flow it was used to pre-brexit to keep their current life style.
That is about goods, which are totally separate to immigration.
> - the current british population doesn't seem very motivation to take back the job they delegated to immigration.
Motivation has nothing to do with it. The employers offering these shitty jobs have gotten used to being able to pay poor wages to cheap imported labour. They need to pay more, and the jobs will be filled. Look at the wage growth of truck drivers as an example.
> - the UK education system sucks.
Disagree here. The UK punches way above its weight in terms of high end education institutions per capita (trailing only the US) The UK also has more STEM grads per capita than any European country.
I'll upvote you because I love people being civil on HN.
> That is about goods, which are totally separate to immigration.
Yes, but you need trucks, boats, and loaders for those. A lot of the people dealing with that are immigrants.
> The employers offering these shitty jobs have gotten used to being able to pay poor wages to cheap imported labour.
Working in agriculture as always been low paid back breaking work. There were no "good old time". We are just not used anymore to have a part of our own population is been used as semi-slave and shutting up.
Not saying it's a bad thing, we should strive to make for better working conditions. But those jobs still need to be done, and we don't have android doing them for now.
It could go the Australian way, and make those jobs decently paid, but even there, immigrates still do those jobs! Not to mention their economy is sky rocketing so they can afford it.
> The UK punches way above its weight in terms of high end education institutions per capita (trailing only the US) The UK also has more STEM grads per capita than any European country.
The number are high yes, but the level of graduates is terrible. In fact, I got my diploma despite not handing out my final memoir or reaching the minimal passing grades. I spent a year drunk, and got a paper stating I studied hard and was worthy of my profession.
And damn, most of my friends were even worst than me.
Papers were a joke. Nothing was reproducible, arguments were barely holding together. Not to say my mates in French universities were geniuses, but the difference is glaring.
I guess I would then argue that your points one and two are the same, and therefore have a similar fix - that the lack of labour imports will impact standard of living (either through logistics, farming, or other unskilled work)
We need to pay more people to do these jobs, not rely on exploiting people from poorer nations to fill the gaps. I think the evidence suggests this is indeed happening. Wages are rising at their fastest rate on record.
I think your suggestion of applying the Australian model to filling those vacancies is also a good one, but most of the people filling those jobs are tourists on extended holidays, not people looking to build a life in Australia. Working holiday visas require that you must work in service or agriculture, and the will kick you out when your time is up (I'm Australian so I've seen it happen to a number of my friends).
I guess I can't disagree with your personal experiences at University, but I do suspect a lot of it comes down to the institution you attended. I studied at two universities in the UK and found them both academically rigourous.
> We need to pay more people to do these jobs, not rely on exploiting people from poorer nations to fill the gaps.
Agreed, but giving our consumption rate, it's going to be hard to do that and be competitive with a global market. Australia can pull that off because they are swimming in wealth right now, have a lot of land, and also because they have very favorable import rules they can maintain because the country produces a lot of basic goods. I doubt the UK can pull it off, but I'm not an oracle so let's see.
> Wages are rising at their fastest rate on record.
In Australia, yes. Again they have plenty of resources to go by right now. In the UK, I'm not sure how sustainable it would be. But I hope it will. Maybe the UK will prove to be the poster child for a successful economy after closing your borders. I doubt it, usually trades needs flexible borders, but I could be dead wrong, I'm not an economist.
> Working holiday visas require that you must work in service or agriculture, and the will kick you out when your time is up
Yes, but right now you can't even do that in the UK.
> I guess I can't disagree with your personal experiences at University, but I do suspect a lot of it comes down to the institution you attended. I studied at two universities in the UK and found them both academically rigourous.
It's not just universities, though. Just chatting with people of my age kinda scared me. They didn't handle basic geography, couldn't read subtext and were very sensitive to pandering. All in all, they had a very narrow-minded point of view compared to their belgium, german or irish counter parts. Of course, as soon as you reach business London, this changes. But that's true everywhere in the world.
And although I understand arguments in favor of brexit, it's really hard to take the average brexiter seriously. They are the facebook anti-vax of politics, and I suspect their education system didn't help with that.
It's already the case, that's why you see some mistaken tutorials telling you to "sudo pip install".
A venv is just a dir and some path, and your os already have some dedicated to their own python install, even if it's not called a venv, but something like dist-packages + manual PATH fudging.
This proposal makes the distinction clearer by putting safeguard to NOT to mess up with the system stdlib.
You should be using "--user" or a local venv, and "-m" to call commands.
I'm missing something I guess - I don't understand your comment in the context of this claim near the start of the article:
"The root cause of the problem is that distribution package managers and Python package managers ("pip" is shorthand to refer to those throughout the rest of the article) often share the same "site‑packages" directory for storing installed packages."
Having a system venv - managed by the system's package manager - would mean this "root cause" would go away, no?
Of course by poking around the filesystem tree and messing with managed files as root, you could mess up the system venv but that's possible with any installed package.
As I said, a venv is nothing more than a directory + PATH setup. OS already have a dedicated one (in fact, sometimes several). The problem is not that the separation does not exist, the problem is that pip is using the OS real estate.
Creating a new-new separate directory using virtualenv is not going to do anything, just add one more dir to the mix.
>Having a system venv - managed by the system's package manager - would mean this "root cause" would go away, no?
Oh, but you need to explain to pip that it's managed by the system's package manager. Which is what this does with the "EXTERNALLY-MANAGED" file.
And you need to do that because the system's package manager and pip have separate sources - with `pip` you get the packages from pypi, with the system's package manager from its repo.
To supplement, the “root” issue of the whole situation is some users are actively installing things into that system env. So the PEP is providing a mechanism for pip to detect it’s being used to install things into a system env and have a chance to refuse.
* Exactly as parent comment says, feel free to use the system Python but install packages with the system package manager (sudo apt install python3-numpy etc.). NEVER run sudo pip anything.
* Use virtual environments (either venv or virtualenv) and do not use sudo for this. Keep these within your home directory (either within or next to relevant project directory if that's your preference, or within something like e.g. ~/envs/).
So using Docker is making your workflow harder IMO because it's harder than the above and doesn't offer any benefits.
Not sure what you mean by "no script using pip will break your install". Why would you run pip from a script? Once you've activated your virtual environment, you'd just run pip install -r requirements.txt. And what is "your install" that you'd break? You can't break your system Python with pip because pip doesn't have sudo access to break it. And it can't contaminate any virtual environments except the one that's active because it doesn't even know about them.
While the best practice is to use a venv even in docker images, docker users often believe that because they are into an isolated container, they can pip install their heart away directly in it. Also, docker image default users are often root.
They may also want to install tooling that they want to use outside of a venv, such as a command line tools, and do so outside of venv, not even using pipx.
Or they may just don't know about virtualenv, or follow a bad tutorial/documentation.
Or just do one of those curl stuff | bash that pip install something.
Or think they are in a venv, but they are not (because they don't use -m) and it doesn't work so they try to solve their problem with a bazooka.
This is also true outside of docker.
Bottom line, plenty of people do the wrong thing, and since we can't prevent all of them from doing so, or educate the entire world, the next best thing is to mitigate the impact.
All, in all, I still think out best bet is something like pyflow (see my comment about it). It solves so many problems at once, including this once. I hope it becomes the de factor standard.
I think you misunderstood the original comment. They seem to be using Docker in the first place because they think it will help simplify managing Python dependencies. The answer is surely that their life would be simpler if they didn't use Docker at all, and instead use venv/virtalenv on directories within their home directory. Your comment about how to use pip if you're going to use Docker doesn't apply - the best answer is not to use Docker at all (if you weren't going to anyway).
There's not point talking about pyflow or some other tool to simplify your Python dependencies until you've said me _why_ I need another tool. What's wrong with the plan I listed above (with no Docker)? If I want a new set of dependencies, I just make a new virtal env, activate it and pip install away. If I want to get rid of it, I `rm -r ~/envs/foo`. What could be simpler than that? There's just no problem to solve in the first place.
I actually use a similar setup as yours, but that's no the point. All my comments are in the context of the posted article, which explains why we need such a move.
People don't always use this setup, whatever is the reason. You won't change that.
The next best thing is to make sure if people don't, it doesn't break the world.
> People don't always use this setup, whatever is the reason. You won't change that.
You might change that if they're explicitly asking on HN what the best setup is! For those people you can just recommend to use the same simple setup as us.
It is not as if I manually work in a terminal inside a container that is somehow permanent and valuable.
I just once create a dockerfile with the commands to set up the application I want to use. That's it. From then on, I use that container whenever I want to use that application.
If pip installs something that has a dependancy with the same name than any of the packages you install, it can cause a conflict, because by default, they will override each other.
Sure, but I'm not saying everyone should adopt Nix; I'm saying the solution to problems caused by global mutable state is to avoid adding more global mutable state.
If 'apt-get dist-upgrade' is breaking the packages we pip-installed into /usr/local, I'd ask (a) why we're using two package managers, and (b) why 'put these files in these locations' is being solved implicitly by running imperative, non-deterministic commands to mutate things in-place, rather than e.g. extracting a .tar.gz of known-good dependencies.
> If 'apt-get dist-upgrade' is breaking the packages we pip-installed into /usr/local, I'd ask (a) why we're using two package managers,
debian packagers won't use pip for obvious reasons. dev need pip because it's portable, and doesn't need packagers validation.
It would have been better if something like nix would have been adopted 30 years ago by both community, but it hasn't been so we have several packages managers. It's even worst now with poetry, conda, snap, flatpack and docker.
> why 'put these files in these locations' is being solved implicitly by running imperative, non-deterministic commands to mutate things in-place, rather than e.g. extracting a .tar.gz of known-good dependencies.
pip now uses wheels, and basically does that. Whl files are zip, and pip just unpack it at a know location. The problem is, this location is currently shared with the OS if you don't use --user or a venv. This article address that problems. My other comment also talk about some other solution.
It's very hacky, because, well legacy and all that.
> our entire legacy stack is stateful, there is no way around that unless you nixify the entire stuff
AFAIK Docker images aren't stateful: they're immutable snapshots, content-addressed by their cryptographic "digest". That's an elegant idea, which works perfectly well for legacy systems without "nixifying" anything (although personally I prefer to generate container images using Nix rather than Docker, but that's besides the point ;) )
Images are typically used as a "base", with further "layers" on top; those layers are also immutable: they use the same content-addressable "digest" mechanism, then get union-mounted together. This lets us use a known-good base, and add our application-specific parts on top, like a /var/www/html directory or whatever.
The problem with creating a layer via 'apt-get dist-upgrade' is that we no longer have any idea what will be in the image (it depends on when we build it, as repos get updated); we have to store/transmit both the old and new versions (the base and our layer); any security audits for the base image are invalidated; etc.
This isn't a "legacy" problem either: it's simply a rejection of the 'base image + layers' concept. It's fine to reject that concept, but there are cleaner ways to do it like debootstrap (which is the legacy solution, around far longer than Dockerfiles running 'apt-get dist-upgrade').
Also, the Dockerfiles controlling this stuff are usually kept in git repos and hence are also immutable. However, there's another anti-pattern of pushing their results to some ":latest" tag, which defeats much of the point of versioning.
These sorts of bodges are everywhere.
Consider the problem actually being discussed in this article: that files managed by 'pip install' may conflict with those managed by 'apt-get'. APT has been able to detect such conflicts for decades (even longer than Nix!). The article even mentions 'dpkg-divert' (part of APT's conflict-resolution system), but only as a patch for Dockerfiles to avoid the new pip-install conflict detection! A slightly different patch (e.g. running 'dpkg-deb -b' on the pip-installed libraries) would not only avoid problems from the new pip-install behaviour, but would also allow APT to tell us if we're breaking things.
I don't see these issues as a 'legacy problem' either; it's more like a WorseIsBetter approach which (a) undermines improvements when they're introduced, (b) propagates throughout the industry as "best practice", then (c) has to be supported forever, even by those who don't care to use it.
Each things you talk about are plateform dependent, but the solution for pip is platform independent and those way easier to implement. A perfect solution you can't implement is of no use unfortunately.
It's not just about finding a technical solution. A lot of packaging stuff is dealing with organizational barriers.
It basically uses the concept of node_modules, making python interpreters load any local directory names `__pypackages__` . There are 2 differences though:
• Unlike JS, python can only have one version of each lib for a given setup.
• Since having several versions of python often matters, you may have several __pypackages__/X.Y sub dirs to cater to each of them.
It also forces you to use "-m" to call commands written in Python, which is the best practice anyway. I hope it will push jupyter to fix "-m" on windows for them because that's a blocker for beginners.
If you are not already using "-m", start now. It solves a lot of different problems with running python cli programs. It's an old flag, but too few people know about it.
E.G: instead of running "black" or "pylint", do "python -m black" or "python -m pylint". Or course you may want to chose a specific version of python, so "python3.8 -m black" for unix, or "py -3.8 -m black" on windows.
At last, some other tools that I wish people knew more about that solves packaging issues:
• pyflow (https://github.com/David-OConnor/pyflow): it's a package manager like poetry, but it also installs whatever python you want like pyenv. Except it provides the binary, no need to compile anything. It's a young project with plenty of bugs, but I wish it succeeds because it's really a great concept. Give it a try, it needs users and we would all benefit from it becoming popular as it's a very sane way of setuping a python dev env.
• shiv (https://shiv.readthedocs.io/): it leverages the concept of zipapp, see PEP 441 from 2013, meaning the ability that python has to execute code inside a zip file. It's a successor to pex. Basically it lets you bundle your code + all deps inside a zip, like a Java .war file. You can then run the resulting zip, a .pyz file, like if it were a regular .py file. It will unzip on the first execution automatically and run transparently. It makes deployment almost as easy as with golang.
• nuitka (https://nuitka.net/): takes your code and all dependencies, turns them into C, and compiles it. Although it does require a bit of setup, since it needs headers and a compiler, it results reliably in a standalone compiled executable that will run on the same architecture with no need for anything else. Also it will speed up your Python program, up to 4 times. In my experience, it's also easier and more robust than pyinstaller, cx_freeze and so on.
• pyodide (https://pyodide.org/en/stable/): python compiled to WASM to run in the Web browser. Useless for web programming given the huge size of the runtime, but great for teaching, as it allows students to basically access a zero install full featured python dev env by clicking a link. Try it out, it's awesome, you can even create and query a sqlite db, thanks to the virtual FS: https://notebook.basthon.fr/
As per the parent comment, it's not exposed to the javascript running on the page, which is arguably the "dev" in this case.
>Not you the client, hence the user.
I fail to see how it being exposed to the javascript running on the page or not makes a meaningful difference to the user. Are we expecting the average user to interact with the page via the developer console or something?
>and we all have seen where it leads to.
The proposal doesn't have any DRM element at all. At worse you can just inspect network traffic to download the model file to disk and open it in whatever app you want.
> I fail to see how it being exposed to the javascript running on the page or not makes a meaningful difference to the user. Are we expecting the average user to interact with the page via the developer console or something?
No, but I'm expecting the average user to be able to enjoy an open ecosystem. Such as bookmarklet, addons, client side widgets, electron apps, WASM plugins, and all the wonderful things we can't yet think of now.
Instead, we will have one more walled garden.
And just like that, piece by piece, the next step in the IT revolution will be more and more close. Because that's the plan, that's what they want. Something the computer runs, but has no control over.
>No, but I'm expecting the average user to be able to enjoy an open ecosystem. Such as bookmarklet, addons, client side widgets, electron apps, WASM plugins, and all the wonderful things we can't yet think of now.
But there's still webgl and <canvas>. I'm failing to see what type of use cases we're missing out on here, because javascript access is missing.
People also consume refined sugar, alcohol, saturated fat and cigarettes regularly.
Just because something is common doesn't mean it's good for you.
I'm not against closed software mind you, I'm using some myself, but the web as a platform has reached this amazing stage because it's been built on openness. And now the very same companies that became rich thanks to this openness want to close it to get all the benefits for themself.
I suppose the rational is:
• Diminish the friction that multi-culturalism and diversity integration can bring.
• Free jobs for the local population.
• Having to share less of lands, resources, and infrastructures.
And I see the point, but it also fails to take in consideration that:
• The UK needs the import/export flow it was used to pre-brexit to keep their current life style. They don't produce much good internally, and they need to export their service. Brexit added huge barriers to that, not just for people.
• The current British population doesn't seem very motivated to take back the jobs they delegated to immigration.
• Even if it were, it can't be instant, since that requires training an entire workforce and sometimes even require permits or certification. Trucks, bus and cabs come to mind. And the inertia will do some damage. The supply chain is already seen glitches, although not as much as I would have expected. They are surprisingly pretty resilient.
• The Northern Ireland situation is a ticking bomb. Yeah, immigrates are not just brown people. People have a short memory though, so I suppose IRA stands for nothing nowadays. They may get a nasty reminder that ignored history is doomed to be repeated.
• Scotland nationalism has been exacerbated since they were pretty anti-brexit. Also, as an immigrant, scots were pretty welcoming, they need people there.
• Big players that needed to stay in Europe are sometimes moving to France or Ireland. So the promised jobs have to take that in consideration.
• The UK has an aging population, with a 0.53% growth rate, and that was when compensated by immigration.
• The UK education system sucks. People think Oxford and Cambridge, but the reality is more Leeds Metropolitan. I've studied there, and it's not going to be easy to find qualified workers only in the island.
• The UK managed to negotiate a very privileged deal inside the UE. It had most of the advantages, and way less of the down sides that other members. This deal is dead now, they have the worst of all worlds. So even if they get some benefits, the cost may be huge.
• France and the UK never really liked each other. The UE brings peace. But now, all bets are off. Also, no more bonding program such as Erasmus to bring future generation together. And good luck to make you BSC valued outside of the UK. And I guarantee the already little patience we had for the retired brits in the South of France that never try to learn proper France will not hold forever now that we are not bros.
• All that in the middle of covid. Not their fault, but damn, that's a terrible timing.