Hacker Newsnew | past | comments | ask | show | jobs | submit | more BiteCode_dev's commentslogin

There has been a long debate about it between core devs, pypa members, conda CEO and a few key elements of the community recently.

The conclusion has been that:

- distributing story is handled ok, but distributing program is not.

- python has some unique problems due to its popularity, diversity of user base and long history

- the py command should be available everywhere

- communication on best practices is lacking, and people mixes project deployment, dependency management, lib packaging and program distribution

- the way python deals with imports is the root of all evils, not the packaging system, which actually is a consequence of it

The best course of action to take is still not clear though, as creating yet another layer on top of the mess is probably not the right thing to do.


Given none of them are used outside of their specialty, no.

For Python, it's the reversed: it's used everywhere, except for a few topics.

It's the only scripting language like that.


People have this bias that Ruby only used with Rails. Yes, Rails made it popular in the first place, but it has overgrown it. Other than data science/machine learning (because of the lack of libraries/frameworks) Ruby used in all over the place, just like Python. Ruby basically ruled the DevOps space before Go and still very relevant.

Yes, Python is winning the popularity battle no doubt on that, but that nothing to do to the language or the surrounding tooling itself. It's like saying VHS is an incredible format compare to Betamax(Yes I'm old :D).

I used Ruby (not Rails) and Python for many years by now. I think Ruby and its tooling are far better than Python's. But if anybody asked me which language should they learn as their first, it would be Python, just because there will be more jobs available.


Phone calls, emails, and text messages don't have e2e encryption, and we have been using them for 40 years. We are still using them.

Nobody cares.

The USA communications are under constant mass surveillance, by a group of people that denies it, abuses it, uses secret courts to rule about it even when unconstitutional, coerce others to submit every data and shut up about it, and destroy the life of whoever point finger at it.

And people still don't care.

Good luck with telling people that are recording their entire life on facebook, snapchat and tik tok to be mindful about the tools they use.


What’s the point of lumping private phone calls in with things you post publicly to the internet?

Like even someone who posts to IG and TikTok still probably wants to have a private conversation. I think the issue in tech circles is that people outside of them don’t necessarily seek “mathematically private” conversations. My conversations on FB, Hangouts, phone calls, email, etc. aren’t literally mathematically probably private but neither is any IRL conversation.

And ultimately the point is that I don’t particularly care that FB in theory can see my conversations because my threat model doesn’t include Google and Facebook — it includes my friends, family, peers.

It’s not a lack of caring. It’s a misunderstanding about what people’s threat models actually are.


The people at Facebook are peers you don’t know. Not figuring them into the model, but figuring everyone you do, is poor threat modeling.

People’s threat model is to protect their physical/literal life. Without it, as far as we know, emergent emotional ideas of threats is nothing to be bothered with.

Death isn’t to be feared. I’ve been medically dead. It’s just blank. It’s pressure humans bring at scale to have their work externalized into easy to use tools that freak me out. How easily they seem to be willing to dismiss threats of human behavior

So long as society stays stable they’re the least of my concern while billions are destroying the environment. But they are part of a real threat at scale.

The pandemic response in action right here: this thing that will grow into a problem isn’t a problem. My stable neighbors, when measured by their activity of routinely going to a job, are the real threat.

Yeah that’s true but not at the scale you’re imagining it.

Pretending we should, at scale, explicitly model protectionism for all those private customization is a fools errand.

Let’s customize social protection for shared biological traits, and let people wank their specialness in private. Oh, but we have to live like the romantic stories old people carry forward. Where external social pressures rule us.

What’s the point of carving up data into pools generated by any one person except when scientifically interesting? Oh right to build my custom little emotional castle.

It’s people like us, instigating others to give their special details to our DBs over empty false promises of something special occurring at scale if they do.

How anyone smart doesn’t see the obvious parallel to religion there, I don’t know.

You’re not owed a cool tech job while someone else grows your potatoes. You should include the pressure you put on people that don’t owe you personally in your threat model


> my threat model doesn’t include Google and Facebook

Who does it include? Does it include the government if Facebook develops a sentiment model trained on public and private data that governments can use only public input to estimate [dissent|radicalism|political affiliation|etc]?

Do not underestimate how companies can abuse your privacy without direct disclosure.


I think he means that he doesn't care if Google and Facebook or even the government finds out his aunt sue's Apple pie recipe that she posted on his wall.

If he was throwing an overthrow the government party, then he would put in some thought as to which tool to use and probable stick to verifiable, open source utilities.


Presumably they have conversations somewhere between those two poles. But maybe they really doesn't care.


> Nobody cares.

Repeating this mantra all the time is not helpful.

People have been told that modern computing and privacy cannot coexist. That all software companies spy on them and users can only choose between giving up privacy or giving up technology.

On top of that, privacy, by itself, is meaningless. What matters is information inequality. Inequality is power.

When people can monitor a government you have democracy.

When people cannot monitor a government and the government monitors people you have tyranny.

Unsurprisingly, there are many paid privacy and anonymity services for wealthy people.

Please don't say that people just don't care. People have been educated to be meek to authority.


Yes. Now what are you going to do about it?

Nothing.

Because alone you can't.

And the others not only won't help you, but their actions will hinder yours.

Because they don't care.

Are you going to educate them?

Here is a list of all the people that did that: https://en.wikipedia.org/wiki/List_of_whistleblowers

Most of them got their life ruined.

Nothing changed.

Because nobody cares.

Educated people have been repeating "don't put things on facebook", "use free software", "gov is spying on you".

The answers have been "I don't have anything to hide" and "you are paranoid".

Because nobody cares.

You can craft the best technological solution to that, if nobody cares about it, nothing will happen.

You can give the best information, if nobody cares, nothing will happen.

You can provide the most secure thing, if the competition gain an advantage from not doing it and you gain nothing, you will lose.

And deep inside you know that because:

1 - you are using a throwaway account

2 - you don't provide any solution to the problem

The situation is exactly the same as with all other topics where we gain a lot of comfort individually to do the wrong thing collectively.

Did we slow down global warming? Didn't we prevent 60% of the insects from disappearing? Did we stop delegating slavery to asian countries? Did we even stop buying from all those companies that enrage us in the news?

Nope.


This is not the only way how political fight happens. There are indeed many whistleblowers with runied lives. But there are also many more people who devote a part of their life to educating others and encrypting what they can. I put GNU/Linux to every computer of my relatives I could reach. I participate in I2P and Tor networks regularly. I also have a day job and I am not going to risk it. But if a tiny part of the population did what I do, the world would be entirely different. People do care about anonymity and privacy. It's just that they have very limited resources and not everyone can devote a significant part of their time for that. Please help them and do not spread the mood of giving up.

tl;dr: Privacy is not binary. You can always increase the price of hacking you and others.


You make the mistake a lot of geeks does, thinking it's a technical problem. It's not.

You can have the best tech in the world, if the state is against you, if laws are against you, if society is against you, you'll only have scraps of a life.

It's all about people caring. Because only the mass of human can oppose anything. The rest are just sparks.

And you can state "do care about anonymity and privacy" as much as you want. Words don't matter. Actions do.

And people are still putting their entire life willingly on Facebook.


I never said it was a technical problem. It is the problem of asymmetry in information and resources. When more people know that you can confortably use devices without spying and violations of privacy, more people will oppose stupid laws removing our basic rights and spyware.

However, not everyone has time/energy to think about those issues. You need to help people by showing them better options, especially if they do not impose any restrictions. And a lot of free software is like this in 2020.


I wish I could upvote your post 1000x. People don't want to spend time and energy on these things. They just want to live their life. It is only when they can't that people start to care. Despite everything, quality of life in the US and western world is far too high to have people (in large numbers) pushing back in any real way.


Let me remind you that revolutions happened throughout history.

After thousand of years of cynics saying "nothing changed", one day, a tipping point is reached.


I've never heard of a bank account being compromised because of lack of e2e encryption by Facebook, Google, Microsoft, etc. When that happens then post about it and I'll start caring.


> And people still don't care.

yep, and that is correct.

People will not give up convenience and performance for encryption/privacy. And they only barely care when it comes to their financials - because they assume the bank carries the credit card fraud risk for them (whether it's true or not is moot).

And encryption is important, but not important enough for the general public. This is why nobody uses TOR for everyday usage, even tho doing so is massively better for the privacy of society.


Tor is an absolute pain to use if you visit any sites using Cloudflare, which is quite a lot of them.



By this logic GPS is "not secure" because the US government pays for that too.

But GPS receivers don't even have to transmit anything, they calculate your position by receiving broadcasts made by satellites with timestamps and calculating your position by triangulation based on how long it takes for messages from particular satellites to reach you. It would be difficult to have better security properties than this, which was done on purpose because the US military can't have the positions of their units leaking to the enemy.

We know this is true not just because their incentive are aligned but because the system's mechanism of operation is public and anybody who discovers a vulnerability can publish it. And so it is with Tor. The US government funds it because they use it, but it's also completely open source. If there is a vulnerability in it, as they say, PoC or GTFO.

Most vulnerabilities don't come from breaking GPS or Tor or AES. They come from, you have a suitably private way of calculating your position using GPS and then your phone takes that information and uploads it to The Cloud. They come from people visiting a website using Tor and then typing their name into it. They come from people uploading their TLS private key to a public repository, or generating it with a predictable random number generator.


The US government can and does turn GPS off for certain regions of the globe when it feels the need to.


Which still doesn't give them the location of your GPS receiver.

It also applies even less to Tor, because they "turn off" GPS by encrypting it so that only their own receivers can use it. Doing the same thing with Tor both doesn't work because they don't run all of the relays, and doesn't work because it would compromise the anonymity of their own users if they were ever the only ones using it.


I for one don't care as long as my important communication is. And should the US government ever turn authoritarian and malevolent then we can all use VPNs and switch to better communication as happens in China. It's not like Americans are forever accepting unsecure communications, they just don't feel a need for more security. When the need arises Americans will adapt. Hopefully they don't have to.


I'm not sure I could adapt at this point. Yes, I know how to use encryption to properly secure a conversation, but that assumes I can trust my device, my recipient's device, and that the underlying algorithms are really as unbreakable as advertised.

At this point I go through life under the assumption that the government (or bad actors within the government) could eavesdrop on every facet of my life if desired, and live with the anxiety that entails, which is low enough to be drowned out by other, more traditional anxieties. And I focus on political solutions rather than technological ones.


My point is that other countries get along. For Chinese citizens VPNs are a way of life.


This is false.


https://www.pcmag.com/news/breaking-down-vpn-usage-around-th...

>31% of Chinese internet users use a VPN.

>17% of North American internet users use a VPN.


> And should the US government ever turn authoritarian and malevolent then we can all use VPNs and switch to better communication as happens in China.

Should that happen, it is now relatively trivial for carriers to block VPNs—as is being done in China.

It's also trivial under such a setup to make unlicensed VPN use punishable by imprisonment.

Don't be so sure of technical or political solutions exclusively.

Secure by default is extremely important, regardless of your personal emotions about your own communications.


Everything goes out the window if you have a government willing to do anything and everything. It's a moot point. All this end-to-end encrypted technology would just be banned as you suggest.


People seem to care when made aware of it. I think most people are ignorant of most of the facts in your message.

An awareness campaign would help, but both the state and the large media companies are against that.

https://youtube.com/watch?v=XEVlyP4_11M


So what? Conference tools are a replacement for face-to-face meetings. The expectation in such a meeting is privacy.

Further, tapping your phone is unlikely to hijack your computer, steal your credentials and company data.

False equivalences like this are not useful. Security should work. Most folks are unaware of how vulnerable they are, and the potential for abuse is astronomical.


How many consumers of these services would find themselves targets of state-level actors, though?


All of them. See PRISM and XKeyscore.


>And people still don't care.

How do we make them care?

If one looks at an issue that would emotionally charge the population, the protection of children, and then looks back 150 years, you'll find that much of society didn't appear to care. In the US, for a period of time, pets had more legal protection than children. I'll avoid the grimmer details, but people were willing to tolerate things that today would cause mass outrage. It may have taken over a century to change public opinion, but public opinion did change and massively.

So how can we do the same with privacy and protection of basic rights?


> Nobody cares

That was because nobody really paid attenion to your specific calls/emails/text messages and nothing really happened. The government could do it, but most of us don't show up on their radar. We didn't care about them because they didn't care about us.

Nowadays that is changing. Computers are able to listen to everything, match it all up to people, and with a little capitalism and profit, use the information. But don't let on.

Voila, personalized pricing. Hire only the age you like. Don't just rent to anybody. Date only thin people (with poor impulse control). Loan money to desperate people.

I hope that this will change.


> Voila, personalized pricing. Hire only the age you like. Don't just rent to anybody. Date only thin people (with poor impulse control). Loan money to desperate people.

All of this stuff was happening at least 30+ years ago -- and probably much much longer (just I can personally recall it from 30 years ago).


Because people don’t care the architects of such systems need to care.

We should build systems that protect their naive users.


This exactly. No one cares. That is the end of the discussion for most people.

Techies might want to talk further to hear themselves blather on.

Everyone else moved on.


Is it necessarily a problem?

I've actually been bitten more by things that were encrypted without telling me when I went to try to restore things or do some kind of archiving.

Encryption should be explicit and available, but optional in the same way I could leave my front door unlocked.

The forcing of either direction is the real problem


> Is it necessarily a problem?

Yes, you neglect the concept of precaution. I draw the parallels to

• wearing face masks,

• neither confirming nor denying when asked whether one did something naughty. (See the Plausible Deniability story arc in HPMOR.)

If one does not do it regularly and only in times of necessity, it stands out as weird, socially not acceptable, drawing attention.

We want encryption everywhere, every time, even when it's not strictly needed, so it is a socially normal thing to do, which in turn fuels wide-spread use, which benefits everyone through strong network effects.


Ok let's leave the theoretical analogy world and talk about the material, concrete, actual reality in actual practice.

the crda regdb database in the Linux kernel, this one https://wireless.wiki.kernel.org/en/developers/regulatory/wi...

It's encrypted and there's a long (somewhat silly) explanation of why it's not in a plain ascii format like everything else. You can't just crack it open and try something if you have a country hardware locked wireless device (via it's firmware). It's still possible to do here with a bit of effort but the next one isn't.

Android adoptable storage, unlike Android traditional storage will encrypt without telling the user https://source.android.com/devices/storage/adoptable

You can not get access to the decryption keys if you haven't rooted your phone. You need to hack your own device to look at what you stored on your sd card. If you're having a failing sd card and want to do recovery on it, well too bad, it's now utterly impossible, even if you own the phone.

These are just two real world actual problems I've run into in the past week alone.

Encryption can be, and often is used to be an extremely hostile, closed-data system. So that devices I Own, with My data I don't have access to and cannot see. That's not a better society, just like closed source isn't a better world.


> the crda regdb database […] It's encrypted

That's false. It's just a binary database file. See man 5 regulatory.db

    $ hex /lib/firmware/regulatory.db | head -n 10
    0000  52 47 44 42 00 00 00 14  30 30 03 61 41 44 03 c5  RGDB␀␀␀␔00␃aAD␃Å
    0010  41 45 03 bb 41 46 03 cc  41 49 03 dd 41 4c 03 cc  AE␃»AF␃ÌAI␃ÝAL␃Ì
    0020  41 4d 03 a3 41 4e 03 cc  41 52 03 bb 41 53 03 79  AM␃£AN␃ÌAR␃»AS␃y
    0030  41 54 03 e4 41 55 03 58  41 57 03 cc 41 5a 03 bf  AT␃äAU␃XAW␃ÌAZ␃¿
    0040  42 41 03 e4 42 42 03 f5  42 44 04 15 42 45 03 e4  BA␃äBB␃õBD␄␕BE␃ä
    0050  42 46 03 bb 42 47 03 fd  42 48 03 a6 42 4c 03 cc  BF␃»BG␃ýBH␃¦BL␃Ì
    0060  42 4d 03 79 42 4e 03 da  42 4f 04 10 42 52 03 bb  BM␃yBN␃ÚBO␄␐BR␃»
    0070  42 53 04 05 42 54 03 cc  42 59 03 cc 42 5a 04 24  BS␄␅BT␃ÌBY␃ÌBZ␄$
    0080  43 41 03 66 43 46 03 a9  43 48 03 98 43 49 03 bb  CA␃fCF␃©CH␃�CI␃»
    0090  43 4c 03 da 43 4e 03 f8  43 4f 03 bb 43 52 03 9f  CL␃ÚCN␃øCO␃»CR␃�
The documentation talks about optionally signing the database with a key. You confuse things. A signature is not encryption.

> Android adoptable storage […] will encrypt without telling the user […] You can not get access to the decryption keys if you haven't rooted your phone. […] If you're having a failing sd card and want to do recovery on it, well too bad, it's now utterly impossible, even if you own the phone.

Works as designed? In this scenario, you're not the recipient of encrypted transmission, you're the adversary.

You should not be upset at encryption as a generic tool. You should be upset at the creators of this system that turns people who paid good money to own a mobile computer into dependent thralls: they cannot exercise the freedom to do with their property what they want. The answer to that injustice is not to abolish encryption, but to abolish the use of non-free software that enables the creation of such systems.

The predications from my previous post still stand.


Of course it's the implementation that matters and how it's used, that's the whole point. The reality is important. It Has To Be Done Better.

Sorry about the regdb example, that was me being frustrated with it and not looking deeply into it yesterday. I saw the binary format, saw the signing and just kinda assumed the rest (I've been trying to get a stronger 802.11 signal from these usb alfa and panda adapters on this pi4 I'm using)

Anyway, it could be a bold protector of personal liberty or another cargo cult ceremony on the checklist and callously shoehorned in inappropriate places.

It could be a defense of privacy that Schneier holds dear or a way for a company to hide from the user what it's recording and tracking through closed encrypted data

It could be Zimmerman's dream or make it so people get locked out of things they own. Locked out of their Bitcoin wallets, hard drives, phones, just about everything.

The "encrypt everywhere" assumes the rosiest of intentions from one group and the most malevolent from another. It assumes perfect technology without failure or fault, perfect human memory, and perfect organization.

I call it homo securitis. Humans don't work that way. (After https://en.m.wikipedia.org/wiki/Homo_economicus)

That's why I can sit around all day and give you endless real life horror stories. For the bitcoin example, I've personally lost north of a million dollars through mismanagement of wallets (I wasn't monitoring disk failure on a raid array around 2012 and 2 of them finally went). You can look at this list (https://bitinfocharts.com/top-100-richest-bitcoin-addresses....) - there's a number of wallets, hundreds, with histories like this: https://bitinfocharts.com/bitcoin/address/12ib7dApVFvg82TXKy..., last activity 2010. Either this person has the discipline of a soldier or they lost access to it. Hard disk crash, accidental deletion, who knows - normal human failure in an imperfect world.

Human practicality matters. Imagine if a traditional bank kept 243 million locked away forever because you lost a special ID card they sent you in the mail in 2009. This stuff matters. An excuse of "well it's your fault for not being perfectly organized on perfect systems" doesn't cut it.

Encryption and security is too often an unsophisticated hammer that's getting tossed and applied without consideration of the nuances of reality.

It Has To Be Done Better. Not "don't do it". But instead "stop foolishly doing it so naively". Otherwise there will be a large push back and nobody will want it at all.


It seems you are conflating transport encryption with encryption at rest. I can not imagine a scenario where encrypted end-to-end transit can "bite" you as there is no recovery scenario apart from snooping network operators/attackers.


E2E encryption can be really resource consuming, especially with the amount of traffic used for video chat. You might not want this just because of a worse user experience.


Video chat is something like 1.5Mbps. Even phones can do chacha20 at >1000Mbps these days. It's not that expensive.

Where people start to feel it is if you want the server in the middle to be decrypting and re-encrypting everything, because it's not that much for one user but it starts to add up for a million users. But the obvious solution there is to use E2EE and then it isn't doing that.


I cannot inspect what closed source software is sending to the server, how it is tracking me or what information it has. It is encrypting my data against me so not even I can view how I'm being studied or recorded. This applies to the public writ large, nobody outside the deep annals inside the company knows how the public is being monitored and sold.

I've tried to make various tools and utilities for things like logging, auditing, inspection, etc, and have had to do self signed keys and a number of interesting hacks to work around encryption. Often times it's impossible.

It's thousands of times more likely that the authorized party is spying, snooping and stealing my information to package and resell it in open bidding data markets and trying to hide that fact and cover their tracks then they're trying to be bold protectors of my liberty against roving blackhat ne'er-do-wells. Although they'll run to that defense whenever necessary.

Ever see the amount of data trackers on say, a local news site? It's astounding, and they are the technically incompetent. Imagine what happens when people with the same wanton disregard for scruples knows what they are doing.

I've convinced myself at least that instagram listens to the microphone and services ads based on conversations. I have blackboxed tested this in many ways --- and that's the only way I can do it, blackboxed - that's the problem. I could lay out the tests if you want, none are of course definitive, they can't be, that's the point. It's uninspectable and can't be proven either way because it's encryption that none of us can inspect.

Like any other powerful party trying to hide information and abuse their subjects, they claim to actually be protecting us against the dark treachery that surely lies beyond the forest. Let's all tremble in fear and thank them for their service! Give me a break. It's a slight of hand, they've instead taken our digital stuff and protected against us from inspecting the process, the protections against an additional third party is merely a side effect.

If I can't turn encryption off and see what's going on OR if I can't inspect my own device then I don't trust the network OR the supposedly "trusted" party with access to it. I don't trust Google, or Facebook with access to my microphone or ad networks running on apps and you shouldn't either. If I can't inspect the contents of the payloads, then I don't know what's happening. Platitudes and assurance be damned, what data is going OTA, that's all that matters.

I understand there's brigading and hivemind against this basic reality, but I'm dedicated to reality, not to popular opinion.


See, this is the one complicating factor for L7 E2E, and I think you make a compelling case; you pretty much need open source/reproducible builds to be able to have any kind of protection against leaking PIIs over encrypted channels.

I think the best one can do today with services like Facebook/Instagram is to use third-party clients, or, in lack of that, the web apps.

IMO DoH/DoT is the main venue where this tradeoff becomes obvious today; privacy with regards to infrastructure providers, or ability for the user to see/control what your software reaches out to?


Unfortunately, you need to:

- discover it

- install it properly

- run it properly

- make it create a venv with the proper python version

- potentially import data from existing setup.py/cfg

- make sure you are always in the right dir when you use it

- not be on windows (poetry shell doesn't work there)

- IDE integration is not great

- people have to chose between this, pip, pip-tools, flit, pipenv, dephell, conda...

And you cannot use it if you are not "in a project". But not all venv are project related.


> - not be on windows (poetry shell doesn't work there)

Ok that's a bug, probably fixed soon. Otherwise, most of these are true of many popular packaging tools in other languages.


Well, not pip and venv. It's provided on windows and mac automatically. venv forces you to chose the version of python. venv doesn't care if you are in project mode or not. IDE integration is great.


> There is no greater hell than talking them through installing a working Python environment. It’s impossible.

I give a lot of Python professional trainings, and I get to do that regularly, in very diverse situations. It's indeed full of gotchas.

Since it's not going to be solved quickly, meanwhile, here is what works if you need to help people setuping python:

1 - Install Python correctly

The first version Python download link for windows is 32 bits. You want 64 bits, so you should actually not click on that. Also, make sure they use the latest minor release possible, as early ones can have weird bugs. Tell people to install from the app store if they are on windows 10, or give them a link to the proper (non web) installer.

Linux: python versions may not be available for their linux distro. Use EPEL for Centos or deadsnake for Ubunto. Other Linux users chose something exotic and must be able to deal with it.

Mac: brew is fine. Official installer too.

Cause: Python support is crazy good. It support 32 bits. 3.4 supports Windows XP. 2.7 supported Atari and Solaris! So there are a lot of installers, and a lot of versions. But also because the official Python website does a poor job at directing the user.

2 - Run Python correctly

This is the great lie of Python running. You cannot just use the "python" command, which is what every doc and tutorials tell you to do.

On linux and mac, tell them to use the suffixed PythonX.Y command, with X.Y being the version of python they need. E.G: python3.6.

On windows, tell them to use the "py -X.Y" command. E.G: py -3.6

Tell them anytime they see a tutorial with "python" in it, they should replace it mentally with "PythonX.Y" or "py -X.Y" depending of who you have in front of you.

Cause: people often ends up with have several versions of Python installed on the same machine, so you can't tell them to just use the "python" command. Of course, windows and unix never agreed an on naming convention. What's more, the Windows situation can lead to a PATH problem, and "python" may not be found, while "py" will always be. We are talking about providing the "py" command everywhere.

3 - Install tools correctly

Introduce them to pip. Tell them to never, ever install stuff using admin rights with it, even if told to by documentation or tutorials. No "sudo". No "run console as admin".

If you need to install a tool, such as black, mypy, pylint, etc., outside of a venv, use "--user" to install it for the current user. This requires no admin rights.

Also, don't use the pip command directly. Use "-m" so that you always know from which python you are installing the command for. E.G:

    python3.6 -m pip install black --user # unix
    py -3.6 -m pip install black --user # windows
Then mention that it's only for tools, not libs. Only install libs in venv. Some tools also should just not be installed at the system level such as jupyter or pytest because they depends on their env.

Cause: installing with admin rights can destroy your python installation. Also, pip may not be in the PATH, or be attached to the wrong version of Python. It's also because we are twisting a lib packaging tool into providing programs.

This is not specific to python. Node had to introduce a whole know command, npx, to solve the same problem, and the ones bellow. It actually takes a command name, and if it doesn't exist locally, but is registered on npm, download the package that seems to contain it, install it in a temp folder with all its deps, then immediately attempt to run it in isolation.

4 - Use tools correctly

It's another lie from docs and tutorials. If you install a tool, just calling the command may fail.

If you installed a command outside of a venv like above, then you should call it using "-m". E.G:

python3.6 -m black # or py -3.6 -m black

Cause: again, you don't know how the PATH is setup, so there is no guaranty that the command will be available, or ran from the proper python version. People can have completely messed up machines and there is nothing you can do about it. So don't depend on the PATH.

5 - Use venv correctly

The solution to all those shenanigans are the venv. Once you are in a venv, you don't need to tell the version of python, you don't need "py", "-m" or "--user". You can call commands directly. You can just use "python".

But first, people need venv installed. It's installed by default on Windows and Mac, but on linux, it's often a package to install like python3-venv, python3-pip or python3-setuptool.

Do not "pip install virtualenv": people will get confused between the tools.

So, make them create a venv: "pythonX.Y -m venv name_of_the_venv" (or py -X.Y).

Show them how to use the python/pip from the venv WITHOUT activating it, so that they understand how it works and show them the that unix have ./bin and windows ./Scripts.

Then show activation, and tell them to install pytest/jupyter and Co inside. Show them that it solves all the previous problems, so that they are motivated to use them.

Tell them to NOT put their code in the venv folder. And that they can't move or rename a venv. Show them "pip freeze > requirements.txt" + "pip install -r requirements.txt" as an alternative.

Cause: historically python didn't have the venv module, and people used to pip install virtualenv, which mean we still have docs about it, and the name stuck. Having a long history makes things complicated. And once again, the differences between Windows and Unix bites us, but that's the cost of being portable. It's not Python-specific. The fact you can't move a venv is an artefact of the venv design that the community never solved.

# Variant

People may want or need to use anaconda or the python embedded in a system (blender, qgis, etc). Then it's a whole different cycle of problems and solutions, so I won't talk about it here, this post is long enought.

Again, Python diversity is playing against it: you don't have one popular python distrib like with nodejs or cRuby. You also have commercial Python distribs, and some are used a lot in the corporate world.

Diversity make things more resilient and encourage innovation. But it makes thing harder.

# Pyproject and setup.cfg

They have nothing to do with all that. They are useful if you want to _produce_ a package to share your code with other devs.


That's a pretty solid guide! It's even useful for me that writes Python (non-exclusively) professionally.

My company settled on pipenv, and we regret it. It's slow and buggy and has crazy behaviour. It's not bad enough that it's worth updating all projects (or rather, there's no good enough alternative) but I regret about once a week having advocated for it.

Note: I think you made a mistake, you say that you can rename or move a venv but then you say you can't.


Thanks. Fixed.

Being stuck at home, I currently give training remotely. If you know people that need trained in anything Python (beginner, advanced, scripting, architecture, web, data analysis, etc.), think about me :)


This is an awesome summary that should be placed on the python.org front page instead of all the fuzzy stuff that's there right now.


I've been tasked to make a PR for the pypa website with something like this. If it doesn't get merged, I'll still have a copy of it to some website as a single page. I bought usepyhton.info for this purpose, since pyformat.info did a great job.


Excellent guide, I just found one thing off here:

> If you need to install a tool, such as black, mypy, pylint, etc., outside of a venv, use "--user" to install it for the current user. This requires no admin rights

I think you should add that on Linux distros it's better to use the package manager to install libraries, because a beginner can, and will, have conflicts with (system) packages relying on a particular version of a library. Only use pip if the package is not available, while with the like of Debian, you will have old versions of libraries, I think fast paced distros like Fedora does a good job at providing up to date python libraries with the package manager.


Wow, I am not a python developer and avoid python as much as I can because I hear about problems with versions everywhere. I have no idea which documentation is correct and which is old/wrong. With your manual I might dare to try python. Anyone who says (as seen in this thread) that python is not that complicated to install and use should take a look at this monster of instructions and then shut up.


>that python is not that complicated to install

This is not the instructions to install Python.

Those are the instructions to teach anybody in any OS with any configuration to install python and deal with 3rd party dependencies.

E.G: if you are a Windows 10 user, instructions to install python are 2 lines:

- install from the store

- run "py -X.Z" to start python

The complication comes when you have people with Windows, and Mac, and Linux, and they already have some Python installed in various ways, and then want to install stuff with pip, etc.

Which is true for Ruby, JS, PHP, etc.


this is an awesome summary; I use python occasionally, and have run into several of the issues you flag in your post, but would probably do it all over again next time I need to put something in production. So thank you for taking the time to summarize things so thoughtfully.


Well, with the covid 19 confinement, I'll give training remotely. So if you know companies that need their IT department trained in anything Python (intro, senior, syadmin, API design, flask/django, numpy/pandas, etc.), think about me :)


Your comment was too long for me to read without my eyes glazing over. Maybe it's because I already know this stuff, but I suspect for someone who doesn't know it and just wants to program, it would be intimidating.


This is not the instructions to install Python.

Those are the instructions to teach anybody in any OS with any configuration to install python and deal with 3rd party dependencies.

E.G: if you are a Windows 10 user, instructions are 2 lines:

- install from the store

- run "py -X.Z" to start python

That's it.

The difficulty is to have general instructions for all cases, including packaging. Which is a problem with every scripting languages.


The setup is easy and the video works very well, including with many participants.

That's it.

There is nothing else. They managed to do one thing well.

This thing that skype used to do correctly but now don't.

This thing that no other free softwares seems to be able to manage correctly either. Not teams. Not hangout. Not Jitsi or BBB.

It makes you think.


Yes, it's imperfect and irritating in many ways. Many other actually (BT, Wifi, battery life, they all have their problems).

But your alternative is one totally locked on ecosystem and one provided with embedded spywares.

It's good enough, and since we are not giving millions of dollars to the editors like we do with products disrespecting us, I understand we can't ask for perfection.


> It's good enough

I've been hearing that for between 15 and 20 years now. I believe this mentality is part of the reason it never seems to get any better.


Oh it did become better.

I've been using linux for 15 years. It's better in every way.

My mother has been using Ubuntu for 7 years now. It would have been impossible in the early 2000.

It is better. It is productive. It is even a good experience. I'm no a masochist, I buy and use proprietary software. I use Linux because I want to.

I'm not just not blind to it's many shortcommins.

Last week I used Windows with a client. I had to disable Windows update because it was eating all the Ram. Some people are literally buying a new mac because the mechanical keyboard is unusable on some models.

There are systems you complain about. And systems nobody use.


That's true, web standard made OS less critical for an average user, and time let some other native apps improve and polish (things like krita and blender comes to mind)


Funnily enough, my 3-year-old Macbook Pro just died, and times being what they are with your usual build-to-order options being severely backordered, I stepped into a local store and bought a laptop off the shelf -- an LG Gram 17.

It's not even a brand that's well-known for Linux compatibility (like Thinkpads), but all I've had to do to get Lubuntu 18.04 running more or less flawlessly was disable Secure Boot in the BIOS. Wifi, sound, and webcam all worked without needing tinkering. Bluetooth appears to work based on it being detected, but I haven't tried pairing anything yet. The only thing I hear doesn't work is the fingerprint sensor, but I wasn't going to use it, anyway.

Suspend on lid down and wake on lid up hasn't been an issue at all. I'm actually very pleasantly surprised at how well this LTS version from 2018 runs on a 2020 model laptop.


> I believe this mentality is part of the reason it never seems to get any better.

Or maybe it’s that authoring and supporting drivers for actual hardware is tremendous effort that is mostly thankless.

Even Microsoft can’t get something much better and they literally have billions they need to throw at it.


> Or maybe it’s that authoring and supporting drivers for actual hardware is tremendous effort that is mostly thankless.

Oh, if only drivers were the only part of Linux Desktop that was broken...


+1. I'll add:

* The screen locks, unlock, but the network card cannot wake up anymore and requires a reboot.

* The screen locks, let you enter things, but doesn't seem to react, then suddenly, replay all the things for 20 seconds in 1 seconds, and displays an error message

* the screen locks, but you can see the unlock screen for a few seconds when you wake the laptop up


The first 2 sound more related to suspend rather than screen locking.


It is, but the whole experience go together. Just like booting flickers is caused by many problems, but it is about one experience.


I've had the second, and it wasn't related to suspend. For some reason, the GUI decides to do a software bit-blit of the entire screen, which takes tens of seconds, before it lets xscreensaver put up its dialogue box. So don't assume that this is always suspend-related. There are several quite varied possibilities.


Most modern computers can software/CPU blit to a 4K screen at multiple frames per second. So this explanation doesn’t make much sense as is.


Indeed. But since it's so slow that I can see the blit happening, scanline by scanline, it's definitely what is happening, even if it's not apparent why. (I do know that the particular window manager in use decides to move windows back and forth by 2 pixels to work around some sort of bug.)

Not being able to explain why the software does this does not negate the fact that it is doing this, and that it is a quite different possible reason for lengthy pauses in the user interface, one of several, that is nothing to do with suspending the machine; indicating that, as I said, one should not assume that these things are always suspend-related.


So, the entire nation uses whatsapp and gmail, the services made by the biggest data collectors on earth, known to have taken part of the PRIMS program, that send record of said data to private 3rd party for analysis, to have people inside abusing the access to that data, and they focus on zoom ?

I mean, it's good that people suddenly care about privacy, but it's like raising awareness on acne while Covid19 is out there.


please report them to your data protection officer the same way you'd report any company including Apple/Google if they violate GDPR: https://edps.europa.eu/data-protection/unsere-rolle-als-aufs...


GDPR can't stop PRISM, in fact it explicitly allows govts to use data for security purposes


Dolt is not Git for data.

Git take existing files, and allow you to version them.

Git for data would take existing tables or rows, and allow you to version them.

A uniform, drop in, open source way to have an history or row, merge them, restore them, etc. that works for Postgres, Mysql or Oracle in the same way. And is compatible with migrations.

You can have an history if you use big table or couchdb, not need for Dolt if it's about using a specific product.


You can nitpick any comparison like that by pointing out the different ways the metaphor breaks.


It's not a metaphor. It's a sell pitch.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: