Hacker Newsnew | past | comments | ask | show | jobs | submit | more BiteCode_dev's commentslogin

Frenchman here, and have a very close Kevin.

It's true there are stereotyped jokes about Kevins, especially in the IT word.

But it's nowhere near the the scale of what the article or the documentary suggests.

Kevin, working in IT, confirms that he hasn't heard the joke in years, and even in the 90, it was probably once a year. It never affected his life in any way. Honestly being a nerd in the 90' was the hard part.

It reminds me when we had a couple of cars burning in Paris and the US medias reported it as civil war. I had clients calling me, asking if I was OK.

At this stage, I'm assuming anything the medias are reporting is BS until proven otherwise. They destroyed the credibility of their own job.



pypi contains 399 794 python packages.

debian repositories have 23 168 packages, all languages included.

Or are you suggesting each python package author should release one version rpm, one deb, one nix, one msi, etc? And then that the deployment script branches for every single OS isolation isolation there is?


It's just a comic, it doesn't have to be accurate, and I think it delivers the message.

Besides, "you understand how a venv works" is not that simple.

- if you use anaconda, do you use conda env or something else? This will have huge consequences on the availability of packages and modes of failure.

- do you use virtualenv or poetry? In that case, how do you install them? How do you run them? The answer to this will change depending of your OS.

- do you use homebrew or pyenv? Then you env may just break on you one month later, will you be able to fix it?

- do you use venv? In that case do you know about the "py" command ? Do you know about "-m"? Do you know which packages to install on linux so that it's available?

Last week I was playing with a telegram bot with a friend of mind. I asked why he didn't use a venv. He told me he could never remember how it worked.

He has been running very lucrative a django site for 10 years.


Devs are definitely no fungible, the difference in the productivity, team moral and new bug introduced by just changing one person in the team can be huge.

Even simple boilerplate is done differently by people. Some will automate them, some will do them manually forever. Some will naturally organize to discuss how to limit or improve them, some will stay with the status quo ad vitam. Some will document how to do things, provide templates to limit mistakes and mentor new comers. Some will just do their job.

And that's not even touching the fact some are simply bad at what they do.

In all the successful projects I've seen, hiring the right people or replacing the one leaving were critical processes, not just swapping.

This idea you foster is probably half the reason 2/3 of IT projects fail.


Python is still a lot of fun in certain areas:

- machine learning, because AI is fun, and it's very easy to play with it in python. Also because you can easily stitch services together that allow you to make cool apps without having to code the hard part, like creating a discord bot to talk to midjourney.

- anything that will use pydantic and type hints to generate parsing and validation. Fastapi, typer, etc. It's really a blast to just define types, and see your web api take off.

But it is to be expected that when something has reached a critical mass, it becomes banal, and therefore not exciting anymore.

Also, the Python ecosystem and resources have accumulated a lot of legacy stuff, and has been flooded with new comers content, so the noise to signal ratio is not great these days.

I though as a python expert I would be less relevant given the number of new devs entering the market. Boy was I wrong. It just feel like having a cheat code to ask for more money to my clients, by the sheer amount of things I learned because I was there 15 years ago.

The dependency thing is a good example. Dependencies is in actually a better state that it used to be, but because there is so much noise, very few people can enjoy it because they would have to setup python in "the right way" to do so. However, unless one invests a huge amount of time in looking up for this right way, filtering all the noise is going to be impossible. Hence most people have a broken python setup, and when they try to install something, it has a high chance of breaking.

Of course, technical dept, and our inability to pay it back as a community, doesn't help.


What's the cliff notes of the right way / where's the signal in noise?

I don't write Python, but Python is like Java these days, it finds you.


In short, a huge number of packaging problems come not from packaging, but from the way you install, configure and call python. Unfortunatly, everybody and their mother tell you how to do this, and you'll find 1000 articles to get you in trouble, but very few to tell you what you should actually do.

The full story is pretty long and most people don't have the time and resources to learn everything about it.

The easiest way is to follow a receipe for the least pain possible, which includes limiting yourself to a single path for bootstrapping python. Here is the one I currently recommend to my teams at the end of the comment:

https://news.ycombinator.com/item?id=32805483#32807220

It can't solve as much as knowing the whole deal, but it will remove a lot of ways to shoot yourself in the foot, and work around a lot of the legacy issues. And above all, it's easy to keep around and to apply. I noticed that anything more complicated will lead to people, so I removed a lot of things from it years after years (like -m, pipx, etc).

Of course, this will not solve the setup that are already in place, which you will likely have. I can't provide a generic way to solve that, there are too many possibilities.

Hopefully the community will eventually solve most of this by providing better default in the future, but this is an excrutiatingly slow process.


My impression is Python packaging is stuck in a local maximum, and that any attempt to move out of it will be as disruptive as 2->3. Dropping support for conda et al will only split the ecosystem, hasten the rise of Julia, and make Python packaging more acrimonious. I like your simplified approach (and take a similar approach where I can), but much of the value of Python comes from its ability to be inserted without too much fuss (assuming you'd build things from source) into an existing system. I'd suggest while wheels were a good idea at the time, they've become a parallel ecosystem and source of division.


Not at all.

As I said, most packaging problems don't come from packaging itself, but for bootstrapping. Symptoms from a terrible bootstrapping situation creep up and they come out in packaging.

Unfortunaly the solution is of the worst nature possible for a FOSS project, because not technical: it's policital. This is the kryptonite of volunteer efforts.

The community has to get organised around one way to install, configure and run python. But it's really hard to do, and requires to synchronized a lot of different group of people, efforts, and make sure everyone are on the same page.

E.G, of a single point:

To chose which version of python to run, the installer for windows provides the "py" command, anaconda provides the conda prompt, but the windows store provides a suffixed command like most unix do. Most people are not aware of those differences, and when they attempt to follow a tutorial that has been written by somebody knowing only one of those combinations, the person will fail. Either with a "command not found", a broken install, "import error", a "syntax error" or installing things for one version of python and running the other version by mistake.

This will lead the person to believe packaging is broken, while it's not. The way we install and run python is broken.

But how do you solve this problem? By engaging in a long, frustrating and thankless talk with each of the parties involved. There will be debates, people will talk about taste, diversity, backward compatibility and so on. It will be a nightmare.

The alternative would be for the core devs to come up with a single recommended setup for all plateforms, with a unified configuration, and promote that massively. This will cause another huge political problem: you now have 10000 of tutorials that are referencing one of the old ways, and you will spend the next 20 years explaining to people any previous configuration are not supported. There will be a lot of complains from devs, and pressures from companies.

Either way, you end up with a load of work, and unhappy people, which nobody wants to sign for. It would be better on the long run, but nobody wants to sacrifice themself to be hated in the end.

And that's just one single point.

You still have to solve homebrew troubles, linux distro splitting python and limiting versions, bootstrapping tooling outside of venv, solving the uncanny valley of venv, and so on.

Note that we haven't even touched packaging yet.


> A recent example is WakaTime, who replaced Celery with a custom-built queue. This effort took one week to build and productionise, and consisted of just 1,264 lines of Python.

What a load of BS. Wakatime doesn't even have tests, has been used by one person and has no error handler. It's 1k lines because it's alpha sotfware.

Celery is not the most fun to play with, but it's not big for nothing. It supports many routers AND result stores, and you can mix an match. It has several policies regarding errors, has a beat daemon, provide an API for monitoring and managing (see: https://flower.readthedocs.io/en/latest/ for UI), etc.

Also, you can use celery in a very bare bone setup using only the FS for routing and storage: https://www.distributedpython.com/2018/07/03/simple-celery-s...

I understand you may want to avoid celery. I had bad experiences with it myself. Most projects can get away with rq, or even just multiprocessing.Pool.

> Before embarking on the mission of creating a new task queue do survey the existing options, but make sure not to underestimate the hidden costs of using one, or the benefits that may come with writing one from scratch.

Writting your own task queue is like creating your own csv parser. It seems like a very simple task that is barely a for loop with a split. And then you start hitting edge cases after edge cases.

Basically you are coding an asynchronous, fault tolerant process manager including task serialization, communication protocol, priority queuing and lifecycle.

That's not something you usually want to do. The cost of using one is unlikely higher that the cost of creating, maintaining and documenting one. Just go with a simple existing task queue if you need something simple. Pypi is full of those.


The task queue we built at Thread wasn’t much bigger, actually supported multiple queue backends, had tests, and has run millions of jobs a day for a decade. The team knows it very well, and it takes a few days a year to maintain, improve, and iterate to fit with new infrastructure.


The dirty secret about Python packaging problems is that a lot of them... are not relatated to packaging.

Python has first and foremost a big __bootstrapping__ problem, and the consequences of that are often seen when you try to install or use dependancies. So people conclude that packaging sucks.

Now don't get me wrong, packaging have some problems, but compared to most languages out there, it's actually pretty good.

But people cannot enjoy that, because they would have to install, setup and run python in a perfect sequence.

So what's wrong with Python bootstrapping, and why does it affect packages?

Well, as it is often the case in FOSS, it stems from too much diversity. Diversity is tooted as this one great thing, but its benefits also come with a cost.

- there are too many ways to install Python. Official installer, anaconda, Windows Store, pyenv, homebrew, Linux repositories... They all come with different configurations and lead to problems, such as anaconda being incompatible with pip or Linux default setup not providing pip.

- it's very common to have numerous and diverse python installed on your machine. Different versions, different distributions, sometimes embeded Python... Now it's your job to know which Python you start and how. But people have no idea you should use "py -3.10 -m pip --user install black" and not "pip install black" on Windows if you want a clean install at the system level. Yes, this weird command is the only clean way to do it outside of a venv. I'm not making it up.

- the variation between the different ways to run Python and the commands lead to confusion. "python", "python3.10", "py -3.10", "pip", "pip3", "python -m pip", "sudo pip"... There are many wrong ways to do it, and not many good ones, and it all depends of the distribution, the OS, and wether your are in a venv or not.

So how does that affect packaging? Well, people will get a lot of "command not found", and so they will google and paste commands until one works. There is too much to know, and they want a solution, not learning the whole graph of possibilities. Eventually they do something that seems to work, but they probably installed with admin rights, or in anaconda with pip, or in the wrong Python version. On import or on run, something will then break. Maybe now. Or worse, maybe in one month. And they have no way to understand why.

What's the short term solution for you?

- Always use the official installer on Windows and Mac. Don't use anaconda, don't use homebrew, don't use pyenv.

- If you have to use Anaconda (I would advise not to), don't use virtualenv and pip, use their own tool: conda and the anaconda prompt. You will be limited in packages, but it will work. Don't mix conda and pip.

- On Linux, you will have to figure out what magic packages you need to install to get a full Python setup, because linux distributions authors don't provide a full Python by default. It's different for each distribution and Python version sometimes. It must include pip, venv, setuptools, tkinter, ssl and headers. Good luck.

- Don't use Python outside of a virtualenv. Ever. I know how to do that correctly. You probably don't. So don't. Always create a venv for each project you work on, and activate it before doing anything. From there, you can use pip and Python in peace. E.G: Don't install black or jupyter outside of a venv. You want to use poetry? Don't let it create your venv. Create one, activate it, then install poetry with pip, the use poetry from there. I'm not kidding.

- On Windows, the command is "py -3.10 -m venv name_of_your_venv" to create it, with "-3.10" to be replaced with the version of Python you want. On Mac and Linux, the command is "python3.10 -m venv name_of_your_venv". Yes. Not a joke.

- To activate it, on Windows it will be name_of_your_venv\Scripts\activate", on Mac and Linux, "source name_of_your_venv/bin/activate.sh".

Any variation on this (including using -m or pipx, which I used to recommend but don't anymore) assumes you know what you are doing.

What's the long term solution?

I give some hints here:

https://twitter.com/bitecode_dev/status/1567790837848743937

And meanwhile, take the official survey on packaging, this will help pin things down for the team:

https://www.surveymonkey.co.uk/r/M5XKQCT


Definitely conda is more problem and less solution.


So... zombie cells cause by pollution?


If you work for one month and you are at the mercy of the other party to decide if they need to pay you, it's not any more easy money than the lottery.


I've definitely been there! Finding P1s for full read/write access and then seeing the report downgraded to a P3, and having to have the platform arbitrate and bump it back to P1.

However, it was this experience of mine as a part-time security researcher that actually led to us creating the bug bounty program for TigerBeetle's consensus.

For example, if you're looking at another database and find a correctness bug, there might not be a bounty program at all. Whereas with TigerBeetle, there hasn't been a single valid report that we haven't awarded, at least so far.

It's also why we were careful to rather be upfront and explicit about scope, than disappoint anyone after the fact.

And we recognize that consensus is hard and takes time to learn, hence the $8192 award for correctness finds.

That said, I hope you can see from the leaderboard that we've been generous. For example, Alex Miller found a bug in Apple's O_DSYNC and we nevertheless awarded $1024 because it was such a great find (Apple thought so too!).


$1024 is half my daily rate. I doubt someone find "a great find" in 4 hours. And not even time that is guaranteed to be paid.


I think they are pretty open about what they can afford to pay. If you have self selected to be worth $2000 a day due to expertise or necessity, then you know that you're not a good fit for them.

However many people are willing to put in the work, so why are you so critical of their program.

Do you think they are taking advantage of people who should charge more? Or do you think they will not get anyone good for such a low rate, and thus fool themselves into thinking they are secure?


To be clear, this was out of scope of the bounty, it was a bug in Apple, that TB awarded anyway.


Even before talking about the typing system itself, annotations existed by themselves and are not attached to any behavior in Python: they simply hold metadata.

When Python 3 came out, annotations were indeed added to the language, but no use case was decided for them. It was an experiment to see what the community would do with them instead of forcing a use case base on what the core devs would think would be useful.

In order to allow this, annotations where permitted to be any legal python expression.

Nowadays, type hints have won: annotations are widely used to add static type checking in Python. But the way the language treat them have not changed much appart from becomming lazy. While mypy cannot understand dynamic Python expressions in them, those are nevertheless legal syntax, and will result in correct metadata at runtime.

Therefore, were the typing system not turing complete, the annotations would have been anyway by nature of being basically an exec().


That’s not at all what’s going on here. While the Python interpreter permits and executes arbitrary expressions as annotations, Python static type checkers (mypy, Pytype, Pyre, Pyright) do not. This isn’t just an implementation limitation; the typing specifications (PEP 484 and successors) don’t allow it. For example, although these two definitions evaluate to the same result at runtime:

    def f() -> int: return 1
    def f() -> (lambda: int)(): return 1
only the first is correctly typed by PEP 484. mypy rejects the second with “error: Invalid type comment or annotation”.

That’s the whole idea of static type checking: in return for restricting yourself to a controlled subset of the language that isn’t “too dynamic”, you gain the ability to type-check your code without running it.

The result of this paper is that one can nonetheless abuse subtyping to perform Turing-complete computation in the type checker. The computation doesn’t happen when your code is executed (annotations have no effect on the Python interpreter at runtime unless your program chooses to inspect them), it happens when you run mypy on it.

This is generally considered undesirable, because we want type checking to be decidable, and Turing-complete computation is not.


> That’s the whole idea of static type checking: in return for restricting yourself to a controlled subset of the language that isn’t “too dynamic”, you gain the ability to type-check your code without running it.

Eh, you are mixing up distinctions.

All that's required of _static_ typing is that your static type specification finishes running before you start your program.

Whether that static type specification language is Turing complete or not is a completely separate issue. And, yes, you are right that it's generally useful for that language to be decidable. But see eg Haskell's UndecidableInstances.


That's why my comment litterally starts with "even before talking about the typing system itself".


I’m sure that you understood the important differences, but there are likely many readers who didn’t (many more programmers are familiar with Python than with mypy). When you write “even if we don’t have P, we’d have Q anyway”, the reader is left with the impression that P and Q are somehow interchangeable or equivalent or comparable—that P is unimportant in the presence of Q. I think it’s instructive to point out that this impression would be mistaken, and the paper’s result of P is interesting and consequential.


What other use cases did the community come up with?


Dependency injections, foreign-language bridges, data validation and parsing, etc.

But thanks to how we can introspect annotations, we could have a specialized syntax for types that would be also usable for those things, while the other way around would be harder.

So it's nice we went the type hints road, because you can now define types, and yet have wonderful libs like pydantic, fastapi, typer, taichi and so on.


Pydantic typing differs from mypy typing. Optional[T] means something different under the two.

I designed typedload before pydantic was a thing and I made it with mypy in mind already, and with the idea of using the standard library, not defining my own dataclass.

Unfortunately mypy has a limitation so you can't make a function that takes Type[T] and returns T. You can define it but it won't work for all types. There is still discussion on how to provide this feature.


Yeah this ones is really annoying with default values.


Cool, any examples of libraries that use it for dependency injection or FFI or validation?


Dependency-injector uses it for exactly this purpose

    @inject
    def main(service: Service = Provide[Container.service]) -> None:
        ...

FastAPI also does a similar sort of trick, for DI and also request parsing.

https://python-dependency-injector.ets-labs.org/introduction...


django-ninja and fastapi does dep injection, parsing and validation based on annotations.

taichi does ffi with their own dsl, cython with c.

I you haven't tried pydantic or sister projects (django-ninja, fastapi and typer), this is a good place to start. They bring a breath of fresh air and are quite fun to use.


Specifically it's the underlying pydantic that does validation based on annotations

https://pydantic-docs.helpmanual.io/


Indeed.

The dep injection is fastapi and typer code though, and quite tied to it. So it's worth mentioning that somebody is attempting (quite successfully from the look of it) to create a generic lib of dep injection using annotation named DI, inspired by those libs: https://github.com/adriangb/di


I always find it very funny that fastapi has that name when it relies on pydantic that is one of the slowest libraries in that space :D


It's not one of those that you listed, but the @dataclass decorator uses type hints to auto-generate classes and class constructors (__init__ method) for you.

An example:

    @dataclass
    class HackerNewsPost:
        title: str
        author: str
        votes: int = 0
This will generate a constructor that looks like this:

    def __init__(title: str, author: str, votes: int = 0):
        self.title = title
        self.author = author
        self.votes = votes
This is pretty useful, writing __init__ boilerplate has been a common thing in my experience using Python, and it makes code more compact and readable.

It can do some other things too!

https://docs.python.org/3/library/dataclasses.html


For FFI, there is a pure python mode in cython:

  https://cython.readthedocs.io/en/latest/src/tutorial/external.html


I used them at runtime to define the types for my (absolutely stupid) library that takes a Python function and replaces its contents with a C call.

You define a Python function interface but populate it with a string of C code.


Wait, so like this?

    def add(x: int, y: int) -> int:
        """ // c code
            int x, y
            return x + y
        """
And it compiles it and turns it into a C extension?

That's...horrifying yet amazing.


That might be a worthwhile addition to the suggestion at https://github.com/faster-cpython/ideas/issues/453


Combined with the cog (https://nedbatchelder.com/code/cog) to have the c declaration in a proper separate c file, but included in the docstring, you could have the best of both worlds.


That's a neat hack!


There's a library for simulating the Excel calculation engine in Python. It uses annotations to hint functions, but according to Excel's type system, not Python's. Then at runtime, those hints are used to do conversion and promotion based on how Excel does it, so you can do e.g. the equivalent of "1" + 1 and get 2.

It's either xlcalculator or pycel, one of the two. I forgot which one does it that way.



Things like pydantic and fastapi where the type annotations are used for run time argument validation and interface generation.


Does this still work with Python 3.10 changes removing runtime evaluation of type annotations? I seem to recall this screwed me when I was writing something for runtime usage of types.


They postponed that change.

It remained enabled with a future and while it kinda works, it doesn't work in all cases. Non top level definitions can never be retrieved basically.


There was a heated debate because it broke things at the time, but now it works, yes.


Oh it looks like they actually pushed back the PEP 563 changes..

From this thread [0] it does not look open and shut that it will work once these changes go into effect.

[0]: https://github.com/pydantic/pydantic/issues/2678


we still don't know what's going to happen with type hints in future though, but I'm sure things won't break as they might have broken if they changed the default behaviour


Creating argument parsers for CLIs: https://pypi.org/project/argtyped/

I've seen similar (in-house) to generate HTML forms also.


I've created strawberry graphql, it uses type hints to generate GraphQL schemas


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: