Hacker Newsnew | past | comments | ask | show | jobs | submit | more BiteCode_dev's commentslogin

> Can you install any script from arbitrary urls using pip?

You don't need to. Download any module or packgage, put it next to your code, and you can import it. There is nothing to do. There is not even need for metadata.

But yes, pip can install packages from arbitrary URLS, including git repos, if they contains the right metadata.

What's more, with something like shiv (https://github.com/linkedin/shiv), you can actually bundle those dependancies into one zipapp (an official python feature, see https://docs.python.org/fr/3/library/zipapp.html), which is basically a zip containing your entire project that you can just execute like if it were one py file.


I've felt exactly that while testing Vite (https://github.com/vitejs/vite) this morning from Evan You, author of VueJS.

Standard install, nothing to configure, instant hot reload + modern code and tooling yet no perceivable latency for anything.

It felt like suddenly I could enjoy front end dev.

So I get the appeal of Deno.

But I know this ability to put arbitrary URLs as dependancies in the code are going to be terribly missused and abused.

I can see all the tutorials that will start including them, the bad architecture decisions that will spread them accross all files, the interns that will use localhost there, the typo and domain squatting or black SEO, the rise of people just telling you to put an URL from their site (the same that tell you to curl that bash script), the split from npm, the hardship of scanning/mapping/updating dependancies, the mismatch betwen running and installing bitting you, the undebuggable tickets because of proxies/antivirus/firewalls, the cloud hosting potential for failures, some twisted scenario leading to a DoS...

I know it will cause horrible technical debt down the road.

My spider dev sense is tickling and I'm not going to touch this tech for 5 years, until I either see my prediction unfold, and be completly disproved by its success (but I doubt it, I've been really good in the last 10 years at spotting bad technical choices).

There is a reason we have been using separation of concerns since the down of time. Declaring dependencies, installing them and importing them are 3 different concerns. They should not mix.


For the importing from URLs issue you mentioned, it would be good to ask Golang developers they’ve encountered the kind of technical debt you’re concerned about.

I’ve done some Go development, but not enough to really call myself a Go developer so I don’t think I’m qualified to comment one way or the other.

For what it’s worth, I stuck to a policy of only importing from GitHub and forking a repo if I really wanted be sure it would never disappear. If I were doing it today I’d probably expand my list acceptable import sources to include GitLab and Bitbucket as well.


I am not expert on Deno. But what other frameworks and browsers are doing is to include a hash in the URL. E.g. https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

Should not Deno to force the same behaviour to ensure there is less attack surface thru dependencies? I see Deno is taking security seriously.


Sure, this will remove one problem: type/domain squatting.

All the other ones remain.

I can't wait for Stackoverflow spamming to get the scammers answer with a legit solution + url + hash at the top of Google search when typing the name of the package.

Before that, you'd install from npm.

But now, any URL is fair game.

I'll get the pop corn.


You can easily distinguish them:

    if isinstance(msg, str)
So I don't think that's a good argument for not accepting iterables of strings in str methods. Things like replace() would benefit a lot and it's not that hard to do, you can even accept regexes optionally: https://wonderful-wrappers.readthedocs.io/en/latest/string_w...

I agree that iterating on string is not proper design however. It's not very useful in practice, and the O(1) access has other performance consequences for more important things.

Swift did it right IMO, but it's a much younger language.

I also wish we stole the file api concepts from swift, and that open() would return a file like object that always gives you bytes. No "b" mode. If you want text, you to open().as_text(), and get a decoding wrapper.

The idea that there are text files and binary files has been toxic for a whole generation of coders.


The issue is that

    if isinstance(msg, str)
will clutter code that is otherwise clean. A single type has to be specially handled, which sticks out like a sore thumb.

As a second point, do you have more on your last sentence? ("The idea that there are text files and binary files has been toxic for a whole generation of coders."). I have been thoroughly confused about text vs. bytes when learning Python/programming.

The two types are treated as siblings, when text files are really a child of binary files. Binary files are simply regular files, and sit as the single parent, without parents itself, in the tree. Text files are just one of the many children, that happen to yield text when their byte patterns happen to be interpreted using the correct encoding (or, in the spirit of Python, decoding when going from bytes to text), like UTF8. This is just like, say, audio files yielding audio when interpreted with the correct encoding (say MP3).

Is this a valid way of seeing it? I have to ask very carefully because I have never seen it explained this way, so that is just what I put together as a mental model over time. In opposition to that model, resources like books always treat binary and text files as polar opposites/siblings.

This leads me to the initial question of whether you know of resources that would support the above model (assuming it is correct)?


That sounds completely like a correct way to look at it. I'd put "stream of bytes" and "seekable stream of bytes" above files, but that's just nitpicking.

For me the toxic idea about text files is that they're a thing at all. They're just binary files containing encoded text, without any encoding marker making them an ideal trap. Is a utf16 file a text file? Is a shift-jis file a text file? Have fun guessing edge cases. We've already accepted with unicode that the "text" or letters are something separate from the encoding.


Totally agree that everything should be a byte stream. Even with Python 3.x text files are still confusing - if you open a UTF-8 file with a BOM in the front as a text file - should that BOM be part of the file contents, or transparently removed? By default, Python treats it as actual content, which can screw all sorts of things up. In my ideal world, every file is a binary file, and that if you want it to be a text file - just open it with whatever encoding scheme you think appropriate (typically UTF-8).

If you don't know the Encoding? Just write a quick detect_bom function (should be part of the standard library, no idea why it isn't) and then open it with that encoding. I.E.:

   encoding = detect_bom(fn)
   with open (fn, 'r', encoding = encoding) as f:
      ...
That also has the benefit of removing the BOM from your file.

Ultimately, putting the responsibility for determining the CODEC on the user at least makes it clear to them what they are doing -opening a binary file and decoding it. That mental model prepares them for the first time they run into, say, a cp587 file.

I understand why Python doesn't do this - it adds a bit of complexity - though you could have an "auto-detect" encoding scheme that tried to determine the encoding schemes, and defaults to UTF-8 - not perfect, as you can't absolutely determine the CODEC of a file by reading it - but better than what we have today - where your code crashes when you have a BOM that upsets UTF-8 decoder.

I finally wrote a library function to guess codecs and read text files, inspired by https://stackoverflow.com/a/24370596/1637450 and haven't been tripped up since.

But Python does not make it easy to open "text" files - and I know data engineers who've been doing this for years who are still tripped up.


Chardet, written by mozilla, already detect encoding if you need such thing.


The open() API is inherited from the C way, where the world is divided between text files and binary files. So you open a file in "text" mode, and "binary" mode, "text" being the default behavior.

This is, of course, utterly BS.

All files are binary files.

Some contains sound data, some image data, some zip data, some pdf data, and some raw encoded text data.

But we don't have a "jpg" mode for open(). We do have higher API we pass file objects to in order to decode their content as jpg, which is what we should be doing to text. Text is not an exceptional case.

VSCode does a lot of work to turn those bytes into pretty words, just like VLC into videos. They are not like that in the file. It's all a representation for human consumption.

The reasoning for this confusing API is that reading text from a file is a common use, which is true. Espacially on Unix, from which C is from. But using a "mode" is the wrong abstraction to offer it.

If fact, Python 3 does it partially right. It has a io.FileIO object that just take care of opening the stuff, and a io.BufferedReader that wraps FileIO to offer practical methods to access its content.

This what what open(mode="b") returns.

If you do open(mode="t"), which is the default, it wraps the BufferedReader into a TextStream that does the decoding part transparently for you, and returns that.

There is an great explanation of this by the always excellent David Beazley: http://www.dabeaz.com/python3io_2010/MasteringIO.pdf

What it should do is offering something this:

    with open('text.txt').as_text():
open() would always return BufferedReadfer, as_text() would always return TextStream.

This completly separates I/O from decoding, removing confusion in the mind of all those coders that would otherwise live by the illusionary binary/text model. It also makes the API much less error prone: you can easily see where to the file related arguments go (in open()) and where to text related arguments go (in as_text()).

You can keep the mode, but only for "read", "write" and "append", removing the weird mix with "text" and "bytes" which are really related to a different set of operations.


Let’s be clear here that the fault is not with Python but with Windows.

Python uses text mode by default to avoid surprising beginners on Windows. If you only use Unix-like OSs you will never have this problem.


The problem is not "text mode by default". The problem is that the API offers a text mode at all.

Opening a file should return an object that gives you bytes, and that's it.

This "mode" thing is idiotic, and leak a low level API that makes no sense in a high level language with a strong abstraction for text like Python.

Text should decoded from a wrapping object. See my ohter comments.


Splitting it into two parts like that would make seek() kind of funky, but I suppose it is already.


Sadly, there is no possible migration path. Because text is the default "mode".


How would this work

    with open('text.txt', 'w').as_text():


    with open('text.txt','w').as_text() as f:
       f.write("text")


it's just too weird and open-ended.

the next thing will be a bunch of "open" functions:

   with open_binary("filename") as f:
       ...


    with open_text("filename") as f:
        ...
How do I open these files in writeable mode?

    with open_text("filename").writeable() as f:
        ...
This is getting absurd.


Ultimately this 'frustration' is always caused by loose typing/inexistent data model and not by the iterability of strings itself.


If that bothers you, use sigle dispatch.


> The idea that there are text files and binary files has been toxic for a whole generation of coders.

It really is nonsense isn't it? Its like asking a low level api for opening files as a .doc, or as a pdf. Why would that be part of the file io layer?


Well, I guess it’s easy to argue that it’s so common that beginners would expect to open files as text. You can see how it would evolve that way.

Now I’m more familiar with it I’m careful to be explicit with the decoding when using text to make it super obvious what’s going on.


I suspect it’s also to do with Python’s history as a scripting language. Because of Perl’s obvious strengths in this area, any scripting language pretty much has to make it very easy to work with text files. Ruby does something similar for instance.

Even languages like Java now recognise the need to provide convenient access to text files as part of the standard API, with Files.readAllLines() in 7, Files.lines() in 8, and Files.readString() in 11.


You can make it easy to deal with text file without lying to your API users. open("foo", mode="t") could become open("foo").as_text().

Besides, Python has pathlib now, which allows you to do Path("foo").read_text() for quick and dirty text handling already.


My first mistake I made as a beginner was dumping a bunch of binary data as text. Something would happen in the way and not the whole data would be written because I was writing it in text mode.

It just never appeared to me that the default mode of writing the file would _not_ write the array I was passing it.

It’s much more important for beginners to be able to learn clear recipes rather than having double standards with a bunch of edge cases.


I’ve done worse. Using MySQL from php and not having the encoding right somewhere along the way so all my content was being mojibaked on the way in and un-mojibaked on the way out so I didn’t notice it until deep into a project when I needed to extract it to another system.

EDIT thanks, I knew that didn't look quite right. "Mojibaked" - such a great term.


The term is actually "Mojibake", not "emoji baked". https://en.wikipedia.org/wiki/Mojibake#Etymology


More to the point, it's so common that it ought to be supported out of the box by any decent programming language, the same way you'd expect any language to support IEEE floats. That doesn't mean the mechanism for it shouldn't be (effectively) textfile(file("foo.txt")), though.


It's hard to strike a balance for this.

For many FP programmers, the lambdas are not nearly magic enough.

In Swift, you can do something like this :

    reversedNames = names.sorted(by: { $0 > $1 } )
It's a closure expression, and is an anonymous block of code that will magically bind parameters passed to it to numbered variables.

So you see the position Python is in: it has to balance a bit of magic for the power user, and yet not too much for the casual user.

It's a very delicate exercice, and it receives a lot of critics for it.

An F# or Lisp dev comming to Python will complain that it's not expressive enough.

A geographer comming to Python will struggle reading advanced code.

Yet we have to catters to all of them, give the huge Python popularity and it's goal to be "the second best language for everything".

I think Guido did a very decent job at it, although he gets a lot of heat for it. People don't like to hear "no" when they ask for a poney.

And the lambda expression is one of the most controversial decisions. Beginners have a hard time with it, but professional coders may snap when they hear it's limited to one line.


People like this need to be put into managing a team of visual basic devs at an obscure local DMV IT department, and force them to make projects after projects until they realize that if they want something done in those situations, they have to get dumber.


Given how heated was the debate about those, it's good we didn't try to go too fast with it.

I'm full of hopes for this feature, but it's going to be a slow hard work, and we'll only rip the benefit on the long run. So no use to rush it.

I feel like we rushed asyncio and type hints, and it took years to make them usable after they were introduced.


Strings are immutable in Python, and all strings operations return new strings, including all string methods.

So there is no possible confusion.


If I have a string x = "this is a very long string..." and do y = x[:10], then it's a whole new string? If x is near my memory limits, and I do y = x[:-1] will it basically double my memory usage? Is that what you meant by every string is a new string?


> If I have a string x = "this is a very long string..." and do y = x[:10], then it's a whole new string?

Yes. And doing otherwise is pretty risky as the Java folks discovered, ultimately deciding to revert the optimisation of substring sharing storage rather than copying its data.

The issue is that while data-sharing substringing is essentially free, it also keeps the original string alive, so if you slurp in megabytes, slice out a few bytes you keep around and "throw away" the original string, that one is still kept alive by the substringing you perform, and you basically have a hard to diagnose memory leak due to completely implicit behaviour.

Languages which perform this sharing explicitly — and especially statically (e.g. Rust) — don't have this issue, but it's a risky move when you only have one string type.

Incidentally, Python provides for opting into that behaviour for bytes using memory views.


> it also keeps the original string alive, so if you slurp in megabytes, slice out a few bytes you keep around and "throw away" the original string, that one is still kept alive by the substringing you perform, and you basically have a hard to diagnose memory leak due to completely implicit behaviour.

You can get around that with a smarter garbage collector, though. On every mark-sweep pass (which you need for cycle detection even if you use refcounts for primary cleanup), add up the number of bytes of distinct string objects using the buffer. If it's less than the size of the buffer, you can save memory by transparently mutating each substring to use it's own buffer. If it's not, then you actually are saving memory by sharing storage, so you should probably keep doing that.


The cpython mark-and-sweep garbage collector does very little on purpose. It basically only gets involved for reference cycles. Anything else is dealt with by reference counting. This way you prevent long GC pauses.


True, but that's by no means a inherent characteristic of garbage collectors, or even garbage collectors operating on objects of a python implementation in particular.


True, but that is the current philosophy behind the garbage collector in the most used python implementation and it's unlikely to change.


> You can get around that with a smarter garbage collector, though.

That complexifies the GC (already a complex beast) as it requires strong type-specific specialisation, more bookkeeping, and adds self-inflicted edge cases. Even the JVM folks didn't bother (though mayhaps they did not because they have more than one garbage collector).


If x is near your memory limits, and you do y = x[:-1], you will get a MemoryError :)

For those situations, bytes() + memoryview() or bytearray() can be used, but then you are on your own.


Huh, I've had a wrong understanding of that for over a decade! TIL, thanks.


Hey!

https://xkcd.com/1053/

And honestly, I would be rich if I got a dollar every time a student does this:

    msg.upper()
Instead of:

    msg = msg.upper()
And then call me to say it doesn't work.


Scheme does the right thing here (by convention), that mutating procedures end with a bang: (string-upcase str) returns a new string, whereas (string-upcase! str) mutates the string in place.

The details for mutation of data in scheme go beyond that, though. Sometimes procedures are "allowed but not required to mutate their argument". Most (all?) implementations do mutate, but it is still considered bad for to do something like:

    (define a (list 1 2 3))
    (append! a (list 4))
    (display a)
As append! returns a list that is supposed to supercede the binding to a. Using a like that "is an error", as a valid implementation of append! may look like this

    (define append! append)
Which would make the earlier code snippet invalid.


IMO, this is a defect in the language: the lack of a "must_use" annotation or similar. If that annotation existed, and the .upper() method was annotated with it, the compiler could warn in that situation.


But you are free to do

  if title == user_input.upper():
That is, you convert a string to upper without binding the result to a name. You just use it in-place and discard the result, which is fine.

With compiler, you mean mypy or linters?


That's still "using" the resulting value for a comparison. CPython isn't an optimizing compiler, or it would completely remove the call to upper().

    >>> def up(v):
    ...     v.upper()
    ...
    >>> dis.dis(up)
    2           0 LOAD_FAST                0 (v)
                2 LOAD_METHOD              0 (upper)
                4 CALL_METHOD              0
                6 POP_TOP
                8 LOAD_CONST               0 (None)
                10 RETURN_VALUE

    >>> def up(v):
    ...     if v.upper() == "HelloWorld":
    ...        return True
    ...
    >>> dis.dis(up)
    2           0 LOAD_FAST                0 (v)
                2 LOAD_METHOD              0 (upper)
                4 CALL_METHOD              0
                6 LOAD_CONST               1 ('HelloWorld')
                8 COMPARE_OP               2 (==)
                10 POP_JUMP_IF_FALSE       16

    3          12 LOAD_CONST               2 (True)
                14 RETURN_VALUE
            >>   16 LOAD_CONST               0 (None)
                18 RETURN_VALUE
Notice in the first example, right after CALL_METHOD the return value on the stack is just immediately POP'd away. The parent is saying that when you run `python example.py` CPython should see that the return value is never used and emit a warning. This would only happen because `upper()` was manually marked using the suggested `must_use` annotation.


He meant that writing a line of code with only contents:

    msg.upper()
should trigger a warning as this clearly doesn't do anything.


Python is interpretted, not compiled, and completly dynamic. You cannot check much statically.

In fact, any program can replace anything on the fly, and swap your string for something similar but mutable.

It's the trade off you make when choosing it.


I agree, there’s no way to issue a warning about a bare `s.upper()` at compile time. I wonder if it would be possible at runtime?


Don't think so, Python doesn't really care if you dispose of the results of an expression. Think about the problems you'd have with ternaries.


Ternaries don't discard results that are generated, they are just special short-circuiting operators;

  x if y else z
Is effectively syntax sugar for:

  y and x or z
Nothing is discarded after evaluation, one of three arms is never evaluated, just as one of two arms of a common short-circuiting Boolean operator often (but not always) is not. That's essentially the opposite of executing and producing possible side effects and then discarding the results.


What's the problem with ternaries?


One of the two possible sub-expressions isn't used.


It's also not evaluated. There is no discarding, so there would be no problem.


What is this “compile time” you speak of?


When the Python source code is compiled into bytecode.


That byte code is then interpreted at runtime, so the meaning of s.upper() could change. What something does, when it’s parsed, is not fixed.

You can definitely catch most cases at runtime. I’ve done something like this, in an library, to catch a case where people were treating the copy of data as a mutable view.

    interface[address][slice] = new_values # fancy noop
Where a read, modify, write was required:

    byte_values = interface[address]
    byte_values[slice] = new_values
    interface[address] = byte_values
It would log/raise a useful error if the there was no assignment/passing of the return value.


> Python is interpretted, not compiled, and completly dynamic. You cannot check much statically.

The existence of mypy and other static type checkers for Python disproves that; given their existence, warning of an expression producing a type other than “any” or strictly “None” was used in a position where it would neither be passed to another function or assigned to a variable that is used later should be possible. Heck, you could be stricter and only allow strictly “None” in that position.


so what are these annoying pyc files about?


> And honestly, I would be rich if I got a dollar every time a student does this:

> msg.upper()

> Instead of:

> msg = msg.upper()

> And then call me to say it doesn't work.

On this, isn't the student's reasoning sensible? E.g. "If msg is a String object that represents my string, then calling .upper() on it will change (mutate) the value, because I'm calling it on itself"?

If the syntax was upper(msg) or to a lesser extent String.upper(msg) then the new-to-programming me would have understood more clearly that msg was not going to change. Have you any insights into what your students are thinking?


> String.upper(msg)

That was the original syntax [0], before the string functions became methods. I agree that a method more strongly implies mutation than a function does.

Also, for consistency with list methods like `reverse` (which acts in place) and `reversed` (which makes a copy), shouldn’t the method be called `uppered`?!

[0] https://docs.python.org/2/library/string.html#deprecated-str...


'uppercased'


Ah, of course.

Also, it looks like that’s the name that Swift uses.


A student don't know anything about mutability, and since Python signatures are not explicit, there is no way to know they have to do that.

It's just something to be told. A design decision, like there are thousands to learn in IT, that you just can't guess.


Yes. Although you can use `islice` from itertools to get around this problem, when a problem.


Slicing in Python always create a new object. You can test it with a list of integers..


My favorite example of something similar to this, since you brought it up:

  >>> a = [254, 255, 256, 257, 258]
  >>> b = [254, 255, 256, 257, 258]
  >>> for i in range(5): print(a[i] is b[i])
  ...
  True
  True
  True
  False
  False
In Python, integers in the range [-5, 256] are statically constructed in the interpreter and refer to fixed instances of objects. All other integers are created dynamically and refer to a new object each time they are created.


Leaky abstractions at its finest.


I mean, people should be using `==` for this. The fact that `is` happens to work for small numbers is an implementation detail that shouldn't be relied upon.


Absolutely. But because it does work they might start using it without knowing it's wrong, then be surprised when it doesn't work. Python has other areas where the common English definition of a word leads to misunderstandings about how they're to be used.


Though "when" the object is created isn't always so straightforward:

  >>> x = 257
  >>> y = 257
  >>> x is y
  False
  >>> def f():
  ...     x = 257
  ...     y = 257
  ...     return x is y
  ...
  >>> f()
  True
The lesson being that `is` is essentially meaningless for immutable objects, and to always use `==`.


> `is` is essentially meaningless for immutable objects

OTOH it’s recommended to use `is` rather than `==` when comparing to “singletons like `None`”.

https://www.python.org/dev/peps/pep-0008/#programming-recomm...


What is the rationale behind this? '==' works all the time, and 'is' only works sometimes. Using 'is' wherever possible requires the user to know some rather arbitrary language details (which objects are singletons and which are not), wheras '==' will always give the correct answer regardless.


Classes can overload `==`:

    class C:
        def __eq__(self, other):
            return True

    print(C() == None)  # True
    print(C() is None)  # False


> Slicing in Python always create a new object.

It always creates a new object but it doesn't necessarily copy the contents (even shallowly).

For instance slicing a `memoryview` creates a subview which shares storage with its parent.


It'll always create a new object but my understanding is that at least in numpy the new and old object will share memory. Am I wrong there too?


Correct. In Numpy the slices are views on the underlying memory. That’s why they’re so fast, there’s no copying involved. Incidentally that’s also why freeing up the original variable doesn’t release the memory (the slices are still using it).


CPython is pretty terrible. Numpy has the concept of views, cpython doesn’t do anything sophisticated.


> Slicing in Python always create a new object.

Since you said "always", I'll be "that guy"...

For the builtin types, yeah. But not in general; user code can do whatever it wants:

    class MyType:
      def __getitem__(self, *args):
        return self
    my_obj = MyType()
    my_other_obj = my_obj[3:7]
    assert my_other_obj is my_obj


Yes.


If you want a promotion, move to another job. Most companies make it incredibly slow and painful to evolve in their own walls, while giving big rewards to new comers.

Unless you happen to work at the legenday ones that do care about you, don't play according to their rules. Those are here for their benefit, not yours.

Remote or not.


This. You will get a 20-100% raise in a new job, and peanuts for staying with a company.

RSU refreshers might change the equation.


Highly recommend this. I'm a consultant, and I got offered a $12,000 raise after working with my company for two years and working for three seperate extremely high value clients.

I put in my two weeks notice the day I received an offer from a competing company nearby that I'd applied to for a total renumeration package roughly equivalent to a $45,000 raise at my old job.

Shop around. Always. If you don't, you'll never know.

I like to think of it as someone losing out on free stuff. It costs nothing to write up an updated resumé and throw it into the abyss of companies that are hiring. Possible gain by doing this? A lot. Possible loss? Maybe an hour of your time that you spent applying to the job.


There is no Python 4 planned for now.

Python broke compat once in 25 years and gave 13 years to migrate.

It's a very conservative language.


Python breaks compatibility often on minor point releases. Only once as big as 3.0.0 but it happens regularly.

I argued on the list that these should be kept for major releases for planning reasons but they appear to be convinced it is too hard.


Can you provide an example of where Python has broken backwards compatibility recently between 3.x version? I'll admit (despite googling for 5 or so minutes) that i don't actually know if it does. It obviously breaks forward compatibility continuously all the time - new language features are landing, and they just aren't present in previous versions - but I don't know if I've ever run into people being tripped up by that.

I know some Python Libraries break backwards compatibility (Pandas being a big one) - but, for the most part, hasn't the language been backwards compatible since at least Python 3.4? (And possibly further back, for all I know).


On this page you'll see them. Check the "Porting to X.X" items under each release:

https://docs.python.org/3/whatsnew/index.html

Keep in mind they have deferred a number of them because of the impending EOL of Python 2.7. There are fewer breaking changes during the latter 3.X series, which should resume in 3.9 or 3.10 now that Python2 has passed on.

Here's a commonly mentioned one:

Changes in Python Behavior: async and await names are now reserved keywords. Code using these names as identifiers will now raise a SyntaxError. (Contributed by Jelle Zijlstra in bpo-30406.)

Note: I think this is a bad idea, I'd rather all these small breaking changes and parser be deferred to 4.X. But they need to be small breaking changes, of course, not a new language.


async keyword 3.5


If we look back in python history, the rolling breaking changes have been handled mostly fine, and the actual Python 3 caused a lot of pain in the ecosystem. So I hope they stay away from major versions and keep up the other things they are doing.


That was due to the scope of the breakage, not number format. A good way to handle that and maintain predictability is to constrain breaking changes, yet defer them to 4.X.


I'm struggling to think of any other language that has done something like this.

It might seem like a quibble, but it seems better to describe Python 3 as a different language versus Python 2. Newbies seem to get that.

(Or, alternatively, "How many Python 2 scripts will run on a Python 3 interpreter?" Answer: "None of them.")


First, there are not a lot of interpretted (not compiled, that's another matter entirely) languages that are as old as Python.

And there are really few that are even near Python popularity, or used with such diversity as Python.

I mean, you can get away with keeping AWK the way it was 2 decades ago, nobody is going to use it for for machine learning or to teach computing in all universities in the world on 3 operating systems, utilizing C extensions, or processing Web API.

Among the few that would even compare, there are the ones that have accumulated so much cruft that they became unusable from today's standard (E.G: bash). Then you have those who have done like Python (E.G: perl 6). The ones that just tried and failed (PHP 6). The ones that broke compat and told everybody move or die (Ruby in a point release, gave basically 2 years). And the ones that created a huge pile of horror they called full stack to keep going (E.G: JS). Also those that got hijacked by vendors and just exploded in miriads of proprietary syntaxes (E.G: SQL) or completely new concepts (E.G: lisp).

At least, in Python you CAN write Python 2/3 compatible code, and you have a LOT of tooling to help you with that, or migrating.

So, yes, the Python 2 -> 3 transition could have been better. Insight is 20/20.

But I'm struggling to think of any other language in a similar situation that has done better.


Hindsight 20/20, not insight.


Thanks. French here. Can't edit anymore though


Ruby did something like this around the same time Python did. Ruby's was a bit smaller, but overall a roughly similar amount of breaking changes. They forced you to think about encodings more with Strings, they changed the signatures of several operators, they changed some of the syntax for case statements, they drastically changed the scope rules for block variables, they restructured the base object hierarchy, etc. In both cases, it was a deliberate decision to make a clean break. I think Ruby's big break didn't make as big a schism mainly because Rails was very supportive, and Rails holds an enormous amount of influence in the Ruby world.

If Python 3 had been introduced as a separate language, I'm pretty sure everyone would have said "Why isn't this just called Python 3? It's 99.9% the same as Python and it's by the same people and they're deprecating Python in favor of it."


> Or, alternatively, "How many Python 2 scripts will run on a Python 3 interpreter?" Answer: "None of them."

That's obviously untrue. For example, consider the following Python 2 script:

    with open('a.txt', 'r') as a, open('b.txt', 'w') as b:
        for line in a:
            b.write(line)
It works identically on a Python 3 interpreter, and it doesn't even use "from __future__ import ...".


Not quite. If your a.txt contains utf-8 byte blocks you are screwed in Python 2.


Yes, I should have said, "a vanishingly small proportion, and even then mostly by accident".

My guess is that if you sweep GitHub for Python2 code and push it into Python3, that proportion would be under one percent.


how will that number look when you first autoconvert via 2to3?

I did two migrations of >500k loc projects in an afternoon each, and admittedly some days of testing to gain confidence since there where few unit test. But i found it to be very smooth sailing.

I was very familiar with both projects, so that helped a lot.

EDIT: I also want to add that i did this using python3.5, when to ecosystem seemed to be at a sweet spot of dependencies supporting both 2 and 3 mostly. I guess if one has been waiting until now, the divide between library versions will be a lot bigger.


As a big user of logging and little to do with character coding, all of my admin/daemon stuff moved over with almost no changes necessary for 3.0 (actually ~3.3).

For some projects I did bigger refactors for 2.6/7 (exceptions) and 3.6 (fstrings).


Fair point, but:

  int main(int c,char** v)
    {
    printf("Hello, World!\n");
    return 0;
    }
doesn't make C++ the same language as C.


> Or, alternatively, "How many Python 2 scripts will run on a Python 3 interpreter?" Answer: "None of them."

That is completely false. So many libraries have support both for 2 and 3. That's code that run just as well under both interpreter.


Really? The same .py file runs under python2 and python3?

Googling quickly, I find this, which does a bit better than 2to3. I suppose one could write to a somewhat constrained intersection of Python2 and Python3, if one is willing to make at least some boilerplate changes to the original Python2 code.

https://python-future.org/overview.html

That said, if you bring a Python2 script and feed it to a Python3 interpreter, no, in general that will not work. They simply aren't the same language. Even a simple "print x" will do you in.


> The same .py file runs under python2 and python3?

Sure, as long as it doesn't contain any syntax or spellings which are incompatible between the two. That's a fairly large subset of the language.

> if you bring a Python2 script and feed it to a Python3 interpreter, no, in general that will not work. They simply aren't the same language. Even a simple "print x" will do you in.

But this will work:

    from __future__ import print_function
    print(x)
This is valid under both Python 2 and Python 3.

Also, as I said above, there is a pretty large subset of the Python language that has the same syntax and spellings in both Python 2 and Python 3, and any script or module or package that only uses that subset will run just fine under both interpreters. You are drastically underestimating both the size and the usage of this subset of the language.


Say Django 1.11, a massive amount of .py files, works completely fine under both 2 and 3. As do many other libraries.

Yes you often need some precautions like "from __future__ import" statements and sometimes libraries like `six`, but it's been perfectly normal practice for most of the last decade.


No, but it is possible to write Python 2 code than runs in Python 3.

In fact, the vast majority of popular libs had a 2/3 compatible code base for a few years.

The hard part was not the syntax in fact. It's pretty trivial: the language are not that different.

The hard part is the I/O stack, because the stdlib is very different, espacially for this part.


A lot of projects write in that style e.g. compatible with both python2 and python3, it's really common because there's so much py2 deployed (was default on centOS until very recently, still default on osx, etc.)

Nearly every py3 feature was backported to 2 you just need to write it in a compatible way. I'm seeing some drop py2 support now though. Which I'm fine with, I haven't written python2 code in maybe 6 or 7 years now.


PHP which tried to address unicode in version 6, but then abandoned it and went straight to 7. Perl, which amusingly also at version 6 decided on a huge re-write, but then just decided to rename the version as an actual new language "Raku".


Re Raku, in hindsight, this was the right call. Give a new language a new name, and the story stays simple.


Yep. Also people are not shamed and ridiculed online because they did not upgraded to new language.


It if was such a righ call, why did it die ?


Why do you think it is dead?


> Or, alternatively, "How many Python 2 scripts will run on a Python 3 interpreter?" Answer: "None of them."

That is not true though, a lot of 2 scripts will run no problem, especially if import future is used


Swift has had 3 backwards incompatible versions of larger scope in less time.


> I'm struggling to think of any other language that has done something like this.

Perl 6 (now Raku), of course, but they at least had the decency to admit it.



After that will be Python 3.11, and after that, Python for Workgroups 3.11.


Python 95 is gonna blow everyone's minds.


Yeah, but PythoNT will be around for a long time.


With semver, it's going to be too long to reach. I suggest we switch to the Firefox versioning scheme, it seems to already be near this goal.


the first release with Plug and Pep


Start me up!


You make a grown man cry...


Followed by inexplicably jumping to pithon.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: