> Can you install any script from arbitrary urls using pip?
You don't need to. Download any module or packgage, put it next to your code, and you can import it. There is nothing to do. There is not even need for metadata.
But yes, pip can install packages from arbitrary URLS, including git repos, if they contains the right metadata.
What's more, with something like shiv (https://github.com/linkedin/shiv), you can actually bundle those dependancies into one zipapp (an official python feature, see https://docs.python.org/fr/3/library/zipapp.html), which is basically a zip containing your entire project that you can just execute like if it were one py file.
Standard install, nothing to configure, instant hot reload + modern code and tooling yet no perceivable latency for anything.
It felt like suddenly I could enjoy front end dev.
So I get the appeal of Deno.
But I know this ability to put arbitrary URLs as dependancies in the code are going to be terribly missused and abused.
I can see all the tutorials that will start including them, the bad architecture decisions that will spread them accross all files, the interns that will use localhost there, the typo and domain squatting or black SEO, the rise of people just telling you to put an URL from their site (the same that tell you to curl that bash script), the split from npm, the hardship of scanning/mapping/updating dependancies, the mismatch betwen running and installing bitting you, the undebuggable tickets because of proxies/antivirus/firewalls, the cloud hosting potential for failures, some twisted scenario leading to a DoS...
I know it will cause horrible technical debt down the road.
My spider dev sense is tickling and I'm not going to touch this tech for 5 years, until I either see my prediction unfold, and be completly disproved by its success (but I doubt it, I've been really good in the last 10 years at spotting bad technical choices).
There is a reason we have been using separation of concerns since the down of time. Declaring dependencies, installing them and importing them are 3 different concerns. They should not mix.
For the importing from URLs issue you mentioned, it would be good to ask Golang developers they’ve encountered the kind of technical debt you’re concerned about.
I’ve done some Go development, but not enough to really call myself a Go developer so I don’t think I’m qualified to comment one way or the other.
For what it’s worth, I stuck to a policy of only importing from GitHub and forking a repo if I really wanted be sure it would never disappear. If I were doing it today I’d probably expand my list acceptable import sources to include GitLab and Bitbucket as well.
Sure, this will remove one problem: type/domain squatting.
All the other ones remain.
I can't wait for Stackoverflow spamming to get the scammers answer with a legit solution + url + hash at the top of Google search when typing the name of the package.
So I don't think that's a good argument for not accepting iterables of strings in str methods. Things like replace() would benefit a lot and it's not that hard to do, you can even accept regexes optionally: https://wonderful-wrappers.readthedocs.io/en/latest/string_w...
I agree that iterating on string is not proper design however. It's not very useful in practice, and the O(1) access has other performance consequences for more important things.
Swift did it right IMO, but it's a much younger language.
I also wish we stole the file api concepts from swift, and that open() would return a file like object that always gives you bytes. No "b" mode. If you want text, you to open().as_text(), and get a decoding wrapper.
The idea that there are text files and binary files has been toxic for a whole generation of coders.
will clutter code that is otherwise clean. A single type has to be specially handled, which sticks out like a sore thumb.
As a second point, do you have more on your last sentence? ("The idea that there are text files and binary files has been toxic for a whole generation of coders.").
I have been thoroughly confused about text vs. bytes when learning Python/programming.
The two types are treated as siblings, when text files are really a child of binary files. Binary files are simply regular files, and sit as the single parent, without parents itself, in the tree. Text files are just one of the many children, that happen to yield text when their byte patterns happen to be interpreted using the correct encoding (or, in the spirit of Python, decoding when going from bytes to text), like UTF8. This is just like, say, audio files yielding audio when interpreted with the correct encoding (say MP3).
Is this a valid way of seeing it? I have to ask very carefully because I have never seen it explained this way, so that is just what I put together as a mental model over time. In opposition to that model, resources like books always treat binary and text files as polar opposites/siblings.
This leads me to the initial question of whether you know of resources that would support the above model (assuming it is correct)?
That sounds completely like a correct way to look at it. I'd put "stream of bytes" and "seekable stream of bytes" above files, but that's just nitpicking.
For me the toxic idea about text files is that they're a thing at all. They're just binary files containing encoded text, without any encoding marker making them an ideal trap. Is a utf16 file a text file? Is a shift-jis file a text file? Have fun guessing edge cases. We've already accepted with unicode that the "text" or letters are something separate from the encoding.
Totally agree that everything should be a byte stream. Even with Python 3.x text files are still confusing - if you open a UTF-8 file with a BOM in the front as a text file - should that BOM be part of the file contents, or transparently removed? By default, Python treats it as actual content, which can screw all sorts of things up. In my ideal world, every file is a binary file, and that if you want it to be a text file - just open it with whatever encoding scheme you think appropriate (typically UTF-8).
If you don't know the Encoding? Just write a quick detect_bom function (should be part of the standard library, no idea why it isn't) and then open it with that encoding. I.E.:
encoding = detect_bom(fn)
with open (fn, 'r', encoding = encoding) as f:
...
That also has the benefit of removing the BOM from your file.
Ultimately, putting the responsibility for determining the CODEC on the user at least makes it clear to them what they are doing -opening a binary file and decoding it. That mental model prepares them for the first time they run into, say, a cp587 file.
I understand why Python doesn't do this - it adds a bit of complexity - though you could have an "auto-detect" encoding scheme that tried to determine the encoding schemes, and defaults to UTF-8 - not perfect, as you can't absolutely determine the CODEC of a file by reading it - but better than what we have today - where your code crashes when you have a BOM that upsets UTF-8 decoder.
The open() API is inherited from the C way, where the world is divided between text files and binary files. So you open a file in "text" mode, and "binary" mode, "text" being the default behavior.
This is, of course, utterly BS.
All files are binary files.
Some contains sound data, some image data, some zip data, some pdf data, and some raw encoded text data.
But we don't have a "jpg" mode for open(). We do have higher API we pass file objects to in order to decode their content as jpg, which is what we should be doing to text. Text is not an exceptional case.
VSCode does a lot of work to turn those bytes into pretty words, just like VLC into videos. They are not like that in the file. It's all a representation for human consumption.
The reasoning for this confusing API is that reading text from a file is a common use, which is true. Espacially on Unix, from which C is from. But using a "mode" is the wrong abstraction to offer it.
If fact, Python 3 does it partially right. It has a io.FileIO object that just take care of opening the stuff, and a io.BufferedReader that wraps FileIO to offer practical methods to access its content.
This what what open(mode="b") returns.
If you do open(mode="t"), which is the default, it wraps the BufferedReader into a TextStream that does the decoding part transparently for you, and returns that.
open() would always return BufferedReadfer, as_text() would always return TextStream.
This completly separates I/O from decoding, removing confusion in the mind of all those coders that would otherwise live by the illusionary binary/text model. It also makes the API much less error prone: you can easily see where to the file related arguments go (in open()) and where to text related arguments go (in as_text()).
You can keep the mode, but only for "read", "write" and "append", removing the weird mix with "text" and "bytes" which are really related to a different set of operations.
I suspect it’s also to do with Python’s history as a scripting language. Because of Perl’s obvious strengths in this area, any scripting language pretty much has to make it very easy to work with text files. Ruby does something similar for instance.
Even languages like Java now recognise the need to provide convenient access to text files as part of the standard API, with Files.readAllLines() in 7, Files.lines() in 8, and Files.readString() in 11.
My first mistake I made as a beginner was dumping a bunch of binary data as text. Something would happen in the way and not the whole data would be written because I was writing it in text mode.
It just never appeared to me that the default mode of writing the file would _not_ write the array I was passing it.
It’s much more important for beginners to be able to learn clear recipes rather than having double standards with a bunch of edge cases.
I’ve done worse. Using MySQL from php and not having the encoding right somewhere along the way so all my content was being mojibaked on the way in and un-mojibaked on the way out so I didn’t notice it until deep into a project when I needed to extract it to another system.
EDIT thanks, I knew that didn't look quite right. "Mojibaked" - such a great term.
More to the point, it's so common that it ought to be supported out of the box by any decent programming language, the same way you'd expect any language to support IEEE floats. That doesn't mean the mechanism for it shouldn't be (effectively) textfile(file("foo.txt")), though.
For many FP programmers, the lambdas are not nearly magic enough.
In Swift, you can do something like this :
reversedNames = names.sorted(by: { $0 > $1 } )
It's a closure expression, and is an anonymous block of code that will magically bind parameters passed to it to numbered variables.
So you see the position Python is in: it has to balance a bit of magic for the power user, and yet not too much for the casual user.
It's a very delicate exercice, and it receives a lot of critics for it.
An F# or Lisp dev comming to Python will complain that it's not expressive enough.
A geographer comming to Python will struggle reading advanced code.
Yet we have to catters to all of them, give the huge Python popularity and it's goal to be "the second best language for everything".
I think Guido did a very decent job at it, although he gets a lot of heat for it. People don't like to hear "no" when they ask for a poney.
And the lambda expression is one of the most controversial decisions. Beginners have a hard time with it, but professional coders may snap when they hear it's limited to one line.
People like this need to be put into managing a team of visual basic devs at an obscure local DMV IT department, and force them to make projects after projects until they realize that if they want something done in those situations, they have to get dumber.
If I have a string x = "this is a very long string..." and do y = x[:10], then it's a whole new string? If x is near my memory limits, and I do y = x[:-1] will it basically double my memory usage? Is that what you meant by every string is a new string?
> If I have a string x = "this is a very long string..." and do y = x[:10], then it's a whole new string?
Yes. And doing otherwise is pretty risky as the Java folks discovered, ultimately deciding to revert the optimisation of substring sharing storage rather than copying its data.
The issue is that while data-sharing substringing is essentially free, it also keeps the original string alive, so if you slurp in megabytes, slice out a few bytes you keep around and "throw away" the original string, that one is still kept alive by the substringing you perform, and you basically have a hard to diagnose memory leak due to completely implicit behaviour.
Languages which perform this sharing explicitly — and especially statically (e.g. Rust) — don't have this issue, but it's a risky move when you only have one string type.
Incidentally, Python provides for opting into that behaviour for bytes using memory views.
> it also keeps the original string alive, so if you slurp in megabytes, slice out a few bytes you keep around and "throw away" the original string, that one is still kept alive by the substringing you perform, and you basically have a hard to diagnose memory leak due to completely implicit behaviour.
You can get around that with a smarter garbage collector, though. On every mark-sweep pass (which you need for cycle detection even if you use refcounts for primary cleanup), add up the number of bytes of distinct string objects using the buffer. If it's less than the size of the buffer, you can save memory by transparently mutating each substring to use it's own buffer. If it's not, then you actually are saving memory by sharing storage, so you should probably keep doing that.
The cpython mark-and-sweep garbage collector does very little on purpose. It basically only gets involved for reference cycles. Anything else is dealt with by reference counting. This way you prevent long GC pauses.
True, but that's by no means a inherent characteristic of garbage collectors, or even garbage collectors operating on objects of a python implementation in particular.
> You can get around that with a smarter garbage collector, though.
That complexifies the GC (already a complex beast) as it requires strong type-specific specialisation, more bookkeeping, and adds self-inflicted edge cases. Even the JVM folks didn't bother (though mayhaps they did not because they have more than one garbage collector).
Scheme does the right thing here (by convention), that mutating procedures end with a bang: (string-upcase str) returns a new string, whereas (string-upcase! str) mutates the string in place.
The details for mutation of data in scheme go beyond that, though. Sometimes procedures are "allowed but not required to mutate their argument". Most (all?) implementations do mutate, but it is still considered bad for to do something like:
(define a (list 1 2 3))
(append! a (list 4))
(display a)
As append! returns a list that is supposed to supercede the binding to a. Using a like that "is an error", as a valid implementation of append! may look like this
(define append! append)
Which would make the earlier code snippet invalid.
IMO, this is a defect in the language: the lack of a "must_use" annotation or similar. If that annotation existed, and the .upper() method was annotated with it, the compiler could warn in that situation.
Notice in the first example, right after CALL_METHOD the return value on the stack is just immediately POP'd away. The parent is saying that when you run `python example.py` CPython should see that the return value is never used and emit a warning. This would only happen because `upper()` was manually marked using the suggested `must_use` annotation.
Ternaries don't discard results that are generated, they are just special short-circuiting operators;
x if y else z
Is effectively syntax sugar for:
y and x or z
Nothing is discarded after evaluation, one of three arms is never evaluated, just as one of two arms of a common short-circuiting Boolean operator often (but not always) is not. That's essentially the opposite of executing and producing possible side effects and then discarding the results.
That byte code is then interpreted at runtime, so the meaning of s.upper() could change. What something does, when it’s parsed, is not fixed.
You can definitely catch most cases at runtime. I’ve done something like this, in an library, to catch a case where people were treating the copy of data as a mutable view.
> Python is interpretted, not compiled, and completly dynamic. You cannot check much statically.
The existence of mypy and other static type checkers for Python disproves that; given their existence, warning of an expression producing a type other than “any” or strictly “None” was used in a position where it would neither be passed to another function or assigned to a variable that is used later should be possible. Heck, you could be stricter and only allow strictly “None” in that position.
> And honestly, I would be rich if I got a dollar every time a student does this:
> msg.upper()
> Instead of:
> msg = msg.upper()
> And then call me to say it doesn't work.
On this, isn't the student's reasoning sensible? E.g. "If msg is a String object that represents my string, then calling .upper() on it will change (mutate) the value, because I'm calling it on itself"?
If the syntax was upper(msg) or to a lesser extent String.upper(msg) then the new-to-programming me would have understood more clearly that msg was not going to change. Have you any insights into what your students are thinking?
That was the original syntax [0], before the string functions became methods. I agree that a method more strongly implies mutation than a function does.
Also, for consistency with list methods like `reverse` (which acts in place) and `reversed` (which makes a copy), shouldn’t the method be called `uppered`?!
My favorite example of something similar to this, since you brought it up:
>>> a = [254, 255, 256, 257, 258]
>>> b = [254, 255, 256, 257, 258]
>>> for i in range(5): print(a[i] is b[i])
...
True
True
True
False
False
In Python, integers in the range [-5, 256] are statically constructed in the interpreter and refer to fixed instances of objects. All other integers are created dynamically and refer to a new object each time they are created.
I mean, people should be using `==` for this. The fact that `is` happens to work for small numbers is an implementation detail that shouldn't be relied upon.
Absolutely. But because it does work they might start using it without knowing it's wrong, then be surprised when it doesn't work. Python has other areas where the common English definition of a word leads to misunderstandings about how they're to be used.
What is the rationale behind this? '==' works all the time, and 'is' only works sometimes. Using 'is' wherever possible requires the user to know some rather arbitrary language details (which objects are singletons and which are not), wheras '==' will always give the correct answer regardless.
Correct. In Numpy the slices are views on the underlying memory. That’s why they’re so fast, there’s no copying involved. Incidentally that’s also why freeing up the original variable doesn’t release the memory (the slices are still using it).
If you want a promotion, move to another job. Most companies make it incredibly slow and painful to evolve in their own walls, while giving big rewards to new comers.
Unless you happen to work at the legenday ones that do care about you, don't play according to their rules. Those are here for their benefit, not yours.
Highly recommend this.
I'm a consultant, and I got offered a $12,000 raise after working with my company for two years and working for three seperate extremely high value clients.
I put in my two weeks notice the day I received an offer from a competing company nearby that I'd applied to for a total renumeration package roughly equivalent to a $45,000 raise at my old job.
Shop around. Always. If you don't, you'll never know.
I like to think of it as someone losing out on free stuff. It costs nothing to write up an updated resumé and throw it into the abyss of companies that are hiring. Possible gain by doing this? A lot. Possible loss? Maybe an hour of your time that you spent applying to the job.
Can you provide an example of where Python has broken backwards compatibility recently between 3.x version? I'll admit (despite googling for 5 or so minutes) that i don't actually know if it does. It obviously breaks forward compatibility continuously all the time - new language features are landing, and they just aren't present in previous versions - but I don't know if I've ever run into people being tripped up by that.
I know some Python Libraries break backwards compatibility (Pandas being a big one) - but, for the most part, hasn't the language been backwards compatible since at least Python 3.4? (And possibly further back, for all I know).
Keep in mind they have deferred a number of them because of the impending EOL of Python 2.7. There are fewer breaking changes during the latter 3.X series, which should resume in 3.9 or 3.10 now that Python2 has passed on.
Here's a commonly mentioned one:
Changes in Python Behavior: async and await names are now reserved keywords. Code using these names as identifiers will now raise a SyntaxError. (Contributed by Jelle Zijlstra in bpo-30406.)
Note: I think this is a bad idea, I'd rather all these small breaking changes and parser be deferred to 4.X. But they need to be small breaking changes, of course, not a new language.
If we look back in python history, the rolling breaking changes have been handled mostly fine, and the actual Python 3 caused a lot of pain in the ecosystem. So I hope they stay away from major versions and keep up the other things they are doing.
That was due to the scope of the breakage, not number format. A good way to handle that and maintain predictability is to constrain breaking changes, yet defer them to 4.X.
First, there are not a lot of interpretted (not compiled, that's another matter entirely) languages that are as old as Python.
And there are really few that are even near Python popularity, or used with such diversity as Python.
I mean, you can get away with keeping AWK the way it was 2 decades ago, nobody is going to use it for for machine learning or to teach computing in all universities in the world on 3 operating systems, utilizing C extensions, or processing Web API.
Among the few that would even compare, there are the ones that have accumulated so much cruft that they became unusable from today's standard (E.G: bash). Then you have those who have done like Python (E.G: perl 6). The ones that just tried and failed (PHP 6). The ones that broke compat and told everybody move or die (Ruby in a point release, gave basically 2 years). And the ones that created a huge pile of horror they called full stack to keep going (E.G: JS). Also those that got hijacked by vendors and just exploded in miriads of proprietary syntaxes (E.G: SQL) or completely new concepts (E.G: lisp).
At least, in Python you CAN write Python 2/3 compatible code, and you have a LOT of tooling to help you with that, or migrating.
So, yes, the Python 2 -> 3 transition could have been better. Insight is 20/20.
But I'm struggling to think of any other language in a similar situation that has done better.
Ruby did something like this around the same time Python did. Ruby's was a bit smaller, but overall a roughly similar amount of breaking changes. They forced you to think about encodings more with Strings, they changed the signatures of several operators, they changed some of the syntax for case statements, they drastically changed the scope rules for block variables, they restructured the base object hierarchy, etc. In both cases, it was a deliberate decision to make a clean break. I think Ruby's big break didn't make as big a schism mainly because Rails was very supportive, and Rails holds an enormous amount of influence in the Ruby world.
If Python 3 had been introduced as a separate language, I'm pretty sure everyone would have said "Why isn't this just called Python 3? It's 99.9% the same as Python and it's by the same people and they're deprecating Python in favor of it."
how will that number look when you first autoconvert via 2to3?
I did two migrations of >500k loc projects in an afternoon each, and admittedly some days of testing to gain confidence since there where few unit test. But i found it to be very smooth sailing.
I was very familiar with both projects, so that helped a lot.
EDIT: I also want to add that i did this using python3.5, when to ecosystem seemed to be at a sweet spot of dependencies supporting both 2 and 3 mostly. I guess if one has been waiting until now, the divide between library versions will be a lot bigger.
As a big user of logging and little to do with character coding, all of my admin/daemon stuff moved over with almost no changes necessary for 3.0 (actually ~3.3).
For some projects I did bigger refactors for 2.6/7 (exceptions) and 3.6 (fstrings).
Really? The same .py file runs under python2 and python3?
Googling quickly, I find this, which does a bit better than 2to3. I suppose one could write to a somewhat constrained intersection of Python2 and Python3, if one is willing to make at least some boilerplate changes to the original Python2 code.
That said, if you bring a Python2 script and feed it to a Python3 interpreter, no, in general that will not work. They simply aren't the same language. Even a simple "print x" will do you in.
> The same .py file runs under python2 and python3?
Sure, as long as it doesn't contain any syntax or spellings which are incompatible between the two. That's a fairly large subset of the language.
> if you bring a Python2 script and feed it to a Python3 interpreter, no, in general that will not work. They simply aren't the same language. Even a simple "print x" will do you in.
But this will work:
from __future__ import print_function
print(x)
This is valid under both Python 2 and Python 3.
Also, as I said above, there is a pretty large subset of the Python language that has the same syntax and spellings in both Python 2 and Python 3, and any script or module or package that only uses that subset will run just fine under both interpreters. You are drastically underestimating both the size and the usage of this subset of the language.
Say Django 1.11, a massive amount of .py files, works completely fine under both 2 and 3. As do many other libraries.
Yes you often need some precautions like "from __future__ import" statements and sometimes libraries like `six`, but it's been perfectly normal practice for most of the last decade.
A lot of projects write in that style e.g. compatible with both python2 and python3, it's really common because there's so much py2 deployed (was default on centOS until very recently, still default on osx, etc.)
Nearly every py3 feature was backported to 2 you just need to write it in a compatible way. I'm seeing some drop py2 support now though. Which I'm fine with, I haven't written python2 code in maybe 6 or 7 years now.
PHP which tried to address unicode in version 6, but then abandoned it and went straight to 7. Perl, which amusingly also at version 6 decided on a huge re-write, but then just decided to rename the version as an actual new language "Raku".
You don't need to. Download any module or packgage, put it next to your code, and you can import it. There is nothing to do. There is not even need for metadata.
But yes, pip can install packages from arbitrary URLS, including git repos, if they contains the right metadata.
What's more, with something like shiv (https://github.com/linkedin/shiv), you can actually bundle those dependancies into one zipapp (an official python feature, see https://docs.python.org/fr/3/library/zipapp.html), which is basically a zip containing your entire project that you can just execute like if it were one py file.