import random
import time
from concurrent.futures import ProcessPoolExecutor, as_completed
def hello():
seconds = random.randint(0, 5)
print(f"Start blocking for {seconds}s")
time.sleep(seconds)
print(f"Stopped blocking after {seconds}s")
return seconds
if __name__ == "__main__":
with ProcessPoolExecutor(max_workers=2) as exec:
a = exec.submit(hello)
b = exec.submit(hello)
for future in as_completed((a, b)):
print(future.result())
The same exact same API exist for thread BTW. I don't think a tutorial should introduce you to concurrency using manual process/thread management any more. I makes no sense to me.
You may note that multiprocessing still eats more RAM than with typical Go/Rust code since max_workers=n means n+1 Python VM spawing but on modern servers you don't really feel it. That's what most WSGI setup do anyway.
Now, before 2019, there was one more use case that wasn't served well: how do you share some computation between isolated processes that need to communicate to make their work ? Typical use case was people using numpy or pandas crunching numbers that depended on each others. Indeed, communicating between processing using piping is expensive given the cost of message passing serialization.
from multiprocessing.managers import SharedMemoryManager
with SharedMemoryManager() as smm:
sl = smm.ShareableList(name="unique_name", range(2000))
The sl object can then contain int, float, bool, str, bytes and None and its reference can be shared among processes. Each item can be deleted and replaced. You an also get an hold on sl by using the "unique_name" if you don't have the reference at hand.
There is a raw ShareableMemory object for stuff like numpy/pandas array buffers if this is your main concern.
Thank you for this. I did not know of this change in implementation in Python 3.8.
Are you a core dev btw? I have a complaint about single threads and processes and how they have absolutely no way to return values back to the main thread except through some shared memory object. Am I too ignorant to understand how big of a challenge it is to do this?
Instead to implement this, I keep having to use like single process pools that already have return mechanisms encoded. All I need is for a way of starting off a thread (or process) and joining it once my main thread / process is done and retrieving its return value for my use (assuming there is one).
For very simple use cases, you can inherit from Thread and force join() to return the value:
class SimpleReturningThread(Thread):
result = None
def run(self):
try:
if self._target:
self.result = self._target(*self._args, **self._kwargs)
finally:
del self._target, self._args, self._kwargs
def join(self, *args, **kwargs):
super().join(*args, **kwargs)
return self.result
Personally, I'd stick to using a pool with only one worker in it. It's not worth the trouble of doing all this work. Wrap it in a few functions if you do the same thing repeatedly and call it a day, it's unlikely going to be what most your code is about anyway.
Correct me if I'm wrong, but from what you've said I take it that the GIL issue has not been resolved (i.e. the Python interpreter is not thread-safe, Python still can't have a single process, with a single Python interpreter, running multiple CPU-bound threads all doing useful compute in Python, running against a single unified local memory space containing all of the objects, as you would in say C or C++). The only option is to resort to more elaborate schemes involving multiple processes, multiple interpreters, and explicitly managed shared memory windows -- but all of these options are in place now, so there is at least a solution to most common use cases, even if it's not as simple as in other languages with proper support for threads and shared local memory.
Linkedin used to trick people into thinking you have to pay to see someones profile when you can see it just fine by logging out. That is not against Googles policy. Not sure if they still do that, but I used the logout trick a lot a few years ago.
You sure about that? How big qualifies as big in your eyes? Medium implemented homepage cloaking in November of 2019 and within a month they lost 40% of their overall search visibility:
Pinterest? Google is somehow able to crawl images on their site, but I certainly can't view these images without creating an account and logging in. Their site ranks highly enough that I gave up on google image search last year.
Additionally, off the top of my head Bloomberg and NYT both won't allow me to view more than a few articles but let the Google crawler index their articles.
I've definitely seen article text on Google that was not presented to me when I clicked the link, and not because the article was updated or something, but because I was blocked from seeing the article.
There are many cases where what you need is instant soup powder, not home made velouté.
And this will save me from writing tests for those lines. With all major browsers. And documentation. And maintain all that, of course, for the next 5 years.
And when new devs will arrive in the team to provide the training for the custom code. I may even not be here anymore
So yes, it can be overkill, or just what you need.
E.G: some server side dev need a quick fix for his personal web site.
Besides, even a lib with zero tests like this one will have enough users and tickets open that it will be better tested that the server side dev code that don't know JS much.
The problem is, there’s no way to enforce this. If you are a CA, there is nothing technically stopping you from issuing a certificate for google.com. There’s no infrastructure for the ability to say “this CA is valid only for sites that agree with the risks involved in using it”.
Actually, I believe there are. IIRC it's possible for a CA's root cert to be limited to only a certain TLD. There's an extension called "Name Constraints" that limits a Root SSL Cert, and I believe it is honored by chrome.
It's an optional extension, so a client may ignore it without failing the connection. It needs a lot more adoption. I could, for example, instead of getting a wildcard cert, get LE to sign a CA valid for my domain. That way I can issue certs for my domain myself and put load off of LE.
Only if the certificate authority doesn't implement a way of not submitting their cert to the CT log servers. Of course, eventually someone might notice (if the certificate is used widely enough - if it's used to attack specific known people, maybe not) and realise that a certificate is being served that doesn't have corresponding CT info, but browsers don't check that.
The ct-policy link you gave and that section of RFC 6962 are about SCTs. The existence of the Signed Certificate Timestamp does not prove the certificate was actually logged, it just proves that the Log promises to log this certificate within the Maximum Merge Delay (24 hours in the case of the Web PKI). A log might in fact be lying or have suffered a grave technical problem.
Figuring out how best to have browsers check the logs are truly accurate while preserving user privacy (e.g. obviously if you call a log to ask "Did you log this certificate for clown-porn.example ?" it gives them a good idea about your taste in circus-related adult content which you probably don't want them to have) is an ongoing topic of research.
With SCT checks you're in very good shape already today - bad guys would need to compromise several distinct entities to successfully pull this off. It's just not quite "Fire and forget" safe yet.
If the certificate wasn't submitted to an otherwise trustworthy CT log server then there's no way to provide an SCT ("Signed Certificate Timestamp") for a qualified CT log server. Some browsers (including Chrome and Safari) expect to be shown qualified SCTs for certificates issued after a certain date.
The SCT is signed by the log, so if you want them to lie you'd need to control enough logs to get enough qualified SCTs. Bad news, for Chrome that means it must include a Google log, which means now you're asking Google to conspire against themselves.
For now you could sidestep this by issuing bogus certificates with back-dated issuance dates. But that will stop working once those dates are too long ago for a certificate to still be valid. I think that happens some point later this year or in 2021 maybe? If you show Chrome (perhaps other browsers but I've read the code in Chrome) a certificate in which the lifespan (between NotBefore and NotAfter) exceeds the maximum permissible lifespan at that issuance date, it treats the certificate as invalid anyway.
By the way, there have been a number of openly government-operated root CAs in numerous countries for decades. Before I got involved with Let's Encrypt, I was very nervous about government influence in PKI and thought this was quite concerning. (That was also before Certificate Transparency.)
A quick look at a root store turns up
subject=C = CN, O = China Financial Certification Authority, CN = CFCA EV ROOT
subject=C = HK, O = Hongkong Post, CN = Hongkong Post Root CA 1
subject=C = NL, O = Staat der Nederlanden, CN = Staat der Nederlanden EV Root CA [and other roots with the same organization]
subject=C = TW, O = Government Root Certification Authority
I think there are several others, probably especially at the intermediate level.
Just what we need is to get politicians involved in this.
One benefit thought would be they could more easily block access to sites they don't want folks visiting by just denying a CA cert (which would then generate warnings / blocking in many browsers).
I would like to hear from the planetary president to see what he has to say about that.
Seriously this is all great but unfortunately with all the mess going on with "the big powers" I'm not sure you want to give them that. I wished we lived in a world where we could tho.
Cloud flare is American and we know since the PRISM scandal that US based tech companies are directly plugged into the NSA, and everybody in the chain will deny it under the threat of prison.
So, if this rolls out 'as-is' in any other country than the US, we will go from "all DNS requests are clear text, but dispatched among many entities" to "DNS requests are encrypted, but all read and controlled by american agencies".
We (may) have gain (some) privacy (maybe). But we also (certainly) gained a serious dependency.
NSA don’t factor into my personal threat model _at all_ where random ISPs snooping and selling do. I would gladly give the NSA all of my traffic unencrypted in exchange for decent commercial privacy
I supposed not having a dictatorship regime in your country history book helps to see things that way.
Given the way my country went from freedom to "regime de Vichy" in a few years, during my grandpa time, I don't want a state level entity having that kind of power.
Since the US state level entities decided they could now ignore Habeas Corpus and legitimated torture, secret courts and declared impunity for them-self, I especially don't want them to have that kind of power.
If a malicious power takes over your country they’ll hit you with a rubber hose until you give up your secrets much before they give a shit about your internet history, I suspect.
Quite the contrary, as shown by as most dictatorships across the world trying to control their piece of the internet. The biggest example being the Big Firewall of China.
This let them control how people think, communicate, consume and inform them-self. But also detects anyone that could oppose the regime. Or make a graph of all allies, suspects, etc.
I don't think it is. If JS had any sane competition, I think it would have lost. By some weird conjunction of events, it landed a monopoly on the most amazing plateform in the world.
PHP is a good example of Worse is Better, because it won over the competition, despite its flaws.
I note that all those ecstatic quotes are from executives.
Because the way I remember it, technical people hated it with a passion, from its design to its implementation.
In fact, the most popular JS projects have always been to avoid writing ES5, its DOM API, or depending of one of its implementation:
- scriptaculous: we hate even the basic types, so we monkey patch our own code into it.
- jquery: don't use any of the language paradigms and use ones from Lisp and Haskell. Avoid "this". Don't touch the DOM. In fact, don't code, just make a few calls and pass parameters. And we rewrite the whole damn browser API to avoid incompatibilities.
- GWT: write JS from Java.
- underscore and co: let's rewrite typeof, the equality sign and array manipulation methods. And provide an stdib for god sake.
- coffeescript: we like Python and ruby but must do JS.
- React: HTML in JS now please. And magic DOM. Oh, and prototyping is horrible, lets use classes. And we strongly advice immutability, but do what you please...
- webpack and babel: ES6 and 7 are not here yet, but ES5 is so bad we will setup an entire ecosystem dedicated to make believe they are. Also pretend import works like in other languages.
- typescript: ok, we were wrong, ES7 is not enough. Not nearly enough. Please make intellisense works.
Now, in 2020, JS is finally not so horrible to work with. You can use map(), () => and "..." and fake namespaces. Which is amazing to access the crazy cool plateforms that the modern browsers and the web are.
So I guess, in a way, this quote from the AOL guy was true.
JavaScript has a very high learning curve. But it becomes great once you are fluent. When manipulating the DOM it kinda sucks that you can't just monkey-path attributes, you have to use el.setAttribute. The DOM is also very slow. But the DOM do a lot of work - ever tried to make your own UI framework from scratch only using low level GUI calls? z-index, styling with CSS, screen-reader support, etc.
> I still have people argue with me that the DOM is slow even though all the evidence and browser vendors claim updates to the DOM is simply switching a bit in memory. In other words many web developers have utterly no idea how their platform works.
I guess not much has changed in 25 years. Most developers still fear the language, have no idea how it works, and will do everything in their power to pretend this language is some other language.
> ever tried to make your own UI framework from scratch only using low level GUI calls? z-index, styling with CSS, screen-reader support, etc.
I meant creating a GUI from scratch. Creating a GUI using the DOM is easy. Working with the DOM overall is simple. What I mean is that the DOM do so much work for us (that we take for granted).
What I mean by slow is for example adding a element to the screen, or updating some text. Compared to for example everything that happens on the screen in a 3d game.
I do everything using vanilla JS and I cannot understand why anyone would use a web framework, (although most web project does). I do however have a vague memory that the learning curve was fairly steep, as I have been working with vanilla JS for over 20 years. (whereas the average developer has only used JS for one year) it wasn't until I started to use JS full stack (with Node.JS) that I started to like it!
I worked with vanilla JS for some time (before Prototype.js or jQuery existed) and used a bunch of incantations of different libraries and frameworks over the years and I cannot understand why anyone would choose to use JS for platforms where you can use any other language (aka... any platform that isn't a browser).
No construct or design decision of JS is better than in other languages, the lack of a basic standard library hurts a lot. Dependency management was inexistent until recently, the same for a lot of basic constructs, data types and other utilities that should be baked into any modern language.
The current landscape of JS is so convoluted and with such a varying level of quality that is intractable for me just trying to get things done without getting into the community and everything around it, it's not an easy tool to use and it's not an ergonomical one.
I have used JS for the past 15+ years for different kinds of software and I dislike it, not intensely but I'd never choose to work solely on JS. Yes, you can be productive in it but there are much better tools for most of the job it does around, its only power was to be the only language you could use to script a browser and I hope that WASM kills that.
JavaScript has taught me that there are only two things separating a novice programmer from an excellent programmer: writing and data structures.
In order to improve as a programmer over the years I have had to learn to improve my writing. This requires a clear vision before putting keys on a keyboard and a solid sense of organization. You should know what you are going to work on with a strong confidence on how to go about it before doing any of the actual work. Sure rough spots will emerge along the way, but that shouldn't stop you from solving for the original vision. Then once the original vision is achieve refactor, debug, and refactor again. In writing that is call drafts and second opinions. This applies to any programming language equally.
A confidence of data structures is also critically important. Once skills and conceptual mental modeling of data structures becomes as common as spelling your name algorithms and logic almost write themselves. JavaScript does some things right by providing simplified data structures (objects, arrays, sets, maps, and such) and is really expressive in how you use them. JavaScript also does a lot wrong in this regard with its loose typing and some of its sloppy conventions. Fortunately, TypeScript interfaces coupled with consistent use of strict type declarations close many of those gaps.
---
I have heard so much hope for WASM killing JavaScript over the last 3 years, and almost all of it based on ignorance of how the technologies work. So much of this hope hinges on the availability of the page's DOM to a WASM instance by people who have never written to the DOM and have no idea what they are really asking for. While the WASM working group says this will never happen I suspect that very simple technology reasons will get in the way first regardless of what anybody wants.
If, however, WASM does replace JavaScript because the page's DOM becomes available to a WASM instance I suspect it will be the same clusterfuck that those developers were hoping to replace. I suspect if those developers couldn't figure out the relationship-oriented tree structure of the DOM using JavaScript they fail all the same when using their favorite language.
I can understand replacing JavaScript for other reasons, but rarely (extremely rarely) are the well grounded reasons ever a primary motivation for wanting JavaScript replaced.
I've just began to learn ES5 and ES6. I've compiled a small list of JS quirks, things that I'd take for granted because they're well stablished in other high level programming languages. For instance, 'for/in' picks the keys instead of the values of either array or dictionary ( object in JS terms I think), dictionary doesn't have a keys 'method' but you can call Object.keys for that functionality, arrays are key/value indexed, so array['a']=1 is valid, and so on. I've lost a little while just laughing when I learned about how ES5 does OO.
JS oftentimes reminds of Apple: there's the industry standard or some already well stablished way of doing things, and they go like "Well, flip it. We're doing our own way, just because".
jQuery feels like the list monad, if you see chaining as similar to bind and $() not so far from return.
And for Lisp, jquery provided error handlers instead of relying on native try/except.
Of course, it's more inspired from it than the real thing, but it's far from any JS paradigm of the time. Remember that JS didn't even have map() and the only place you used your first class functions was to setInterval. Plus the joke is funnier that way :)
Method chaining (aka. fluent syntax) is not really comparable to bind. Bind is nested lambdas, which means an operation might be executed zero or multiple times depending on the bind. In method chaining, each call will be executed exactly once, unless an exception is thrown.
The use of method chaining in jQuery is cool, but it is not very similar to Haskell.
Stealing this for my next JS training, beginners always have a hard time with callbacks, so it's a good first step to actually explain the benefits of callbacks.
I suggest that, if possible, try to set up a cheap server on a DO droplet or somewhere with a built in amount of latency and use that in your presentation.
When I was learning callbacks I saw people use the "setInterval" example and while it did help to an extent, the real world implementation still left a good deal to be desired.
Why would you direct new comers attention to eval'esque functionality? They should stay far far away from it. Come to think of it, everyone should stay away from eval functionality!
Agreed. In my class I always start with the hard/wrong way. Point out why it's hard/wrong. Let them hate it. Then show the easy/correct way. You can't just point them at something and say, "Do it this way or else." That's not teaching. That's directing.
I feel like you could start with manually calling the function, and then introduce callbacks as a way to get your code to run in response to some event. A callback is no different to a regular function after all.
If your way works, then it works though I suppose.
> - webpack and babel: ES6 and 7 are not here yet, but ES5 is so bad we will setup an entire ecosystem dedicated to make believe they are. Also pretend import works like in other languages.
They just peeked over at their low level programming neighbours and realized what they did to avoid programming assembly and program C instead was a smart idea.
Genuinely curious about your last point (don't know much about the topic). Is english intrinsically better at this, or is it because of the presence of jargon? Is it a studied phenomena, or is it something most people feel?
English is usually shorter than other latin-based languages. It's longer than ideogram based ones but you don't have to learn 100000 symbols to express yourself in it.
It also has a very simple grammar compared to most languages. Take this sentence:
"I would like not to go to school today"
The french equivalent would be:
"Je voudrais ne pas aller à l'école aujourd'hui."
"would like" is a simple combination of two words, but in french you need to know the precise conjugation of it.
"not" is actually expressed as 2 words with "ne pas", which can be positioned in several ways.
Infinitive, like with "to go", is simple in english: just add "to". In french, each verb is different, like "aller".
Then you got "the" in any circumstances in english, but the "l'", could also be "le, la, or les" depending of the word after it. A;so remember that each word is either feminine or masculine in french, even a stone or the sun.
Then "à" and "école" got an accent. French has many of them, you need to know the right one, where to place it, how to pronunciation it and type it on the keyboard.
Finally, "today" vs "aujourd'hui". I know which one is easier to type in a bug report.
Not to say English doesn't have weird traps, but it's very, very relaxing compared to the rest. And much more efficient.
Also describing a view of the country side with it feels a bit limiting. But I'm not Shakespear :)
Hangeul is a syllabary so anyone can read it, just like Latin. It was specifically designed in response to peasants not being able to read Chinese hanzi.
Things like the subject and topic can be and is omitted when it’s superfluous. (The above sentence omits the subject and subject particle. There is no topic here). English almost always needs to specify the subject, asides from very casual speech/slang. Korean, like english, doesn’t need to specify gender of nouns, and it also doesn’t need “a” or “the” markers. The location particle above could be dropped too.
However Korean does have complex honorifics and formality conjugations, which typically get longer the more formal/polite it is. Above we have the plain or dictionary form, which is usually the shortest form as well.
"I don’t want to go" and "I would like not to go" express different things.
One is expressing opposition right now, the other one is expressing desire or even a request, potentially while the action is already engaged. The first one is definitive, the second one is wishful thinking or negotiation.
I don’t see any practical difference other than using more polite, indirect language. The effect of “would like not to” is the same in the end with opposition.
I don’t think there’s a 1:1 translation of “would like not to”. I’d probably say something like “it’d be good if I did / didn’t do X”. Which is less direct than the equivalent “I do / don’t want to do X”.
What French really needs is a comprehensive spelling/grammar reform. In Haitian Creole, "ne pas aller à l'école aujourd'hui" is written simply "pa ale lekòl jodi"! And one can just write "mwen" instead of having to differentiate "je" and "moi". (There's also a marker word "ta" that's entirely isomorphic to the English "would" in that it introduces a conditional modality.) Not coincidentally, English is basically a creole language as well. That's why it's so incredibly simple.
English has like 17 tenses, many languages only have 3 without any loss in expressivity. When learning English you need to learn each word twice (once to write, once to speak) - that's not the case with many phonetic languages.
English isn't inherently better as a world's language than anything else, it's just there, so it benefits from the network effect (it's more beneficial to learn language that has the most users, so it gets even more users).
In colloquial Japanese that would be three words: Kyou gakkou ikitakunai. The first word is "today" (unconjugated), second is "school" (unconjugated), and the third is the negative "wish" (volitional?) form of "to go" (iku > ikitai > ikitakunai). Subject (the speaker) is implied and neither the time nor the the topic/object needs to be explicitly tagged.
I can see that being true. Many of the web sites I build are multi-lingual, and when designing for many languages, you have to take into consideration that certain languages take many more characters or words to express an idea than in English.
Off the top of my head, I believe we factor in 15% more text space for Spanish. German is something like 60% more.
When localizing software also it's a general rule of thumb that your on-screen UI spaces need to be something like 40% bigger than the English text that goes in them since the German equivalent is always going to be way bigger. It's common for it to turn out half-way through localization that some of your text doesn't fit anymore.
(FWIW, this also happens when swapping from ideographic languages to English - western localizations of Japanese video games often end up with very small text as a result.)
In my opinion, jargon helps, but English is just easier to learn and shorter to communicate.
My first language is Portuguese, and English was not so hard for me to learn. I studied a bit of French and it was ok, but not as easy as English. And this considering it should be a bit easier since Portuguese is my main language.
Now I'm studying German and... it's way harder than English and French. At least, way more verbose and which more complex grammar rules and conjugations.
English has an incredibly simple grammar, comparatively speaking. There are only three verb conjugations (if you don't count auxiliary verbs) and one gender. Even naming variables is easier!
However, it became especially easy with Python 3.2 (10 years ago) which introduced the ProcessPoolExecutor (https://docs.python.org/dev/library/concurrent.futures.html#...):
The same exact same API exist for thread BTW. I don't think a tutorial should introduce you to concurrency using manual process/thread management any more. I makes no sense to me.You may note that multiprocessing still eats more RAM than with typical Go/Rust code since max_workers=n means n+1 Python VM spawing but on modern servers you don't really feel it. That's what most WSGI setup do anyway.
Now, before 2019, there was one more use case that wasn't served well: how do you share some computation between isolated processes that need to communicate to make their work ? Typical use case was people using numpy or pandas crunching numbers that depended on each others. Indeed, communicating between processing using piping is expensive given the cost of message passing serialization.
However, in the previous Python release (3.8), we introduced a mechanism to share memory for almost free (https://docs.python.org/3/library/multiprocessing.shared_mem...):
The sl object can then contain int, float, bool, str, bytes and None and its reference can be shared among processes. Each item can be deleted and replaced. You an also get an hold on sl by using the "unique_name" if you don't have the reference at hand.There is a raw ShareableMemory object for stuff like numpy/pandas array buffers if this is your main concern.