And that's one of the main show stopper for me with kivy: it comes with very few built-in UI controls, so you have to code a lot of things yourself.
I much prefer Python to JS, but things like react native win because of the community libs you can install save you tons of time, and produce a better result.
This is especially true when you use a lot of tooling. I love jupyter, but installing it in a venv means pulling a lot of deps which will affect a lot what I can install.
Fortunately the Python community is much more serious about making deps that work together than the JS community, and the fact it works at all given the cartesian products of all the python modules is kind of a miracle and a testament to that.
Unfortunately, that's a problem that is unlikely to be solved in the next decade, so we all live with it.
The reverse problem is true for JS, and I see many projects shipping very heavy frontend code because despite all the tree shaking, they embed 5 times the same module with different versions in their bundle. That's one of the reasons for the bloated page epidemic.
I guess it's a trade-off for all scripting languages: choosing between bloat or compat problem. Rust and Go don't care as much, and on top of that they can import code from 10 years ago and it sill works.
However, and while I do know how hard it is to ship python code to the end user (at least if you don't use a web app), I don't think the version problem is the reason. We have zipapp and they work fine.
No the main reason iscompiled extensions are very useful and popular, which means packaging is solving more than packaging python, but a ton of compiled languages at one. Take scipy: they have c, pascal and assembly in there.
This can and will be improved though. In fact, thanks to wheels and indygreg/python-build-standalone, I think we will see a solution to this in the coming years.
My ideal situation is that the system should maintain authoritative versions of every package and version that is ever requested, and they should not need to be shipped. Multiple versions of a package should coexist.
When a package requests 2.1.1 it fetches it right out of there, installing from PyPI if it doesn't.
The same should be true of JS and even C++. When a C++ app's deb package wants libusb==1.0.1 it should NOT overwrite libusb-1.0.0 that is on the system, it should coexist with it and link to the correct one so that another app that wants libusb-1.0.0 should still be able to use it.
> Fortunately the Python community is much more serious about making deps that work together
This is very not true at least in ML. I have to create a new conda environment for almost every ML paper that comes out. There are so many papers and code repos I test every week that refuse to work with the latest PyTorch, and some that require torch<2.0 or some bull. Also, xformers, apex, pytorch3d, and a number of other popular packages require that the cuda version that is included with the "torch" Python package matches the cuda version in /usr/local/cuda AND that your "CC" and "CXX" variables point to gcc-11 (NOT gcc-12), or else the pip install will fail. It's a fucking mess. Why can't gcc-12 compile gcc-11 code without complaining? Why does a Python package not ship binaries of all C/C++ parts for all common architectures compiled on a build farm?
I'm assuming by system you mean OS, which is a terrible, terrible idea. Dev stack and system libs should not coexist, especially because system libs should be vetted by the OS vendor, but you can't ask them to do that for dev libs.
> I have to create a new conda environment for almost every ML paper that comes out
That's how it's supposed to work: one env per project.
As for the rest, it's more telling about the C/C++ community building the things bellow the python wrappers.
That causes 50 copies of the exact same version of a 1GB library to exist on my system that are all obtained from the same authority (PyPI). I have literally 50 copies of the entire set of CUDA libraries because every conda environment installs PyTorch and PyTorch includes its own CUDA.
I'm not asking the OS to maintain this, but rather the package manager ("npm" or "pip" or similar) should do so on a system-wide basis. "python" and "pip" should allow for 1 copy per officially-released version of each package to live on the system, and multiple officially-released version numbers to coexist in /usr/lib. If a dev version is being used or any version that deviates from what is on PyPI, then that should live within the project.
Actually conda creates hardlinks for the packages that it manages. Found this out a few weeks ago when I tried migrating my envs to another system with an identical hierarchy and ended up with a broken mess.
> but rather the package manager ("npm" or "pip" or similar) should do so on a system-wide basis.
I basically agree with this. With the caveat that programs should not use any system search paths and packages should be hardlinked into the project directory structure from a centralized cache. This also means that a dev version looks identical to a centralized version - both are just directories within the project.
Kind of, but not really. Nix is extremely complicated. Programs / projects including their dependencies is exceedingly simple.
Also, Windows is my primary dev environment. Any solution must work cross-platform and cross-distro. Telling everyone to use a specific distro is not a solution.
It is complicated... but honestly I have found claude 3.5 to just 'fix it'. So you hardly have to spend much time spelunking. You just give it all your dependencies and tell it what you want. It'll whip up a working flake in a few iterations. Kinda magic. So yeah when you can abstract out the complexity it moves the needle enough to make it worth it.
It’s not the researcher’s fault if the libraries they use make breaking changes after a month; proof-of-concept code published with a paper is supposed to be static, and there’s often no incentive for the researcher to maintain it after publication.
At this point, venvs are the best workaround, but we can still wish for something better. As someone commented further up, being able to “import pytorch==2.0” and have multiple library versions coexist would go a long way.
I install most tooling, including Jupyter, using pipx. The only thing I then need to install in the project venvs is ipykernel (which I add as a dev dep), and then create a kernel config that allows Jupyter to be run using that venv.
The problem I see a lot of JS developers having when they start using Python is they try to do the "import the entire world" strategy of development that's common in JS, and there isn't good tooling for that because Python just doesn't have that culture. And that's because it's a bad idea--it's not a better idea in JS, it's just more part of the culture.
Pick one package source. Stick with it. And don't import every 0.0.x package from that package source either.
There are obviously reasons to use more than one package source, but those reasons are far rarer than a lot of inexperienced devs think they are. A major version number difference in one package isn't a good reason to complicate your build system unless there are features you genuinely need (not "would be nice to have", need).
And it doesn't provide any way to use a link to any other package repository if you want to stick to vanilla pyproject.toml + build (the official build tool). So if you want to use the CUDA or rocm version of torch, for example, you have to add a direct link to the package. That means that you'd have to hardlink to a platform specific version of the package. There's no way to just make a package look at a non pypi repository to get the version you want otherwise.
So say you want to add pytorch, with GPU acceleration if it's possible on a platform. You want to make it multiplatform to some extent.
You can't add another index if you want to use vanilla build, as that's not allowed. You can add a direct link (that's allowed, just not an index) but that's going to be specific to a platform+python version. Pytorch doesn't even provide CUDA packages on pypi anymore (due to issues pypi), so you need to be able to use another index! You'd need to manually create requirements.txt for each platform, create a script that packages your app with the right requirement.txt, and then do it again whenever you update. Otherwise, I think the most recent advice I've seen was to just make... the user download the right version. Mhmmmm.
The other option is to use poetry or something like that, but I just want to use "python build . "...
But you can do that, obviously not with this syntax. It’s non standard but I have built programs that install all dependencies as a first step. It’s pretty trivial.
Besides the fact this article avoids giving any concrete examples, showing the author probably doesn't use the technique himself, it's going to fail when the opposite side cannot be convinced by rationality, which is most of the time.
Indeed, you cannot "patch up the weaknesses of the other side’s proposition" of someone supporting a sports team or a religious belief. Because there is no real argument, but rather some kind of emotional commitment.
However, if you engage in the exercise a lot, you'll notice that a huge number of debates are around such notions, where ego, identity, and feelings are more important than ideas.
It's certainly true for lots of political beliefs, but also many technical ones where your social situation depends on the result of the debates more than being wright or wrong (e.g: at work).
Not to mention when you are debating, it happens regularly you win not by convincing the other side, but by winning over the popularity contest with the audience of the exchange. This again, has nothing to do with the logic of the discussion.
So even if the technique works for, say, your discussion with your physicist or philosopher friend (in which you probably didn't need it anyway), there is really little value in it if it can't work in the situations with more at stake.
> it's going to fail when the opposite side cannot be convinced by rationality, which is most of the time.
As many people have noted, arguing on the Internet is a spectator sport. You don't need to convince your opponent, but by steel manning his arguments and addressing them, you have a greater chance of convincing the audience.
If there’s one argument counter to what you’re saying, it’s that the front pages of most social media sites are filled with clickbaits and shitposts. Including HN of course.
That is often my reluctance to use it, especially when talking to folks prone to confirmation bias. I'm not sure there is a way to convince people who only accept evidence that fits their existing worldview. It took me years and a lot of reading to finally seriously entertain the idea that the religion I was raised in could be false. Once I really opened up and stepped back it unraveled pretty quickly.
Yet being that open can be very socially costly, as it was for me. If I'd heard steel man arguments from opponents while still closed to inconvenient truths, then I'd just have accept them without critical thought. And any attacks on that steel man would have been dismissed with whatever was convenient. It devolves into Rube Goldberg levels of mental hoop jumping to stay in that mindset. Though if the alternative is strained relationships or losing ones church community then it becomes second nature, even unconscious.
Steel manning an argument is an extraordinarily powerful tool for rational thinking. I do it all the time, every time I encounter someone who holds beliefs I cannot understand. But I'm very hesitant to post the result of my steel-manning online, for this and other reasons. It's a private exercise.
It's also quite dangerous if you're attached to your beliefs. I rarely walk away from steel manning an argument without being persuaded to some degree into a compromise position.
> You don't need to convince your opponent, but by steel manning his arguments and addressing them, you have a greater chance of convincing the audience.
I try to ask myself: "would I make this argument privately?" If not, I probably won't reply publicly. (I often fail to make the right choice, however.)
> when the opposite side cannot be convinced by rationality, which is most of the time
Which possibility is the "failure" possibility here, that the opposite side gets convinced, or that the opposite side doesn't get convinced? I'd argue the opposite side getting convinced is the failure. This one singular exchange somehow managed to convince him of your viewpoint amidst his entire lifetime of experiences that led him to conclude the opposite.
I think it's the old rationalist way of thinking to think "If your mind isn't changed by a syllogistically correct argument, you must not be arguing in good faith (or otherwise can't be convinced by rationality)."
Everyone arrives at conclusions about various topics from the data they obtain throughout their lives through a continuous complex inference process they cannot communicate. Included in this process is a judgment of which sources are credible and which are less credible. Your friend is more credible than the guy who is showing you a study that contradicts your belief, for the purpose of changing your mind. (There are lots of studies out there so it is not hard to find one that supports your argument.) Trying to distill this complex inference process into a linear argument will necessarily lead to an inaccurate representation of how they arrived at the belief.
Doesn't mean everyone's right, though. Governments, cults, and friend groups are very good at shaping the data an individual receives. Even in these cases, you can see how based on the limited information that they do receive, it is quite reasonable from that internal perspective to believe what they do believe.
Arguments would be better if by "argument" we mean some effort, no matter how difficult, to communicate how you arrived at the beliefs you have, rather than a "well" crafted linear rational argument.
> Arguments would be better if by "argument" we mean some effort, no matter how difficult, to communicate how you arrived at the beliefs you have, rather than a "well" crafted linear rational argument.
Strong point, have you. How did you arrive at the belief that an argument strengthened by such effort, would be better?
To be clear, I meant that the conversation (the argument) would be a better experience for everyone involved, not the argument itself being somehow strengthened by it.
Nevertheless, I'll try to recall how I arrived at this conclusion.
The biggest influence was seeing for all my life, the left and right in America each claim the other is stupid, irrational, and un-convinceable by rational argument. When I was 16 it really did seem to me that the right was the irrational side here, but seeing everything come full circle with what's happening on the left these days, I realized it's kind of weird and unlikely that half the population, divided on a political line, is actually worse at thinking than the other half.
(See the bottom of this thread for a vivid example.)
Thanks for this. Indeed I can confirm the conversation is conceived to be stronger. That said, still I am not convinced that about half the population would not be worse at thinking, the weird it may be.
If the other side is not amenable to good faith rationale, its usually easily identifiable rapidly, say for the small population that are hyper trained for 'cloak and dagger' type operations, in which case you practice zero trust paired with due diligence verification.
The big value add to steel manning is that you can persuade yourself that your own ideas may infact be the inferior one and more concretely explain why that is the case to other people.
Oh but shoot, cats dont speak English.
Nevermind, dont try and use steelman technique against my cat Luna.
There's a difference between oral and written debate as well, and when a lot of people thinking about "arguing" or "debating" they are thinking of live impromptu discussion.
That's not a place to do detailed reasoning. Most people are bad at impromptu and have loads of shit habits. And live impromptu is not necessarily the right setting to shed yourself of shit habits either.
I don't understand how one can be bored in this situation.
You can learn a new language or to draw. Read novels. You can start writing a program or a blog: write it on a different computer, but you are in this situation nobody cares anyway.
You can listen to podcast, even binge watch a tv show discreatly.
You can practice maths, go back learning things you wish you spent more time learning in school.
You probably can even start a business if you are careful no to leave trail.
But also: you can put money on the side, and prepare to leave.
It takes time to transition and get in the headspace of doing something new.
Every time I’ve had weeks of open time to myself the first few days/week is getting oriented. It’s only after being bored for a while that the really exciting stuff comes.
Also when you’re expecting your situation to change “any day now” it dampens the energy to start something new and ambitious.
Yes, this and also you're wondering/worried about where your income is going to come from. Having to job-hunt when you are about to be unemployed is a lot more stressful than doing it while you still have income. Unless you've been able to save a sizable emergency fund, that is going to be weighing on your mind.
The pretending part is the problem. As soon as you have requirements like answering on a chat, replying to email within a half-day of work, sitting at your desk, using corporate hardware... it adds mental workload and physical constraints that drastically reduce your options.
You underestimate the likelihood that you are being watched by colleagues in the same position who are more than happy to maintain the status quo.
See my comment above.
My mother won't install a heat pump because she claims she doesn't want to be dependent on electricity. She likes the idea that if it goes down, she can still heat her home.
It is irrational because her gas burner needs electricity to run anyway, and if electricity shuts down, she lights a wood fire, which she could have as an alternative to the heat pump.
Humans have religion-like behavior in every aspect of their life, not just God-related things.
If you haven't heard about the French pushback against the linky, a smart electricity meter, google it. You will have a good overview of what can go wrong if we force the issue on people.
As a nerd, I get you think it was the main problem.
But most people didn't care about that.
The main contenders were:
- Linky calculates my electricity bill badly. It was in fact more accurate, but sometimes it meant a raise in bill.
- Linky emits dangerous radiation or magnetism. That was, of course, a total hoax.
- Linky didn't allow me to cheat on my bill anymore (lots of people around me were hacking their meter to pay less).
- Linky is forced on us by Big Electricity. So it's bad.
The privacy thing is legit, but all those people had full facebook accounts, most of them had full sync on their smartphone, use chrome without an adblock, and so on.
Only geeks like us really care about the privacy aspect.
On the other hand, it's a greener solution:
- Less people in cars to check meters.
- Live load balancing on the power grid.
- Clear consumption stats so that you can optimize your own bill.
So the linky, which eventually got into most homes, is deployed with success.
But it took so many years because of the pushback.
Eventually, if the pushback would have been just about the privacy, we could have focused on fixing that and would have had a cheaper, faster deployment, with privacy.
We had the worst: expensive, long deployment, and the privacy invasion is still in place.
While the sphere is more modern, and features the very hyped exterior LED screen, the concept of a sphere with a cinema screen wrapped inside it is quite old.
I got good memories of this as a kid, and if you are in the area, it's definitely a must-do in the city. Especially since it's attached to one of the most fun science museums you'll get to visit with a lot of interactive bits.
I do want to get into the sphere to experience what 3 decades of progress have given us.
Actually, the french Panrama [1] invented by Philippe Jaulmes (1958) totally inspired the Imax Dome, after a demo the Expo'67 in Montreal [2].
The Panrama story is long, documents only in french, and I'm still learning bits. For la Géode, the Panrama team tried hard to make a deal to use their french-made tech, so they would have installed 4 smaller omni screens next to the big Imax dome. This was scrapped from the beginning, as la Géode was built on the Omnimax spec from the ground-up. The panrama team still managed to operate a 18m screen in Paris Montparnasse [3] (Espace Gaîté / now Théatre Rive Gauche), and there is one screen left fully equipped in Montpelier, but closed [4]. The Panrama tech was originally 16mm wide, then 35mm, and finally horizontal VistaVision 35mm.
La Geode was an institutional place sold to the Pathé Gaumont cinema group in 2018. I got news in 2022 that the issue was that it's full of asbestos. I heard there is an ongoing legal procedural because it was not noticed in the contract..
I would really like to know more about what they are doing inside. I followed some architectural projects, and they might have removed the imax dome to operate a simpler wide screen + divided the place for others screens. So sad, it was one of the last omnispherical screen in France, there is only one left in Le Futuroscope.
You know, today we talk about nazi and fascist because their model is back. Simply. Some fails to recognize it, as fails to recognize it's economical and philosophical origin, so think the others are crazy. Happen frequently.
The tendency to exaggerate small things is also a normal part of this process, because our society start to fracture, things became tough and so anything is seen as a giant thing.
It's not different anyway than the various plethora of managers who attend "schools of management" learning essentially a kind of religion and apply it in their life as a religion, failing to see that. We have on the records stuff like some real estate C-levels hospitalized due to feet burns because at a team building events they have marched on a carpet of hot coals [1] and others in the Nederland who get hospitalized for serious concussions because they have play to drop themselves in the arms of their colleagues... What do you classify such hilarious extreme episodes? They are rare of course, but not so rare, and even without certain outcomes they show a certain approach.
Well, kxfx whom you replied to was the OP in this thread. And according to their other top-level comment[1], that seems to have been pretty much exactly what they were talking about. (Yeah, a bit far-fetched IMO too.)
For deduplicating people, the Fellegi Sunter model is also a powerful approach. Splink[0] is a free Python library that implements this for big datasets. Probably you could combine parts of both approaches as well. Full disclosure: I am the lead author.
I've also written up some interactive tutorials on how the method works [1] if anyone's interested
The Fellegi Sunter model is able to estimate the importance of different types of information from the data itself (i.e. unsupervised learning).
For instance, a match on a date of birth column lends a greater weight of evidence in favour of a match than a match on first name (since dob has higher cardinality).
The method is also able to estimate weights for fuzzy matches (how much evidence in favour of a match is close match on dob with one character difference), and also how much evidence against a match a mismatch is.
For instance, if you have very high data quality on gender, then a match on gender doesn't tell you much, but a mismatch on gender is quite strong evidence against the idea two records match.
It doesn't sound like the approaches are incompatible. You can use minhash LSH to search a large set and get a top-k list for any individual, then use a weighted average with penalty rules to decide which of those qualifies as a dupe or not. Weighted minhash can also be used to efficiently add repeats to give some additional weighting.
The performance is pretty good because the prediction is ultimately just adding up match weights. Much of the performance is dictated by
(1) The blocking approach you choose (how wide you cast the net in searching for matches. This is actually somewhere were minhash can be used in conjunction
(2) whether you choose to use complex fuzzy matching functions and how many - this is chosen by the user.
Overall it's an approach which makes a pretty good trade off between speed and accuracy. That's why it's used by many national stats institutes (UK, US, Aus, Germany etc.) - because it's capable of working on country-population sized datasets.
There is also gaoya ( I am the author ), which is also written in Rust and has python bindings. datasketch is great, but it is not performant enough for my use case. gaoya is used in a production system for large scale clustering
https://github.com/serega/gaoya
