Hacker Newsnew | past | comments | ask | show | jobs | submit | more BiteCode_dev's commentslogin

When doing dev, there a be other problems with pyc files. On of them is deleting the original py file, but still being able to import the module because Python will import any pyc file it if exist.

So if performances allow you to do so, disabling them when you dev is not a bad idea, as long as you keep them enabled when you run CI.

IMO, when you dev, you should set PYTHONHASHSEED (random is predictible), PYTHONDEVMODE (verbose warnings + tooling + sys.flags.dev_mode=True) and PYTHONDONTWRITEBYTECODE (remove the need for cleaning them).

In CI and prod, you should make sure those are NOT set, and use PYTHONOPTIMIZE=2 (remove assert, __debug__ lines and docstring) if you trust your dependancies to be well written (which I would check by running tests in a pre-push hook).


I use Linux every day. I love it. But I don't think your list reflects reality.

> Bollox. Gnome/Mate Display-Manager does an excellent job. If you're still using X11 manual configs than what do you expect..

I have to go from multi-screen configuration to configuration because of my job. I happen to try a lot of combinations of connectors, dongles, screen brands/resolutions...

And no, this is really, really not solved. I have problems 1 time out of 3.

> "Linux drivers are usually much worse (they require a lot of tinkering..)" - Broad statements are usually wrong. Especially this one based on an opinion from the 90's.

They are, here. For BT, Wifi, graphic cards and printers at least, I attest that I most of the time have less features and/or stability using Libre drivers.

Does it work well enough for day to day use? Certainly. Can I use my "scan" button on my Epson laser? Absolutly not.

> "Resume after suspend in Linux is unstable and oftentimes doesn't work." - Bollox. Resume from Hybernate - maybe. But suspend works perfectly most of the time.

Most the time is not enough for sometime like suspend. The whole idea is that you _trust_ the computer to restore you work space the way it used to be. If you can't, if sometime it may not boot, or disable the network card (common problem), then it failed.

Can you imagine if VSCode would restore your file tabs, but only, most of the time?

In fact, one of the first thing I do on a fresh install is to go the the "close lid" settings, and disable suspend, because I'm afraid to lose state by mistake.

> "X.org is largely outdated" - duh... the whole list on this topic is dated issues resolved by WM.

X is largely outdated. That why we still have flickers when booting a linux desktop, which means you will never ever make a Mac lover use Linux because they will not trust it.

Yes, appearance matters.

> "Applications development is a major PITA" - this is ridiculous.

If you talk about web or cli apps, yes. But desktop apps, no.

It IS harder to dev a desktop app for linux, because you have to support a HUGE diversity of configurations: distros, versions, desktop managers. Want to add a system tray icon ? Well, are you targetting Gnome 2, Gnome shell or Unity? Want to make a package ? rpm, deb, snap, flatpack or appimage? Want to use QT4? Not installable from the official repos in the last Ubuntu LTS.

Meanwhile, apps that worked in Windows XP often still works on Windows 10.

The alternative is to limit yourself to a weak % of users from something that is already a niche.

I love Linux, but being in denial doesn't help getting it in better shape.

Not to mention this lists is tiny compared to the actual article.


>Most the time is not enough for sometime like suspend. The whole idea is that you _trust_ the computer to restore you work space the way it used to be. If you can't, if sometime it may not boot, or disable the network card (common problem), then it failed.

I do not undertand what you guys are talking about. It works flawlessly 100% of the time on my Librem 15 and Lenovo T400 both in Debian and Qubes OS. Just find proper harware to run GNU/Linux.


> Just find proper harware to run GNU/Linux.

I suspect this is indeed the root of the problem; people tend to just pick any hardware first and then try to run Linux on it, whereas in reality they should pick hardware to run Linux on.

They seem to get this when it comes to Mac, but not Linux for some reason.


> They seem to get this when it comes to Mac, but not Linux for some reason

The reason seems pretty obvious.


Care to elaborate?


Sorry, I thought your post was sarcasm.

Well, you have literally no choice for macOs supported hardware. Only apple devices.

Whereas for Linux, you can in theory buy any laptop (there is no official list of supported laptops).

So it's not that they get it for Mac, it's just the lack of choice.


But that's precisely my point.

You don't really have a choice of just any random hardware anywhere, not on Linux, not on macOS, not even Windows.

The only difference is that most BestBuy laptops come preinstalled with Windows, so of course they're going to support Windows. The support is often not excellent either, but in Windows world that's somehow just dismissed as expected or something.

Also, try picking a TALOS II POWER9 workstation and installing Windows on it. Would not go well.

My point being, people should buy Linux-supported hardware, preferably from a reputable Linux-friendly vendor, which would spare them a lot of trouble.

They're already doing this with macOS and they're doing it with Windows too, just not realizing it because that's what comes with the majority of PCs.


My point is that no-one is doing it on purpose for Mac or windows. They either have no choice (for Mac) or it comes by default (for windows).

What you're asking is for people to do it on purpose, which is harder, and requires a lot of thoughts into it, reading about hardware vendors, ..


I agree it's harder, but my opinion is that at this point we should market Linux on the desktop as being a very specific thing requiring hardware explicitly for that purpose and should try to work with vendors to make it an option for more hardware, rather than trying to support everything and it not being the best experience as a result.


Yeah, I agree with that.


but it's very hard to do. if you have some arbitrary requirement (for me it's hidpi[0] and enough ram), it may be very hard to find a provider who meets your requirement, or a review for a product that is on the market. (today, it seems i could get a purism machine, but i couldn't last time i was in the market in 2017.)

unfortunately, for linux users, hit-or-miss may be your best option.

[0] i guess my eyesight started degrading, and now i find low res screens confusing. glasses help but hidpi is important too.


It used to be somewhat hard, agreed, but this is a "2020 edition", where now you have plenty of options, from Purism to System76 to DELL and soon Lenovo - all of these would meet your requirements as listed.

What frustrates me is that these lists get posted every year and never seem to get updated. It's like it's not from daily experience, but still the same list compiled when they tried Mandrake Linux 15 years ago.


Oh, but I do research a lot my machines before.

There are many problems with that:

- a casual user will just not do it. They don't have the know-how, but would they have it, they don't care enough to do so. They just want something that works. For now, Linux is and remains a tech saavy user systems, which is and will always remains a niche. Which of course means less resources will be put into it.

- it's hard, and time consumming. It take no joy in hunting down "that one laptop" that will work everytime I want to buy a new machine. Then being stressed out when I actually install the system, because of course I will never be 100% be right. There will always be something.

- you get twice the constraints. Search for a quality machine AND search for machine that will work on Linux.

- it's not permanent. Upgrade may just break support.

- it's not total support. I love my 2in1 xps. I chose it knowing I couldn't use the webcam and that I would have to forgo the tablet mode use. It's a compromise I had to make. Which a windows user doesn't have to make.

- it means I have 300 great laptops I could potentially buy, but in reality probably only 10 among the most recent ones.

- it means you never get to enjoy modernity. You always have to wait one year or two after a tech is popular to get decent support.

None of this is a deal breaker for _me_. They are mostly 1st world people problems, meaning problems I'm happy to have :)

But hand waving is not how we should approach this.


It all depends on your experience doesn't it. Last time I switched back from Windows to Linux, it was because my laptop's Intel wifi card would constantly stop working under Windows, and because that laptop had no reliable sleep mechanism because Windows was trying to push something called 'connected standby' that basically prevented the laptop properly entering S3 sleep.


> And no, this is really, really not solved. I have problems 1 time out of 3.

This is surprising, since I do the same for work and don't ever have issues. I don't do anything fancy DPI-wise, though, so that might have to do with it.

> Can you imagine if VSCode would restore your file tabs, but only, most of the time?

I can, since Azure Data Studio (which is VSCode-based, IIRC) has this exact issue. So do browsers, on that note.

My laptop, however, does not have this issue, nor have any other laptops on which I've installed Linux in the last 5 years or so (including a lot of random consumer-grade Dells and HPs and Lenovos, let alone the business-grade ones I prefer to buy for myself or for corporate deployments). I close the lid, it sleeps, I open the lid, it wakes up. Every single time.

> It IS harder to dev a desktop app for linux

I do all my desktop application development on Linux (internal corporate applications, nothing public yet). Developing on Linux with PyQt5 (or PyQt4, or TKinter) is an absolute breeze. It's porting to Windows and macOS that's a royal pain in the behind (and has got me looking at C# and Avalonia, which is similarly painless on Linux and hopefully will be less painful for Windows and macOS than my current PyQt5+fbs flow).

> Want to make a package ? rpm, deb, snap, flatpack or appimage?

AppImage. It works everywhere, and effectively makes cross-distro compatibility a non-issue. There's literally no reason not to use it.

If distros want to build and maintain RPM or APT or AUR or Nix or whatever packages, then that's on them.

> Want to use QT4?

Why would I want to use QT4? QT5 is much nicer all around.


I moved to linux on desktop after windows 8 blasphemy (but I am developer of linux applications for decade)

> "Applications development is a major PITA"

This is so true. But not (or mostly not) due to kernel. The whole problem lies in distributions and libraries, they break the ABI on all levels, even libc is not stable. This is the largest showstopper for linux on desktop - it is a moving target, from system file locations to ABI. It is just horrible.

Too many cooks spoil the broth. And the answer is to just add another layer. Docker. No it is not a answer, it is just another patch to the anarchy. In theory there shouldnt be any need for it.


Playing devil's advocate, the anarchy is a major reason for me choosing Linux. Yes, the choice and "disorganization" of the whole thing can be overwhelming and inconvenient, but once you set up things just the way you want, it hits a sweet spot from me that no other OS can, (I've tried them all).

Granted, I mostly work on server-side and CLI software, but both Qt and GTK were reasonably pleasant for the odd job I needed them to do here and there.

I am on Arch, but I've found that targeting the latest Ubuntu LTS gives you a popular enough denominator of libraries that it isn't too hard to make it work on other distros, (the Arch community will usually even do a good job for you via the AUR).


Then you'll have the Rust/Go devs that static link everything for this reason. And the flatpack/snap/appimage advocates to ship a virtual containers. But of course you forgo the trust you get from the repository maintainers curation, and break the security update mechanism.


> I use Linux every day. I love it. But I don't think your list reflects reality.

> > Bollox. Gnome/Mate Display-Manager does an excellent job. If you're still using X11 manual configs than what do you expect..

> I have to go from multi-screen configuration to configuration because of my job. I happen to try a lot of combinations of connectors, dongles, screen brands/resolutions...

> And no, this is really, really not solved. I have problems 1 time out of 3.

> > "Linux drivers are usually much worse (they require a lot of tinkering..)" - Broad statements are usually wrong. Especially this one based on an opinion from the 90's.

> They are, here. For BT, Wifi, graphic cards and printers at least, I attest that I most of the time have less features and/or stability using Libre drivers.

> Does it work well enough for day to day use? Certainly. Can I use my "scan" button on my Epson laser? Absolutly not.

> > "Resume after suspend in Linux is unstable and oftentimes doesn't work." - Bollox. Resume from Hybernate - maybe. But suspend works perfectly most of the time.

> Most the time is not enough for sometime like suspend. The whole idea is that you _trust_ the computer to restore you work space the way it used to be. If you can't, if sometime it may not boot, or disable the network card (common problem), then it failed.

> Can you imagine if VSCode would restore your file tabs, but only, most of the time?

> In fact, one of the first thing I do on a fresh install is to go the the "close lid" settings, and disable suspend, because I'm afraid to lose state by mistake.

> > "X.org is largely outdated" - duh... the whole list on this topic is dated issues resolved by WM.

> X is largely outdated. That why we still have flickers when booting a linux desktop, which means you will never ever make a Mac lover use Linux because they will not trust it.

> Yes, appearance matters.

> > "Applications development is a major PITA" - this is ridiculous.

> If you talk about web or cli apps, yes. But desktop apps, no.

> It IS harder to dev a desktop app for linux, because you have to support a HUGE diversity of configurations: distros, versions, desktop managers. Want to add a system tray icon ? Well, are you targetting Gnome 2, Gnome shell or Unity? Want to make a package ? rpm, deb, snap, flatpack or appimage? Want to use QT4? Not installable from the official repos in the last Ubuntu LTS.

Have you ever tried teaching someone how to set up a dev environment on windows? The things you need to install is a huge PITA, especially knowing what to install. Here's a challenge for you, try to compile a cython extension using clang. In Linux I just set CC=clang.

> Meanwhile, apps that worked in Windows XP often still works on Windows 10.

Except when they don't which is all to common (let's not even talk about OSX).


> They are, here. For BT, Wifi, graphic cards and printers at least, I attest that I most of the time have less features and/or stability using Libre drivers.

Wonder what bt, wifi, graphic cards and printers are you using. Realtek and Intel are fine, so Broadcom? So are AMD and Intel, so Nvidia?

Yes, you have pick your hardware, you cannot throw Linux on random garbage and expect it to be perfect. You would not expect it from MacOS either.

If your vendor didn't do the integration needed to run the system you want, it means you will have to do it. Yes, it takes time; I prefer the vendor doing it too.

> Does it work well enough for day to day use? Certainly. Can I use my "scan" button on my Epson laser? Absolutly not.

So Epson it is?

I have a Samsung MFP in my office that cannot scan in color with Mac (the result is corrupted file). It scans fine with Linux (needs binary-only driver though, uld) and Windows (which cannot find it on the network half of the time I need it). So you can always find some wrinkle, on all systems.

> Most the time is not enough for sometime like suspend.

I always wondered, what is the use case for hibernate. You have to dump the entire RAM on your SSD, and then read it back on resume. Incredibly wasteful and slow, if your RAM is 16GB+.

Suspend is something else; but I haven't seen it fail in years. Thinking about it, the last time I've seen it fail, it was a Mac.

Anyway, both suspend and hibernate is what the firmware is responsible for; the host OS just calls the firmware and it does the job. So contact your system vendor if it doesn't work for you.

> X is largely outdated.

Yes it is. But did you tried to discuss with some people, that X11 is dead and Wayland is the future, and the sooner the switch is done, the better for all? If they cannot hack in their spacebar room heater (ref: https://xkcd.com/1172/), it is not ready.

> That why we still have flickers when booting a linux desktop, which means you will never ever make a Mac lover use Linux because they will not trust it.

X11 has nothing to do with flickering at boot; it is not even running yet. Anyway, the user-friendly ditros do use BGRT nowadays (Ubuntu starting with 20.04, Fedora for some two or three release cycles). So unless you configure Plymouth to use something else than BGRT, or you are not using UEFI, you should have flickerless boot on a recent distro.

But while we are talking anecdotes, I do have a machine that flickers at boot with Windows 10. Do I blame Windows 10 for that? Does that make Windows 10 unsuitable for a normal user? Of course not, it is UEFI GOP on the graphic card that sets up the 1024x768 mode, despite the native being 4K. No OS is going to do anything with it, it can only read EDID and set different mode for itself, which means it will flicker (btw, thanks, Asus, for never updating the firmware for your card, that AMD did release).

> It IS harder to dev a desktop app for linux, because you have to support a HUGE diversity of configurations: distros, versions, desktop managers.

No, you don't. Package it in flatpak, done. If, for some reason you don't want flatpak (why? do you like going against the grain?), pick a distro that most of your customers are using and do QA on them (most pick duo of Ubuntu and RHEL); if it runs on anything else, it is a nice bonus, but unsupported. If someone is trying to make it run on riced out Gentoo with i3, let them, it is their time and effort.

Do you do QA of your app on all possible Windows releases combined with all possible Citrix releases? We don't either.

> Want to add a system tray icon?

Don't. You are not in Windows 95 anymore. Do exactly the same thing you would do in Android: background service + notifications + frontend. The frontend runs only when it has any window open. When the user closes it's windows, the frontend quits, no remnants running in background. When the user stops the service, your app is down, nothing keeps running. There's nothing more annoying that trying to close an app and it thinking it knows better and keeps running and hiding in systray.

> Want to use QT4? Not installable from the official repos in the last Ubuntu LTS.

Being obsolete has something to do with it. If it was available, would you wonder why it doesn't support HiDPI? You don't target PowerPC Mac either, do you?

> Meanwhile, apps that worked in Windows XP often still works on Windows 10.

Might work on Windows 10. There's a chance, that it will, or it might not. I've seen few of them...

> I love Linux, but being in denial doesn't help getting it in better shape.

It is not denial; it is using same standard for all systems. I run all three desktop systems: Windows, Mac, Linux and all of them have their strengths and warts. Each of them is more suitable for some purposes more than the other two. If Linux users are in denial, so are Windows and Mac ones.


> I always wondered, what is the use case for hibernate. You have to dump the entire RAM on your SSD, and then read it back on resume. Incredibly wasteful and slow, if your RAM is 16GB+.

> Suspend is something else; but I haven't seen it fail in years. Thinking about it, the last time I've seen it fail, it was a Mac.

> Anyway, both suspend and hibernate is what the firmware is responsible for; the host OS just calls the firmware and it does the job. So contact your system vendor if it doesn't work for you.

i rely on hibernate since my battery is dead and non-replaceable. effectively, hibernate has added several years of life to my laptop.

i have found that in linux, hibernate and suspend take a round robin approach in the first few years of a device. now, suspend works and hibernate doesn't. next, hibernate works and suspend doesn't. sometimes there is back-and-forth. and then they both work. definitely seems to require coordination between os and firmware.


I've read the same sort of arguments for 20 years. They haven't changed or evolved.

Either a minority of lucid fans have been right all along about the complains of the masses, or this is denial.


Containers, Flatpak, AppImage, AMD's new drivers, libinput, Glamour drivers, Plymouth, bluez and the whole bluetooth stack, PulseAudio.. none of these are 20 years old. It disingenuous to suggest nothing has changed or improved in 20 years.


Anecdotally, I do have to regularly support Windows systems that are a horrible mess, the difference being that Windows users are expecting this and while they may curse Windows, it's just another case of 'it acting up again' and this is so common that is not even a major point against it anymore.

As for macOS, just search for 'catalina' here on HN and you'd find endless complaints.

Not saying Linux doesn't have problems, but it does seem like its proprietary competitors are given a lot more slack.


It is not masses that complain; it is just few loud mouths.


Do you have a link for that? I often get ask questions about secure messaging, and I always have a hard time giving anything better than "it depends".


——————————

UK example reference: https://www.theregister.co.uk/2019/12/20/uk_conservatives_br...

——————————

EU example reference: https://www.theverge.com/2020/2/24/21150918/european-commiss...

——————————

US example drafted by NSA: https://media.defense.gov/2020/Apr/24/2002288652/-1/-1/0/CSI...

* Note this document is for public release. If you dig around you will find references to deploy US Special Forces even using Signal for official communications within combat zones.

——————————


>US Special Forces even using Signal for official communications within combat zones //

Do they have some other systems for using Signal on, they can't use it on regular mobile phones without being tracked by base-stations, surely?


Thanks a lot.


I'm a strong FOSS proponent, most of my software is released under MIT, I use Linux daily and help others to use open source as much as I can.

And yet I find this idea total BS.

The reason I could even migrate to Linux when I was a teen was compatibility. It's because free softwares (VLC, Firefox, OOo...) were ported to Windows, a proprietary plateform, so that I could then feel at home later on Linux. And I made a definitive switch thanks to Ubuntu, because it made using most hardware easy, by accepting to use proprietary drivers when required.

Strong arming the industry into Libre never worked. The GPL is one of the least popular licences for this very reason. I, myself, never uses it.

Representing close source as evil doesn't work either.

It's like being vegetarian (which I am) and guilt triping people about meat. It's completly counter productive.

The result we got is that the year of Linux on the desktop never happened, because playing on Linux is hard, because the last cool gadget doesn't work, because you have to carefully chose your laptop to be sure it will work with it (E.G: my XPS 2-in-1 webcam doesn't work on Linux).

It also adds a lot of work to the kernel devs to try to keep up with the outter world, but they have a limited bandwith.

So Linux BT still sucks (even more than regular BT). Wifi is still not on part with the XP on other OS. Batter life is abysmal (I'm dual booting, and the difference is X2).

All for an ideal we never reached anyway.

Help and encouraging for FOSS makes a better world.

Being pushy about it hurts us.


I disagree. For starters, strong arming the industry has worked in a number of cases; this has been covered in a few other comment threads so I won't retread them too much here, but the state of free drivers on Linux is significantly better than it was just a few years ago.

More broadly though: what kind of world do you want to live in? A world where it's easy for companies to create new proprietary software, or a world where it's easy for the users to shape their computing experience?

Look around you, at the consumer devices you own. Can you name one that's: 1) aimed at consumers (i.e. not a dev board, Raspberry Pis and such don't count here) and 2) that you have most or all of the source code to?

Look, if your goal is just "make better software" in general, maybe permissive open source has been a success. But if your goal is to increase practical user freedom, it's been an abject failure. It's just helped companies to make proprietary software more cheaply. And sure, there's some interesting technology that comes out of that for developers, but in the vast majority of cases the users don't see the benefits! Surveillance capitalism marches on, bolstered by the free labor of the community.

This is why I've been so disheartened that parts of the community have turned so anti-copyleft recently. In a world of permissively licensed software, the incentives are stacked against free software for end users. In a world of GPL'd software, you can release your code as GPL too, or you can pay more to write it all yourself.


> the state of free drivers on Linux is significantly better than it was just a few years ago.

Of course, but I think you can thank phones, the internet and the economy of scale for that. Not strong arming.

You will always find examples of times where something worked. But each one I can see 10 examples of companies that said "nope" during one of my mission.

> what kind of world do you want to live in? A world where it's easy for companies to create new proprietary software, or a world where it's easy for the users to shape their computing experience?

It's a false dichotomy. It's not us vs them.

There is no them. It's just all us.

> Surveillance capitalism marches on, bolstered by the free labor of the community.

That's a sociological problem, you won't solve that with software. Or licences.

For that you need to get into politics or business.

> This is why I've been so disheartened that parts of the community have turned so anti-copyleft recently.

I haven't see that at all. Open source has never been so popular. Creative common as well.

What I do see is a complexification of software, that requires more and more resources, which the FOSS world has a hard time to keep up with.

What I do see is a mass of people comming to software that don't care at all about those topics, but only about usability, which the FOSS world has a hard time to keep up with.

I'm glad VLC didn't decide that they couldn't read DVD, or OOo read .doc because there were proprietary formats, otherwise they wouldn't have been the popular softwares of today.

Today LibreOffice is replacing MS office in many administrations and schools. The alternative would have been Google doc.

You think making Linux more restrictive would have forced industrials to be more conciliant?

No, they would just have used more proprietary softwares. The entire stack if necessary, and we would never have had Linux on mobile. Instead, we would have had a black box.

The open source mouvement is not "free to everyone to use and modify, except the ones that don't contribute and profit from it".

That would be a very bitter point of reaction, not positive action.


>There is no them. It's just all us.

In many contexts I would agree, but not here. Say I buy a device, and people discover later that it has surveillance code in it. Why can I not just remove it myself? Who do I need to talk to? The developer of that proprietary code. That's "them."

>That's a sociological problem, you won't solve that with software. Or licences.

>For that you need to get into politics or business.

There are many different angles you can take to approach a problem. One of which is politics, sure: we could outlaw software that surveils its users without consent. That's one approach. But a technical approach is also valid: if all software were free, do you really think users would just accept big tech companies surveilling them? Don't you think we'd have people removing the tracking code, and telling their friends how to do the same?

And by the way, one of the things we see over and over again from the business world is that incentives matter. No, I don't think all companies would have freed their code if the GPL were prevalent, but if it's significantly easier to make their code free than it is to make it proprietary, certainly more of them would. Just look at the state of Linux drivers on x86: how many of them are free these days, and how many of them are proprietary? There are a few high-profile proprietary drivers, but by and large most of them are free. That's meaningful progress! How many of your drivers are free on Windows?

>I haven't see that at all. Open source has never been so popular. Creative common as well.

"Open source" has had a surge in popularity, sure, but its current popularity has more to do with corporate interests than an interest in user freedom - which, of course, was why the term was coined in the first place. As a percentage of that software, the use of copyleft licenses has declined in recent years.

>I'm glad VLC didn't decide that they couldn't read DVD, or OOo read .doc because there were proprietary formats, otherwise they wouldn't have been the popular softwares of today.

Of course supporting existing proprietary formats for the purposes of drawing users to free software is a good thing! I just don't think we should be contributing to the development of more proprietary software. VLC and LibreOffice are both extremely valuable free software projects. (And by the way, they're both copyleft - OpenOffice isn't, but most development has shifted to LibreOffice these days.)

>The open source mouvement is not "free to everyone to use and modify, except the ones that don't contribute and profit from it".

Of course not! Anyone is free to use and modify any free software, including copyleft software. But I think copyleft is a valuable tool to help us build a software commons.


I see that in the Python and JS world sometimes.

The fact is, most scripting languages are written using a compiled one, often C, under the hood.

And those implementations have been optimized for decades.

A lot of hot loops are such because of complex sorting, filtering, and so on, and those are quite hard to do right.

Hence, it's possible to write a way slower implementation using C manually than using the automatic way in Python or JS, not to mention the later has a very good JIT.

If you add the fact some code may use crazy fast libs (like numpy in Python), then you may end up in a situtation where you have to really know what you are doing to implement a faster version.

However, somebody knowing what they are doing would measure and find the bottle neck first before talking of a rewrite, and suggest it only if it's worth it.

Which it can be, of course. Mercurial, Dropbox and Youtube are all good stories about it.

But the junior thats been tasked to improve the UX is probably not the best to advice you on the matter.


I sometimes use DDG for getting a UUID too :)

https://duckduckgo.com/?q=uuid&t=hj&ia=answer

If you like that kind of things and you dig Python, the lib "faker" (https://faker.readthedocs.io/en/master/) is a real threat:

  from faker import Faker
  fake = Faker()

  fake.name()
  # 'Lucy Cechtelar'

  fake.address()
  # '426 Jordy Lodge
  #  Cartwrightshire, SC 88120-6700'

  fake.text()
  # 'Sint velit eveniet. Rerum atque repellat voluptatem quia rerum. Numquam excepturi
  ...
There are many different generators, and you can even use l10n to generate names, phone numebers, plates, IP or ssn from different parts of the world.

It's amazing to create test data, either for dev or teaching, and I just have it autoimported in my PYTHONSTARTUP because of it.

It also plays well with "factoryboy" (https://factoryboy.readthedocs.io/en/latest/) to mass generate complex data set, and even add them automatically to the database.


This is the equivalent library for PHP https://github.com/fzaninotto/Faker


Nice to see we all copied that from perl.


And this is the equivalent for Ruby:

https://github.com/faker-ruby/faker


Don't worry, in 15 years they will use all the money they gained doing this, and start acting well again. Better even.

And everybody will forgive, and people will defend them on HN for being redeemed, and they will be alright.

That's been the trend.


Microsoft only seem to be acting well again because they're in the company of entities acting worse.

Google will have to be specifically not leading the charge, which may well be the case in 15 years. Google will be hiding behind a shield of some other companies worse behaviour in order to seem to be behaving well.

Windows is still going backwards in terms of hostility to users.


Truly, I don't know how one can look at Windows 10 and think Microsoft has in any way improved: the entire OS spies on you at every corner now, bloatware automatically installs in the background, updates are installed without your consent, and more.


> Windows is still going backwards in terms of hostility to users.

Do you mean that Windows is becoming more user-hostile, or less?


Hahaha, yes, my wording is ambiguous, but you can probably gather from the tone and content of the first sentence that I mean it's becoming more user-hostile.


Well, you can put catchas in case you doubt there is a bot activity.

They also have so many phone number and email they can use that to do additional checks.

They you can use the network effect: bots usually have something in commons: topics, likes, ip, followers...

If you have millions at your disposal, something can be done.

They are not, so I'm more encline to assume there is benefit for them to avoid doing so.


Would it take millions?


At twitter scale, doing anything is costly.


Would you keep a bot that just massively upvote things so that they reached frontpage, then rented to the higest PR firm bidder?


Wait, you only have a couble hundred jobs, but don't want a single point of failure, but think a database is too much, but talk about cloud hosting?

This all seems contradictory.

Personnaly, using Python, I go for Celery (www.celeryproject.org): it's a persistant daemon that can run tasks, provide queues and shedule work like cron .

A lot of people prefer Python-RQ, as it seems simpler, but the truth is you can start using celery with just the file system for storing tasks and result:

https://www.distributedpython.com/2018/07/03/simple-celery-s...

If your needs grow, you can plug it to redis, rabbit MQ and/or a database later.

It can expose an API so that other languages can talk to it and trigger tasks or retrieve results (but not write tasks, they must be in python).


Celery for scheduled jobs seem to not be a supported design pattern at all, and any job that starts to potentially come close to the 1 hour timeout seems to get annoying to work with in celery. It seems primarily designed to send emails in response to web requests, which is not the use case most people are discussing here.


I don't see how you came to this idea. The jobs can be as long as you want, you can have retry, persistant queues, priorities, and dependancies.

Of course, I would advice to put a dedicated queue for very long running tasks, and set worker_prefetch_multiplier to 1 as the doc recommand for long running tasks: https://docs.celeryproject.org/en/stable/userguide/optimizin...

With flowers (https://flower.readthedocs.io/en/latest/), you can even monitor the whole thing or deal with it manually.

I assume your comment is reporting on other comments, but not direct experience?


Direct experience very fresh in memory :)

The issue with long running tasks is that you have to change the timeout to longer than the default value of one hour (otherwise the scheduler assumes the job is lost and requeues it). But this is a global parameter across all queues so this means we essentially loose the one good feature of celery for small tasks which is retrying lost tasks within some acceptable timeframe.

Further flower seems weird - half the panels don't work when connecting through our servers; our vpc settings are a bit bespoke but not completely out there, so it's not fully useful. Also flower only keeps track of tasks queued after you start the dashboard (but then it accumulates a laundry list of dead workers across deployments if you keep it running continuosly).

We were also excited to use it's chaining and chord features but went into a series of bugs we couldn't dig ourselves out of when tasks crashed inside a chord (went into permanent loops). I just declared bankruptcy on these features and we implemented chaining ourselves.

Point is, I'm sure we got some parameters wrong, but I and another engineers spent WEEKS wrangling with celery to at least get it running somewhat acceptably. That seems a bit too much. We are not L10 Google engineers for sure but we aren't stupid either. The only stupid decision we made was probably choosing celery from what I can see.

In the end we still keep celery for the on demand async tasks that run in a few minutes. For scheduled tasks that run weekly, we just implemented our own scheduler (that runs in the background in our webservers in the same elastic beanstalk deployment) that uses regular rdbms backend and does things as we want. Turns out it's just a few hundred lines of simple python.


Fair enough and very honest.

> But this is a global parameter across all queues so this means we essentially loose the one good feature of celery for small tasks which is retrying lost tasks within some acceptable timeframe.

Oh, for this you just setup two celery deamon, each one with their own queues and config. I usually don't want my long running task on the same instance than the short ones anyway.

> We were also excited to use it's chaining and chord features but went into a series of bugs we couldn't dig ourselves out of when tasks crashed inside a chord (went into permanent loops). I just declared bankruptcy on these features and we implemented chaining ourselves.

Granted on that one, they not the best part of celery.

Just out of curiosity, which broken and result backend did you use for celery?

I mostly use Redis as I had plenty of problems with Rabbit MQ, and wonder if you didn't have those because of it.


Our use case was that the timescale of any of our tasks (depending how complex a query the user makes) can go from 1 minute to 45 minutes. We demoed a new task that occassionally went over the 1 hour mark. It's definitely annoying to have separate queues for these tasks but that might be what we need to do!

We use redis. FWIW within the narrow limits of the task properties it's remarkably stable,so no complaints on that!


Yes, I feel the same after spending several weeks to deal with celery, redis, docker compose config, flower, set up celery workflow, rate limit, worker, etc. Testing for a really long time...

When there is a workflow to play with chord & chain, it became unintuitive to find out the issue. I was stuck in an issue and finally I posted on SO to ask for help. Luckily I got an answer

I thought it's my problem due to no experience in scheduling stuffs. I hope there is something which is simpler


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: