Ex-LD engineer here. This was a classic mistake we saw most of our customers make, before they had to switch to LD or its competitors when they scaled a little bit more. Using a db is fine but can put heavy load on the server depending on the number of flags. In addition, it doesn't give you various features such as tagetting by users, segments, etc. Nor does it give you things like progressive rollout or automated rollback (yes we were using that in house and had saved us at least a few times).
I'm not advocating for LD, I'm advocating for any feature management system.
> In addition, it doesn't give you various features such as tagetting by users, segments, etc. Nor does it give you things like progressive rollout or automated rollback (yes we were using that in house and had saved us at least a few times).
These are things you can think of quite easily, or you just survey the landscape of what challenges other implementations run into and they inform you of this. The first iteration of our design had all of those; we could target users, always-subscribe internal users, we could have two features have the same segment/ordering of rollout, etc.
Database load wasn't ever a problem, because you're talking about quite literally a single SELECT on an already established & otherwise in use DB connection. But caching is not that much to bolt on afterwards should it become an issue.
It's never been clear to me that the simplicity of understanding that a single query has, even if it might be short a few features, is outweighed by the complexity of a full dependency to an external service.
As it was, our FF system outlived the company, and was sunset when the company ended.
Eh, I built a feature flagging system for service with millions of daily active users and it: wasn't hard to do, was easy to extend to support targeting, segments, and progressive rollout, and on top of it flags are easy to cache, so the database pressure was not that big. We did this long before LLMs, and it occupied maybe a month of engineering time for me, here and there as we needed new features, while I was doing other things. We did choose the wrong hash for bucketing in the beginning, but we noticed the stairstep patterns in our buckets quickly (an advantage of having millions of requests per day to measure) and picked a better one. It was much better this way since it meant backend flags without a roundtrip to a 3rd party service (that was the main motivation, we wanted this kind of control without adding latency to our requests). I remain baffled by launch darkly or other such services. And today, with LLMs, just build it yourself is hard to argue against.
As I mentioned, I'm not advocating for LD. I'm advocating for feature management systems (over env vars/db values and other similar naive implementations). The build vs buy is a separate argument.
> that was the main motivation, we wanted this kind of control without adding latency to our requests
Right, and most, if not all, off the shelf feature management systems have this problem solved already. For example LD uses a streaming architecture by default to push out flags to your application within 200ms, for client facing applications (android/ios/web), flags were retrieved once on load from a CDN, keeping the rest of your application snappy. I believe Unleash also has some sort of streaming architecture.
Feature management goes well beyond an env var. Sure an env var change is fine on it's own, but a change of the feature flag (essentially a release) requires a deployment.
Feature management separates these two. You can deploy without releasing. Because the code is there, but dormant behind an if that can change at runtime, you can actually target users, segments, or even do progressive rollout, and so much more. What this also means is you can test your changes in production without causing prod to go down. Best of all, a release doesn't mean a bad deployment.
It's not just about non-tech folks. It separates deploy and release in the SDLC, and that opens up a whole new world.
I used LaunchDarkly I agree but it depends on your practices.
Proper CI/CD means potentials hundreds of deployments a week. A deployment is nothing in some environments. And the change gets tracked into the environment that has it switched, into our logs and other observaility platforms.
So we get the same info. LaunchDarkly was amazing when coordinating with stakeholders and product owners. Especially for testing features, A/B, etc.
I'm definitely not trying to downplay it, but starting with just ENVs for each place is simplest and then they can expand into tools they need like LaunchDarkly or PostHog.
But starting there for small engi teams isn't as valuable. That's all, not against it for a product that builds to it, but just too much churn for smaller teams to use. I still advise ENV first.
And yah the realtime flags of LaunchDarkly were amazing , I needed that a few times.
Illegal, where exactly? In various western countries ISPs are by law required to block some websites. In other countries, ISPs ar eby law required to have escape hatches to be able to shut down the internet. In yet others, everything runs behind a firewall. Take your pick, I guess?
I've watched anime since I was about 19 (I'm 36 now), I've barely learned much. If you have tips on how to learn from watching anime, I'd love to hear them. I still watch anime, though much less, but it'd be helpful if I can learn still
I'd be curious to know more about this. Dark Reader, at least on Firefox, is a recommended addon, which means it gets additional scrutiny by Mozilla. I do agree though that if it does additional web requests, it seems fishy. Do you have any details on where these requests go to or how you found them?
- Checking "Enable remote debugging" and "Enable chrome and add-on debugging toolboxes" in devtools settings, opening Browser Toolbox (Ctrl+Shift+I), and looking at the "Network" tab
or
- Enable add-on debugging and inspect the addon from `about:debugging`
For me I cannot deal with dark mode in bright environments, but equally it's fairly hard (though less hard) when using light mode in dark environments. It's why Dark Reader has been such a huge quality of life improvement for me.
I keep seeing this, and I still think it's a ridiculous ask. Whether you like it or not, the vast majority is not savvy enough to think about this, and they generally just want Netflix to work without an additional service. For a majority of people 600 is expensive, and they don't want to have to spend another 30-100 depending on the device.
For the past few days I've been trying to use rea to reverse engineer my Samsung Galaxy Buds 4 Pro to get seamless functionality working outside of the Samsung ecosystem with no luck (the buds 4 pro require Samsung devices + a Samsung account for seamless to work, I only have a Galaxy phone + my macbook). I think we're still far away from having something that works that well.
I'm curious what made think of it in the first place? What was your thought process that made you think that these Linux primitives give the opportunity to make it a platform for distributing and deploying packaged applications and eliminating a lot of the issue around "it works my machine" argument?
You may have already answered this somewhere, so feel free to tell me what to search for or if you have a link to hand, that works too :)
Docker in its earlier incarnations was based on openvz, which predates lxc. The lxc project was initially a kernel development project: a collection of kernel patches to add container support. The userland tool was just a companion. When it became apparent that these patches would "win" the race to be upstreamed even though openvz was much more mature, we switched our tooling to lxc. The patches matured over time, but the userland tool (what we now know as "lxc") was finicky.
The original open-source release of Docker wrapped lxc for low-level container operations. But it was just too unreliable and we swapped it out for an in-house Go implementation called "libcontainer".
As for "commercializing": Docker was famously bad at commercializing anything, we open sourced so much (maybe too much) and still people complained that we were not open enough.
“Docker was famously bad at commercializing anything, we open sourced so much (maybe too much) and still people complained that we were not open enough”
Developers are a tough market to please, especially when your business revolves around open source software.
Unless you have an option to do private cloud hosting (like all these databases and other platforms do), it’s incredibly difficult to monetize.
In the case of docker, I think it’s in the nature of the product that it’s difficult to monetize. Best you can do is enterprise support contracts, but those aren’t good enough to justify all that VC funding.
I don't think it's necessarily that difficult to monetize, they just ignored a lot of the needs of enterprise users (rootless, not punching holes in the firewall, not fighting with systemd, private registries) for long enough that other projects (e.g. Podman, Quay) filled the void.
None of those things were obstacles to commercialization. None of them were common requests from enterprise customers except private registries, which Docker did build.
Podman has never been commercialized.
Docker's difficulties in commercializing were self-inflicted. Enterprise and open source teams were culturally siloed and as a result the enterprise products were not "native". There was a lack of focus and a "checkbox" approach to building product. There was no singular "dictator" who could enforce a coherent product design and vision, and it showed. etc.
I was a DotCloud user before Docker, and I remember keenly thinking - “oh, is that how they went from hosting only PHP to any language/stack?”
I always assumed it was the same internal tech, now productized. Then of course, it became THE tech, overtaking vagrant and anything else for local development and deployment. The latter was the true money shot. Server runs exactly what you test with locally.
BTW just for giggles opened hub.docker.com with Network Tools opened:
368 requests, 9.62MB, Finish: 4.50s.
For a thing which should be a static cached copy with everything bundled. Sure, it's a progress from the previous iterations when it could hang for 30 seconds before actually loading... but this is still ridiculous.
>>[0] 'Bridge' was and still is an established network term for joining two broadcast domains into one. Why the hell you decided to name your NAT'ed network layer a 'bridge'?
Like they told you 9 months ago, the terminology is exactly right. They're called "bridge" networks because they use Linux `bridge` devices to bridge `tap` adapters of all containers attached to the same Docker network.
There's nothing exotic about this terminology at all and complaining about it betrays your own lack of knowledge.
> PS there is still (AFAIK) no support for a transaction-like image building in Compose. So instead of something like this:
Do you mean Dockerfiles? RUN command has had heredoc support for 5 years now.
> Like they told you 9 months ago, the terminology is exactly right
Oh, well. I guess you ride to your work on wheels, right? Because while you sit in the car/bus/train it's the wheels what actually rides on the road/track, right?
And if you bothered to look at that comment then why do you ignore what 'macvlan' - which is an actual bridge both in terms of a Linux device and the functionality - is not called 'bridge'?
Because you don't understand the problem. More so, you ignore the actual question there about the broadcast domains.
No, if even if you use a bridge internally when your container is on a isolated network which is invisible to anyone on your network and the container access the network behind your host through the NAT - this can't be called a bridge.
You see, if they just called it as it should, ig:
bridge => hostnatnetwork // or internalbridgenatnetwork or whatever else
macvlan => bridge
then they would be both named correctly and be the technically correct - and 99% of Docker users couldn't tell a difference - because they are not net or sys administrators.
But the last 1% is the people who actually makes the workspaces for these developers where they actually do run their fancy-shmancy containers. And for that 1% it surely pays naming this as they are.
And there is also that dumb decision to silently rewrite 127.0.0.1 to Google DNS without a question. Especially when 98% of Docker users are incapable to diagnose it by themselves.
So it's not only the naming of a network functions.
But hey, I'm just knowledge lacking guy without experience at all, just yesterday saw a computer a first time in my life, sure, sure.
> RUN command has had heredoc support for 5 years now
So ten years late? Adoption is off the limits, as you can see - that snippet is from the official Docker image.
And if I need to run multiple commands and not just one RUN statement? Like the people are asking for years? And you so knowledgeable person of course do know the reason for that, right?
I think there are some interesting points there, but your rude, arrogant, and aggrieved tone overwhelms them.
If you decide to edit your tone, I'm happy to discuss more here re: filesystem transactionality tradeoffs, since that's something I've struggled with in both directions WRT containers.
I'm only responding in kind to GP - no less but not more.
I think you are the one who built an alternate layering system? On mobile, don't like to search rn.
If yes the basics are in the first comment- just not limited to the one RUN statement ofc. And if I do remember how the layers are made then making it transaction-like would be a great dedup saver eg for adding a static rarely changed artifacts - there could be essentially a ~one layer for how many you want images.
Implementation wouldn't be so difficult - you just stop committing until TRAN END; but the main is too provide a mental model for a not very experiencied dev how this supports his work and makes it more efficient on the storage and execution.
>I'm only responding in kind to GP - no less but not more
bullshit, you've been cunty right out of the gate since your first reply. I'm just surprised anyone else in the thread had as much patience as they've shown here.
you seem exactly like the type of person who is incapable of realizing they're a complete asshole and wonders why people choose to avoid them.
> And if you bothered to look at that comment then why do you ignore what 'macvlan' - which is an actual bridge both in terms of a Linux device and the functionality - is not called 'bridge'?
Because renaming things that map 1:1 onto the underlying concepts is a terrible idea that makes learning and communication harder by introducing ambiguity and indirection. "bridge" now means multiple things and not only is it needlessly obfuscated, it's wrong (explained later). I _hate_ projects that do this.
> Because you don't understand the problem. More so, you ignore the actual question there about the broadcast domains.
A Docker 'bridge' network driver creates a broadcast domain between containers attached to that network.
Rhetorical question, maybe it'll help you find a source of misunderstanding: Why do you expect it to join the broadcast domains of your containers and your LAN? And which LAN, exactly?
> No, if even if you use a bridge internally when your container is on a isolated network which is invisible to anyone on your network and the container access the network behind your host through the NAT - this can't be called a bridge.
Your misunderstanding seems to stem from the flawed assumption/expectation that every container is automatically attached to your LAN. Therefore you incorrectly expect and demand that term "bridge" accurately describe the logical connection between your container and your LAN.
Docker networks are entities independent of your LAN, they're isolated networks that primarily serve to connect containers amongst themselves.
A Docker 'bridge' network is not necessarily routed to LAN either so your suggestion of calling them "hostnatnetwork" is wrong. Auto-configuration of NAT between the internal bridge network and your LAN is just a convenience feature which you can disable, it's not an inherent property of bridge networks.
> But hey, I'm just knowledge lacking guy without experience at all, just yesterday saw a computer a first time in my life, sure, sure.
No I didn't say that, I think you suffer from the curse of having a lot of previous experience in an adjacent area where you built a strong mental model of how things ought to work, but now you're trying to forcefully apply this same mental model onto Docker.
All of this networking terminology is consistent and applied correctly but for some reason it's not clicking with your previous knowledge and you're trying to fit a square peg through a round hole. Take a step back, investigate.
> So ten years late?
Don't shift the goalposts. You claimed it still doesn't exist.
> No I didn't say that, I think you suffer from the curse of having a lot of previous experience in an adjacent area where you built a strong mental model of how things ought to work, but now you're trying to forcefully apply this same mental model onto Docker.
I think you just proved his point. Docker came onto the scene and hit roadblocks in enterprises where sysadmins lived. These architectural and terminology choices ran head on into well established domains. Yes, docker did change things, but I think his point is it would’ve had more success with paying customers if the authors of docker had a better understanding of established network administration and system administration concepts from the start (though these were the early devops days…). That lack of understanding caused adoption friction in enterprise space, and different choices could’ve prevented that without changing at all the paradigm of what docker is.
The docker mental model isn’t crazy different. Design choices made it unnecessarily less compatible with existing mental models. This probably contributed to the feeling of it being revolutionary, but impeded its adoption with paying customers.
In my experience adoption was primarily driven by developers and sacrificing clarity would hurt adoption. This perfect familiarity was never achievable anyhow. Docker was a completely new paradigm in software development, similar to how modern cars aren't just horse carriages without the horse. It transformed everything from development, testing, deployment, updates.
Experienced sysadmins often tried to approach it with "VMs but with less overhead" mentality, which couldn't possibly work. Coworkers would ask me how they can install SSH daemons in containers to update packages and view logs, or they would refuse to adopt Dockerfiles, sometimes setting up the container manually and exporting the image. They would wonder how to install many different services in 1 container and handle crashes, etc. It was a huge shift.
> Design choices made it unnecessarily less compatible with existing mental models.
Again, I disagree. Whose mental model do you prioritize? How invested are you in making Docker work just like whatever elaborate system those enterprise admins used before?
Let's not forget that by creating new abstractions, you're penalizing users who already possess foundational knowledge of the primitives. If you do this then I'm probably going to get a whiff that you care more about enterprise sales than developer experience (duh) which is a huge red flag for me.
> Because renaming things that map 1:1 onto the underlying concepts is a terrible idea that makes
Sorry, but you do insist on naming 'docker' bridge a bridge because it's underlying tech is a bridge. Make up your mind.
> A Docker 'bridge' network driver creates a broadcast domain between containers attached to that network.
And noone absolutely ever cared about what it's a bridge down there. It could be using gronklew and maybe 20 people on the planet would care to learn what it is.
You are quite bent on it, but nobody cares how the containers talk between each other.
> Why do you expect it to join the broadcast domains of your containers and your LAN?
I expect a setting named 'bridge' in some $thing to actually implement a bridge.
Just like you pressing an AC button in your car expect a very specific thing to turn on/off and not 'Artifically flavoured windshield wipers served in a Container'.
Why do I want it is on me. Maybe I need it. Maybe I control my network and freely give out addresses to anything. Maybe I don't run on some cheap VPS somewere with only 1 IP I can't choose nor change. Bazillion of options. None matters because this is not about 'why do you want it'.
> And which LAN, exactly?
The path of a least automagik.
Only one ethernet interface? Then connect to that ethernet interface. Bitch in the logs what automagik was used and this can lead to the problems.
Many interfaces? Just shutdown (least pain for newbies) or start without attaching (harder to troubleshoot for newbies). Bitch in the logs in either case.
> Your misunderstanding seems to stem from the flawed assumption/expectation that every container is automatically attached to your LAN
'Your misunderstanding seems to stem what there are some people assume what they are driving in like c-a-a-r. When even 5y.o. knows it's the wheels what drives you around.'
> Docker networks are entities independent of your LAN, they're isolated networks that primarily serve to connect containers amongst themselves.
Yes, exactly. Don't call it 'bridge' ~20 years ago and we don't have this talk here.
> Auto-configuration of NAT between the internal bridge network and your LAN is just a convenience feature which you can disable, it's not an inherent property of bridge networks.
It's not even a property of a bridge network. For someone claming knowledge superiority this is a quite lame statement.
> links
Really.
> previous experience in an adjacent area where
Ah, yes, irrelevant 'adjacent area' of computer networks. I'm out of words.[0]
> All of this networking terminology is consistent and applied correctly
No. Go drive your wheels home.
> Don't shift the goalposts. You claimed it still doesn't exist.
I'm just mentioned RUN as an example. You decided what multiline is the goalpost.
And yes, if even the official Docker images doesn't use RUN heredocs then... there is something wrong with them. Like being implemented too late.
[0] but perfectly explains your previous 'flawed assumption'
> And noone absolutely ever cared about what it's a bridge down there.
Just like nobody ever cared that the container isn't bridged to their LAN in the first place - in fact the isolation is a feature.
But you should care, it's only the most common building block of network isolation for containers/VMs on Linux.
---
I'm not going to get into it further. I tried to explain nicely but you seem hell bent on complaining and not learning anything new. At this point the appropriate response is: RTFM, it's all in there, their docs are great.
If you don't have a clue about how things work under the hood and if you have no interest in learning anything, then stop complaining about imaginary technical inaccuracies.
I'm skipping it because it's a waste of everyone's time, you're incapable of accepting new information, but I'd be happy to be proven wrong:
First you incorrectly claimed that multi-line RUN commands aren't supported still require that ugly legacy syntax. When that was proven wrong, instead of saying "good, I wasn't aware", you went on a tangent about its adoption and wanted to interrogate my knowledge -- Yes, obviously I know that running all commands in one RUN block is beneficial because it reduces layer count and image size.
Your revolutionary idea of "transactions" is already implemented through multi-stage builds. Except those are even better than transactions because individual steps are still cached, independent components can be built in parallel and they don't invalidate each other's cache. Finally the artifacts are copied into the final image, leaving behind unnecessary layers.
I never called you stupid. You were needlessly aggressive 9 months ago, you were extremely aggressive before I weighed in yesterday, and now you're playing the victim, again.
> BTW just for giggles opened hub.docker.com with Network Tools opened: 368 requests, 9.62MB, Finish: 4.50s.
lol What on god's green earth does this have to do with commercializing Docker? Do you really think better lighthouse scores on hub.docker.com was what was holding them back?
Did you miss 'up to 30 secs' in the previous comment? It's not about lighthouse scores. It's the whole feeling of a hodge-podged project which is run by people who are - maybe - passionate but ultimately lacking in the basics.
The ultimate buyers of their product was the system operators and the system owners - not the developers per se. They failed hard everything that space. Having a dog slow the main content site didn't added anything positive.
> As for "commercializing": Docker was famously bad at commercializing anything, we open sourced so much (maybe too much) and still people complained that we were not open enough.
yes, and no. Probably both. Docker would not become such a default standard if it wasnt for all the open sourcing.
Earlier incarnations of the project (2008-2011) were python based on openvz+aufs, with the very first prototypes (2007-08) wrapping vserver. We started the transition to lxc+aufs in 2011. Open-sourced the lxc wrapper in 2013. Replaced it with libcontainer in... 2014 I believe?
The LXC way of getting an up per container is pretty great - I’m basic bitch with this sort of thing so I assume it’s less of an issue for the experts but really enjoyed the IP per container when I hit it.
This is sometimes easier, but once you have a docker-compose of what you want, it’s hard to beat. You can have running server with all the services and network shares all sorted in minutes.
Its nice having the flexibility for both system containers and application containers. It does feel though that most people forget that nspawn does both, and that it only can do system containers.
Agreed, I briefly forgot it can do both. Thanks for the reminder.
I went down the rabbit hole of Incus and once I realized it requires the full network aspects to be configured, then I figured using KVM, QEMU, Libvirt was more practical. Then stumbled into nspawn and it was basically ready to go.
Although, I haven’t done enough research to know if Incus provides certain useful capabilities.
I'm a huge fan of Incus. VMs, LXC, and OCI containers all under the same Docker-like CLI interface. I run it on a local server and on a dedicated server in a local DC, and connect to both remotely using Nebula.
It includes project-level resource separation, advanced networking, clustering, snapshots, and more. Plus, Stéphane Graber is super responsive on the forums and issue tracker. I bounced off of Proxmox, and can't recommend Incus more for hypervisor-type scenarios.
Do you have any experience with multi-node operation with Incus? I like the simplicity of Proxmox for moving VMs and LXCs around but I haven't settled on a solution for OCI. K3S is an option but it's a big jump. I'm moving towards Fedora CoreOS VMs at the moment for live migration and security isolation.
I run Incus, but I had to uncluster my home lab to allow to have zfs duplicated containers (as cold standby) on another physical instance. It may have changed now, but it was not possible to have two near identical container, one up one down, on the same cluster.
I'm not advocating for LD, I'm advocating for any feature management system.
reply