This first happened in France over three weeks ago. It's unforgivable that many chains have failed to update their software in time. Or to take the freezers offline if an update isn't available.
I now read some interviews with local chains in Sweden and Finland, and they were still just investigating what to do at this point.
>Or to take the freezers offline if an update isn't available.
Why the heck would this be necessary? We have VPNs holy crap, the entire genesis of the technology was to solve this fucking problem. If it's not literally intended for the general random public to be able to randomly visit and use themselves, it shouldn't be exposed on the public internet it should be (heavily restricted) LAN-only with remote access done via VPN. While it's even better if every single last thing is well secured at the client-level too, it goes a long way with ancient (or new, but with ancient-equivalent software design expenditure/decisions) kit to protect it at the network level.
Creating a false binary of "take it offline!" when businesses have very good reasons for networking many classes of industrial hardware seems self-defeating vs "just get the cheap bare minimum basic network protection done". I mean shit, even if somehow they've gotten suckered into freezers which demand access to a non-self-hostable manufacturer server (hey EU! here's something useful to regulate! if nothing else order manufacturers to have WireGuard end points for connection) they could at least have the firewall allow !RFC1918 solely and exclusively to that address on those ports.
I dont think VPNs are the correct solution. They are one solution, sure, but if it is the correct solution depends on many things.
For example, where does the VPN connect to? The vendor? Do you trust the vendor more than the internet. Sure. Do you trust the vendor, that does not manage the firmware of their devices and uses inbound connections in the first place, enough to put them on a VPN?
How many devices and vendors do you have? How many VPNs can you establish with the existing networking hardware, and how many can you manage without IP clashes etc. IP clashes work both ways, btw. The vendor has to manage this themselves also if all their clients want VPNs.
Retailers are enterprises with 100s of vendors that run on a shoestring. On top, they use a lot of legacy architecture and hardware. They have teams that have a hard time making an architectural decision in the best of times, but even less so under stress. It is probably the same on the vendors side.
In my mind, the correct solution is indeed to deny any inbound connection to the network, now, to prevent further damage.
Incident response is, in order:
1. preparation (failed)
2. identification (done)
3. containment <--- we are here (take offline, put manual rota in place to manage fridge defrost cycles)
4. eradication (VPN, etc)
5. recovery (take it back online, relieve manual rota)
6. postmortem etc
However, we're talking about companies that put unprotected systems on the public network and did nothing about it even after getting three weeks advance warning.
Seems like the biggest chain in Sweden, ICA, had a VPN. They were affected but only the franchise takers that were (for whatever reason) outside their internal network.
>However, we're talking about companies that put unprotected systems on the public network and did nothing about it even after getting three weeks advance warning.
I don't know the companies in question here, I'm not European and my EU family lives in France not Belgium or Sweden, but fwiw here in the states a lot of supermarkets are independent grocers or in a sort of formal or quasi-franchise arrangement where it's part of a "chain" but each store is still somewhat or quite decentralized in management when it comes to the physical stuff at the local store. Which is probably a strength in a lot of cases, makes it easier to adapt to local environment faster, but I suspect there's just plain a knowledge/skill/priority mismatch when it comes to IT. That'd also line up with some of the reports on this I read where there was some official central management system, except certain grocery stores just weren't using it?
The components of solutions for this exist and could even be really cheap, we're not talking much bandwidth here. Not hard to imagine even in a independent grocer association or loose franchise system they could tape together an SBC with preconfigured OPNsense or VyOS or whatever on it and a switch and be able to just tell stores "here, plug your freezers into this, access the portal here" as a default and it'd just route all their traffic to the franchise/IGA's management system. Someone still has to secure that of course, but at least it wouldn't be on the open internet.
Places that wanted to could of course do this other ways, I'm just saying that even with the very tight margins grocery stores tend to run on I think it's very doable to do better going forward, just will take some coordination and knowledge building. "The future is already here, it's just not evenly distributed" and all that.
The skill, and time, required to "take it offline" is far lower than any solution mentioning VPN. And given that, more likely to succeed while a better solution is found.
Completely speculating as I don't know the situation in Europe, but some possible reasons are:
- Remote alerting for the store manager/head office to quickly respond to a refrigerator failure after hours, providing options for saving valuable stock (e.g. move stock to another refrigerator).
- Maintaining an audit trail for cold chain management (e.g. regulator says you must keep an audit trail proving that a frozen fish has been kept below -30oC since it was processed on a ship all the way up to point of sale in a supermarket).
- Refrigerator manufacturer says a condition of warranty is that the refrigerator must provide remote telemetry proving some warranty condition has been met (e.g. regular cleaning cycle operated).
Note: all these speculated reasons are unidirectional data flows. So even if a valid reason exists for Internet connectivity, why can't it be limited to unidirectional data flows?
Also so the temperature can be controlled to make use of cheaper electricity — if you know it's going to be windy from 02:00 to 04:00, then still until 06:00, you can run the freezers a bit extra while the electricity is super cheap.
.. while they leave lighting on around the building and outside for 24 hours a day; they pay for ads who-knows-where.. The efficiency call is galling to me because we have had thirty years to increase efficiency with miracle tech like LED lighting, and yet little has been done.
> Maintaining an audit trail for cold chain management
It's this, primarily, or most importantly.
I work in the restaurant industry. Health departments require you to assure them that your hot things are kept hot and your cold things are kept cold. The old way was staff walking around the store reading thermometers and writing temperatures down on paper; the newer way was your staff walking around the store inputting temperatures on an iPad connected to your business-management software. The trouble with that is that staff forget to do it (despite threats of discipline, and push notifications to their phones), or even blatantly fake it.
Failing to keep an accurate temperature log means fines, and other consequences, up to and including loss of your health license. I am 100% in favor of this legislation, and solid enforcement of it. The main improvement I would make is to harmonize codes and enforcement across municipalities. I'm in the USA, every county has their own requirements - and their own "portal" for submitting documentation. It's a massive pain in the ass to write Standard Operating Procedures for staff across multiple stores in multiple enforcement districts when there simply aren't any SOPs.
The majority of the fines our (responsible, and genuinely trying-to-do-it-right) company received were things like a staff member moving from a store in County A to a store in County B - just for a few days, to cover someone's PTO, or something - and inadvertently followed County A's logging requirements rather than County B's. Two genuine cases: logging every two hours, instead of every ninety minutes; using whole degrees, instead of tenths. We could appeal, but that takes a lawyer's time, so even if successful it wasn't always worth it.
Real-time data, from connected sensors, is unambiguously better. Staff get loads of time back, to make food, or help guests; all the logging gets done "right", for whatever jurisdiction; data's held by a third-party, so nothing can be faked.
That said, we're not connecting our infrastructure to the internet! We have a separate IoT sensor service that tracks the temps for us. I'm sure their security is also terrible, but that doesn't matter nearly as much. A year of their service is ~80% of the "routine", and totally innocent, fines and other costs that we were previously paying. That's the way all of these stores should go, in my opinion.
If you want to be able to do that monitoring from a central location then at some point you will need some form of internetwork to support transferring data between local networks. And nowadays there is really only the one obvious choice of internetwork.
> there is really only the one obvious choice of internetwork
Well, I wouldn't quite go that far - there's a bunch of alternatives to openvpn.
Hey did you notice how when the freezer connects to the on-prem VPN tunnel, it never actually touches the internet directly and has zero attack surface, and yet can still send the necessary data anywhere in the world? And that could even be bi-directional, sending commands back to the freezer, if you really really think you need that ability for some reason (but you don't).
Some large supermarkets (in the UK at least) sell their capacity to the National Grid so it can be balanced in times of demand. They agree to turn off their freezers for a set amount of time to create a "virtual battery".
Not just that. But why the fuck is the control software so weak? Why does it allow remotely bumping temperatures to near boiling without any checks? Which idiot designed this system? Why aren't they out of business? Same with the people who installed these systems. Why on earth didn't they think to secure it?
Go offer your services to these companies and see their answer.
The big IT firms syphon off all the talent with compensation these fridge and supermarket companies can't match. You won't find much talent there. Who is brilliant with computers and would want to work there?
But look also at how underfunded the basic blocks of IT are, how open-source is left to fend for itself. It is the whole culture of modern IT to hack things together and not make them well-defined and reliable.
Very likely they shipped with an ADMIN/ADMIN username/password and nobody changed it. Because who would want to break in to a freezer, right? It's not like there's bitcoin in there.
We've been telling people for years that naive threat modeling was going to bite everybody one day. And finally, here we are.
Approximately. These models had been deprecated by 5 or 10 years, and a successor model last year had a software update issued because they had a secret "password of the day" that allowed you to use the date as the admin password.
Monitoring doesn't need write access. RS-422 is simple and reliable. It uses separate receive and transmit pairs and you don't need to connect both. The freezer can continually transmit its telemetry over unidirectional RS-422 and a separate system can forward it over the Internet.
Hackers can still tamper with the telemetry and create false reports of problems, but actually changing the thermostat settings would require local access, so there's opportunity for somebody to verify the problem really exists.
On top of the monitoring mentioned by others, it's also to create a centralized log of actual temperatures. This then lets you show inspectors that you are keeping up with refrigeration requirements.
You have to wonder about the monitoring - it apparently wasn’t alerting for someone to do something about it in time. What was the point of the monitoring in that case??
It might have dutifully logged that the temperature setting was changed, which will allow regulators to conclude that the safety of the food stored in these refrigerators can no longer be assured.
Compliance tends to warp the underlying goal quite a bit.
I don’t know about supermarkets but for medical freezers it’s useful to be online for monitoring and calibration.
My org has lots of freezers that need to be at precise temperatures. Sometimes they drift so you have to check them a few times per day, log the check, and adjust as necessary. Doing this for a thousand freezers is tedious and error prone. Networking helps. “Cloud” lets it all be done remotely without getting IT very involved for vpns to be set up with specific vendors. It also helps with patching and auditing security so seems odd that these supermarket ones were out of date.
Supermarket freezers are a bit more complicated than that. They can run at different temperatures and can consume a lot of power, so are part of a bigger store HVAC and electrical system as they may load-balance. Stores do not have qualified engineers on site to do anything with refrigeration equipment and it is better to have minimum wage offsite monitoring than expect a minimum wage shelf packer who only gets one toilet break a day to care about the fridges.
What I read from the last wave is that supermarket changes how cold freezers will run depending on how much customers they expect that will open and close the doors. Just having the freezers adapt based on the temperature of every single moment cost too much electricity or can result in unsafe temperatures.
I do work in the retail refrigeration world nowadays and "just updating the software" is much easier said than done. I'm usually touching grocery stores with 20+ year old hardware from the dawn of centralized control and monitoring. It's not s trivial thing to rip out the controllers and install new ones.
They deserve to get burned and should get a hefty fine as well. I'm fed up with businesses completely ignoring all basic security and then blaming culprits for hacking them.
I don't care whether it's the Russians, Chinese, Norks or script kiddies. This should never have happened, especially since a similar incident happened in France three weeks ago so they got ample time to prepare themselves. They just don't care.
Imagine they where internet exposed due to simplicity. It will take longer than three weeks to set this up. Not for you but imagine you are a store owner with 0 IT knowledge.
The issue is this needs to be handled centrally by the parent org also. They have IT ofc but they usually do POS and desktops/servers. The OT/SCADA systems are usually totally separate and not touched by the white collar IT systems.
I worked in IT for a major retail chain in the US. I was on the team in charge of store systems. We knew what devices were connected to the network, and if we had a vulberable device like this, we would have sent an urgent message to store managers to disconnect themm immediatly, and we would have followed up with scans over several days to ensure it happened. There is no excuse. (If we really needed the devices connected, we would have put them on a firewalled VLAN, which we could have pushed out in hours and had across the nation within days at most.). Granted, ideally we would have already had them VLANed.
In the case of France, the vulnerability was announced in August, 2025.
I believe gigantic fines are appropriate. They need to be of a size where the entire management team has significant financial issues for many years to come, even after having sold their houses, big cars, etc.
Nothing will improve until there are personal consequences.
I hear you're passionate about this, but why a fine? They were the victims, in our society the victim of a crime doesn't need to be penalized further, moreover, the only damage was to themselves.
>This first happened in France over three weeks ago. It's unforgivable that many chains have failed to update their software in time.
Came into the thread thinking it would be a new report about the same incident not a new one, yikes.
To be fair, wouldn't the chains probably be reliant on their vendors for an update? and turning off all the fridges for weeks is probably not a sustainable solution... :/
Their video is quite interesting. If that way of using a computer actually becomes mainstream, it would mean every tool or service just needs a UI and an API for the user's AI. The current trend of bolting AI features onto every app is starting to feel very unnecessary.
Nice to hear others are feeling this too. I started using typing websites to improve my speed a few months ago. More recently I've noticed that I'm starting to make lots of typos and use sloppy language in general because AI still understands me. That can't be healthy in the long term.
Little late but would this have actually allowed running early Linux under Windows when Windows 95 came out in the 90s? I remember only dual booting being available at that time.
I’ve been downvoted enough with my comments on this blog post where I’m hesitant to add anything else, but here I agree with you. They’re trying to be everything to everyone, where does the accountability of their customers being responsible for running, you know, up-to-date packages come in? Like, you don’t take just a little bit of pride in your work that you’re continually watching CVE lists and exploits and just have a minimum of effort toward patching your own shit, rather than pawning it off on vendor? I simply can’t understand the mindset.
Core routers don't inspect that field, NAT/ISP boxes can. I believe that with two suitable dedicated linux servers it is very possible to send and receive single custom IP packet between them even using 253 or 254 (= Use for experimentation and testing [RFC3692]) as the protocol number
This is an interesting list; it makes you appreciate just how many obscure protocols have died out in practice. Evolution in networks seems to mimic evolution in nature quite well.
We're about half-way to exhausted, but a huge chunk of the ones assigned are long deprecated and/or proprietary technologies and could conceivably be reassigned. Assignment now is obviously a lot more conservative than it was in the 1980s.
There is sometimes drama with it, though. Awhile back, the OpenBSD guys created CARP as a fully open source router failover protocol, but couldn't get an official IP number and ended up using the same one as VRRP. There's also a lot of historical animosity that some companies got numbers for proprietary protocols (eg Cisco got one for its then-proprietary EIGRP).
Probably use of some type of options. Up to 320 bits, so I think there is reasonable amount of space there for good while. Ofc, this makes really messy processing, but with current hardware not impossible.
It uses them a little differently -- in IPv4, there is one protocol per packet, while in IPv6, "protocols" can be chained in a mechanism called extension headers -- but this actually makes the problem of number exhaustion more acute.
What if extension headers made it better? We could come up with a protocol consisting solely of a larger Next Header field and chain this pseudo header with the actual payload whenever the protocol number is > 255. The same idea could also be used in IPv4.
I didn't mean to imply otherwise. But, as you say, this is equally applicable to IPv4 and IPv6. There were a lot of issues solved by IPv6, but "have even more room for non-TCP/UDP transports" wasn't one of them (and didn't need to be, tbqh).
I now read some interviews with local chains in Sweden and Finland, and they were still just investigating what to do at this point.
reply