The running gag of mobile internet in Germany is EU-roaming (no-charge use of you cellular internet service abroad in the EU): I can travel the 1-4 hours to any neighboring countries for the perfect cellular internet access I paid for. The only country where it doesn’t perfectly work is the one I live in. Germany.
Yeah the EU roaming thing is a godsend. The UK contracts in particular are great cause you get ones that extend data to a handful of other interesting countries beyond EU for another couple bucks
Have you had a look at rclone? Pretty sure you can copy or even sync files from one remote storage to another. E.g. copy from S3 to B2.
https://rclone.org/commands/rclone_sync/
+1 for rclone here. It can indeed copy between remote backends. Just keep in mind that that data all has to flow through the rclone process. You could probably get much better performance by running rclone itself on an ec2 instance. Just keep an eye on your throughput usage.
I like https://www.urbackup.org for Windows boxes. Does incremental file and image backups.
Next system for the Linux world will most likely be https://restic.net with a rclone backend that saves to B2.
Idea is to have an append-only service that is safe from ransomware deleting backups for extortion purposes.
One data point: LTE gives me 65-90ms ping RTT in reasonable non-crowded indoor conditions. Subjectively, working interactively via LTE is totally fine for me. Compare that to UMTS (3G T-Mobile DE) where I think RTT was more like 350ms and working interactively is a pain.
Came to Ctrl-F for Skynet. Was not disappointed. This seems to be a classic situation of reasonable strategic action with problematic potential of stepping over the fine line to automatic actions based on AI-derived intelligence. Maybe not now, but think a decade into the future where enough time and space has passed between the initial planning/implementation personnel and those advancing what "has always been there". We can only hope the others love their children too.
Yep, Miele. Washing ~5 loads/week an a 27 year old machine. Replaced main water valve ~5 years ago (20€ part). Resoldered one relais PCB ~10 years ago (instructions on the internets).
Interesting! How did you go about falling back to an older version if the update was bad? Is there a nice way to do this automatically? Say I update to a really botched version with the kernel panicing before it reaches userland. Does this need manual intervention?
It is possible to make rollback fully automatic. In order to do so you need some integration with bootloader. It needs to be configured so that it can roll back to the previously working partition if update is broken. What is more, you can add some user space runtime checks that can verify the update and if those are not passing (updated image is broken) you can rollback to the previous one as well.
As someone that also implemented A/B booting for the Pi: I wonder how you roll back fully automated? I read a bit of the code but wasn't able to find that. Or is that already handled by u-boot?
In my case, the first thing I do once an unverified version boots is to switch back to the other partition (so the known good version is active during the next boot), then run a detached reboot process that forces a reboot in 5 minutes. Once the system is up and it verified that everything is ok, it commits the next version (by switching back to the partition that booted and marking it as confirmed) so it is now active by default. Finally it kills the still running 'reboot' process.
As far as I understand your update process: You download a complete new version for every update and are able to stream that directly to the new partition? Is there any way to do delta updates? In my experience, most of the disk content is unchanged, unless you do major updates. In my case I download the new version using zsync, verify the downloaded/updated `install.zip` (which is kept on the volatile data partition), then extract that to the new partition. I make sure that `install.zip` is created in a way that it is rsyncable, so updates are pretty small that way. Of course you lose the streaming feature, unless you modify zsync somehow to support that.
I work on Mender, so I can tell you how automated rollback works there.
The update is written to the inactive rootfs partition, uboot is configured to boot from it and the device is rebooted. Using the bootcount feature of uboot it is possible to roll back automatically if booting fails. Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll back. Only after successfully reporting the success to the server Mender will "commit" the update, meaning configuring uboot to persistently boot from this updated partition.
Mender already does compression, but you are right that there are optimizations that can be made for application updates, e.g. delta or other types of updates. We are planning to implement this as well. The first priority for Mender is to make it robust, i.e. make sure the update is atomic and that you can always roll back.
> Using the bootcount feature of uboot it is possible to roll back automatically if booting fails.
I see. Thanks for the info. I suspected that u-boot does have support for that, but I wasn't sure.
> Once the mender daemon comes up it will try to report the success of the deployment to the server. If this fails it will also roll back.
Is there any deadline at all for that? I explicitly spawn a reboot command that ensures that even if everything gets stuck (in software, not in hardware) for whatever reason, the system falls back to the previous version (unless the reboot command gets killed too, in which case a manual restart is required). Any thoughts on that?
This is a valid point. If booting just hangs after the bootloader but before the Mender daemon comes up is actually quite tricky to manage.
We have looked into hardware watchdog for this, but it is in the gray-zone of what an updater should be involved in. This is actually a more generic problem - maybe it hangs even when you did not deploy an update. There is varying support for hardware watchdogs across boards as well, unfortunately.
Most of the time it will not just hang, maybe it will crash or kernel panic and in those cases Mender will rollback. But the indefinite-hanging case is quite tricky and not yet handled.
Hey! Love the product, but I'm out of the embedded game. Thought I'd give my $0.02:
The first step of our boot process was to enable the watchdog. We extend the timeout periodically during the boot process, but generally if userspace isn't reached within 30 seconds or so we reset. Once in userspace, the daemon validates that things look good (this includes things beyond just application of the update -- did services start up correctly? Is the hardware operating as we expect?) before disabling the watchdog and marking the update as a success, at which point rollback isn't possible. At this point we might consider applying new updates, etc.
We also modified our first stage bootloader to be resilient to bootloader update issues, and chainloaded our second stage bootloader from a stub which could rollback.
We also niced the update process to avoid resource contention, allowed the updates to be delayed until the network was quiet, and paused them when it became noisy to make for a good user experience. There was a server-side flag to force updates to apply regardless, with higher priority, as well as one to basically disable all other functionality in the case of a unforeseen serious, perhaps security related, issue.
We actually had a discrete watchdog service which was responsible for petting an always-on watchdog, to rescue the system if it locked up or became unresponsive (if certain processes were not running, or responding, the watchdog would not be pet).
All of this led to effectively 0 failures in the field, a seamless user experience (except for the 30-second reboot when inactive). I wish everything I owned worked this way.
I could talk ad nauseum about this stuff. It's very cool to see the designs of others. I feel this is an under appreciated and under explored problem space.
(Way) back when I was working with OpenEmbedded/Ångström I dreamt up sth like this: make uboot/whatever set up a HW watchdog that is retriggered once from kernel mode and then in the userspace as ususal. The daunting task would've been to implement the (platform specific) watchdog in the bootloader. Looking now, there is support for at least some platforms - nice!
Good luck with Mender - I'm sure you are badly needed!
FreeMind on Mac and Windows. Open-source software. Topics (SW-dev, Ops, Processes) nicely form a foldable tree. Storage is cleartext readable XML. Nodes take formatted richtext if needed. Syncs nicely via Owncloud or Dropbox or git. Has been with me for 10+ years. Downsides: Jumping through Java hoops on MacOS. Copy&Paste prone to non-ASCII punctuation/whitespace. Not really maintained and hosted on Sourceforge.
Remembering the start of it and seeing the US pressuring enough to alter Galileo to be US-blockable[1] makes me feel old.
Really excited though to see it finally getting usable.
