You wrote about deplatforming-- heads up that DigitalOcean does deplatform people. I have direct firsthand knowledge of DigitalOcean deplatforming a security disclosure professional reporting a root breach bug.
For your secure backups, I can suggest Tarsnap by Colin Percival, who's a security expert and a frequent contributor on Hacker News. I don't get anything for recommending Tarsnap; I'm just a customer. https://www.tarsnap.com/
I had a terrible, terrifying experience with DO where they hard locked my account for days (and it would have been weeks if I hadn't made a huge stink) because of a misunderstanding. They ban/lock first, ask questions later, and that's not cool and I can't have my (and my customer's) important infrastructure treated that way.
I now use Linode for most of my stuff, and once they have cloud firewalls available in Dallas I'll be moving the rest of infrastructure over as I can.
I really need to write up my story into a blog post I can link to, but until then, I put a recap here a few days ago for anyone interested in the details. Mine wasn't as bad as some people's, but pretty scary personally: https://news.ycombinator.com/item?id=25806086
I’ve had only positive experiences with Linode, including incidents where I faced absolutely massive DDoS attacks.
On the other hand, my account is suspended from DigitalOcean, I have multiple droplets they refuse to give the data to, and they even had the audacity to keep billing me until I did credit card chargebacks. I still have no idea why I was banned.
Maybe because I used the GitHub student pack promo but I had already finished uni? I don’t know.
Another happy Linode user here. They might not have the nicest interface, but their service is top notch and I much prefer their UI to Digital Ocean's any day of the week. I've been with Linode for ten years now.
I guess I’m not terribly surprised. It fits a narrative, not of being overly protective, but of run-of-the-mill incompetence.
Someone once used a DigitalOcean droplet to run an aggressive layer 7 denial of service attack against a service I host. I notified DO via their web form, didn’t get a reply for three days, then got told to email to their abuse@ email address instead.
Dutifully, I mailed abuse@ and included the relevant server logs, gzipped, as an attachment. I got a reply that they won’t open any attachments, please copy and paste the logs directly into the email.
As an outsider, it seems to me as if they have few people with the appropriate technical background staffing their abuse department, so legitimate reports are left uninvestigated due to dumb technicalities like not being willing/able to open attachments, and bogus reports are accepted without thought or understanding as to the appropriate course of action.
They're definitely on the theater side of security. I've got locked out of my account multiple times for no reason.
Furious the second time, I sent a crafted image of an invalid ID to check if they actually even care. It seems they don't, as the account got unlocked all the same.
I think you should make a blog post with screenshots calling them out for this ridiculous hypocrisy.
Of its own the blog post won't do anything, but the next time they're being overeager and shutting down a legitimate customer, you can link to that and corner them to fix their procedures.
This is the most offensive part to me, I can't even sign up or use DO without having my privacy invaded and sold (they load multiple fingerprinters from different third party vendors that also then share this data) yet they still don't have a handle on abuse.
I get so much rubbish from DO space, nonstop port scanning at absurd volumes (sometimes totaling 6 digit+ pps), from customers that haven't been removed in years. massscan/zmap from people with a fake opt out page (that I shouldn't need to be opting out of), etc.
They flagged my account and pulled 100% of my servers with no mitigation for 12 hours until I convinced them I was legitimate. DO still handles that incredibly poorly, which is the reason we no longer host anything critical there.
Also both tarsnap and S3 have no idea what content they’re storing on your behalf. (Everything is encrypted at rest). So for tarsnap, deplatforming should never be an issue.
I used to use Tarsnap and am still stuck with it in several projects. I wish I weren't. Restoring backups takes ages. If you need to restore a production server in an emergency, and the whole server remains down for an hour because that's how long Tarsnap takes to restore some tens of Gigabytes, then you will very quickly look for alternatives. I raised this issue with Colin, Tarsnap's creator, multiple times. And the answer has always been "maybe some day". I now simply encrypt backups myself and store them in an S3 or Linode bucket. Much faster. For personal backups, I love restic syncing to Backblaze, where I back up 1.4 TB or so for $5 per month.
$0.25 /GB is pretty expensive. I'm using Hetzner Storage Box [1] which has tonnes of connectivity options where 1TB is only 9.40€ (works out at 0.0094€ /GB)
Yeah I've been with them for 8 years as they're the best value hosting provider [1] I've found, which I use for all my memory or computational heavy workloads as they come at a fraction of the cost of what it would cost on AWS.
I would actually compare DigitalOcean to a Facebook or a Google or an Instagram. You’re banned or blocked and cannot reach a human to find out why or what can be done. DO support is just a bunch of auto mailers sending the same template emails, in my experience.
As tarsnap is run by a single person, it seems hard to suggest it to serious businesses due to the inherent risks there as well as the extremely high storage costs and difficult cleanup procedures compared to competing tools. I tend to prefer solutions like restic + B2 for their price and support, where you at least have a larger company behind the storage backend, you could even directly use it with S3 and it'd still be cheaper than Tarsnap.
Oh my word, that article on tarsnap is a frightful bore. I was a tarsnap customer and I thought it was great. If Colin is happy running it more like a utility than a rapacious VC-backed hyper-growth racket, I'm not sure why that is bad.
The only reason I stopped being a customer was because another rapacious racket of a business, Comcast, introduced miniscule but enforced data caps in my area, so online backups aren't attractive anymore and I've gone back to external drives and offsite rotation. When I cancelled, Colin sent me a personal e-mail to make sure it was alright to delete my backups. It was probably the best exchange I've ever had with a service provider.
> If Colin is happy running it more like a utility than a rapacious VC-backed hyper-growth racket, I'm not sure why that is bad.
I don't doubt your happiness, but I confess that I'm having trouble reconciling "running it more like a utility" with charging 25¢/GB per month for storage. That is just staggeringly high. What I'm paying $6/month for with Arq would be over $160/month if I were using Tarsnap, and I'm getting end-to-end encryption, deduplication, and versioned file backups. What advantages does Tarsnap bring to the table that justify such a tremendous cost?
The whole point of a backup service is that most of the time you don't need it at all, but when you do need it you really need it. From that perspective, it's like paying for insurance.
I'm not a tarsnap customer, but I think what you're paying for is a service built by a literal obsessive genius that will 100% work when the chips are down.
Unfortunately restic was a no go for me due to not being compatible with B2 keys that only have the permissions readFiles,writeFiles,listBuckets,listFiles (no deleteFiles). I don't want the attacker to be able to delete any backups if the manage to get to the B2 keys.
In short, give rclone your keys, (small ~/.rclone config file) tell restic to use an rclone backend and add an extra argument when running restic. (See the blog link above, all explained)
There's an open issue for this which doesn't seem to have moved: https://github.com/restic/restic/issues/2134 - it does have a couple of PRs linked on there which implement proper soft-delete.
Yes. Having family help out on a 1-man show is an entirely different beast than hiring a full-time employee with wages, taxes, and healthcare. One of these two is a life-long trusted human that you've shared numerous life experiences with, the other is not.
Looks like the brother just does the merges? (which are mostly automatic) so IMO best it matters because it's still mostly a one man, or at least a one family operation - i.e not very resilient to random acts of nature (which backups are supposed to protect from)
When a business becomes serious? Do you need to hire several workers to look legitimate? If that's still not enough, do you need to make yourself a slave of VC?
Only problem I can see is that one man operation is susceptible to the run over by a bus scenario.
However that is still better than a woke VC asking the "owner" to cancel some users.
I would seriously prefer a service like this to be run by a two man team. Or at least have a guarantee that I'll get automatically notified if something happens to a sole operator.
My issue with tarsnap vs Dropbox is entirely workflow. Right now I can treat my Dropbox effectively as an external drive that happens to live on my file system by making everything selectively synced. If I want something backed up, I just drag it into that folder. But once it’s backed up I can turn on selective sync and free up my disk space.
Backup of personal data is often a 1-way endeavor — here’s a dump of photos I scanned that I don’t need to look at anytime soon. But with tarsnap I can’t do that, nor can I then have the ease of browsing to the file and just opening it while it transparently downloads on demand.
Colin Percival is obviously a smart and very skilled guy but I am mystified why people on HN keep recommending tarsnap, it seems a terrible product for almost any imaginable audience.
If I'm a normal end user, I will probably get a vastly easier to use product at about 1-10% of the price from Backblaze. And if I'm a serious business that can easily afford the > 10x premium and engineering to configure the backup I probably wouldn't want to entrust it to some company with what looks to be a bus factor of one and the apparent technical limitation that a restore might cost me a few DAYS of unanticipated downtime [1].
I mean, sure, if you are some unix nerd wanting to backup your dotfiles and a few small documents (or repos) for maybe around ~$50/year, why not go with some artisanal backup service for HN street cred (and a laudable open source donation policy)? But what other good use cases are there?
E.g. relative to the author's script, tarsnap get you automatic deduplication across backups... which, if you're like me and basically generate more and more data, allows you to store lots of historic backups at basically no overhead over just storing today's data.
Tarsnap does have real downsides - restores can indeed be slow, bare per-GB cost is high, and "like tar" is not a user interface that everyone will like - but there are definitely upsides, too.
That's why so far best idea is to go bare metal with couple of different providers and sync data to multiple servers. In case one provider decides to cancel you, you can still have your data in another place. You can also have a copy at your own server. To have access to all machines as if they were on a local network, you can use tinc-vpn or for convenience ZeroTier.
what's the advantage of tarsnap over say, doing an encrypted backup with borg to my gdrive folder? I've never heard of Google random deleting encrypted blobs from people's storage and it's way cheaper
For your secure backups, I can suggest Tarsnap by Colin Percival, who's a security expert and a frequent contributor on Hacker News. I don't get anything for recommending Tarsnap; I'm just a customer. https://www.tarsnap.com/