There is nothing about passkeys that requires biometrics. Synced passkeys on android will work with your screen unlock code if you do not have biometrics enabled. Same with passkeys stored in a password manager. Same with hardware passkeys that do not support biometrics.
Object store is quickly becoming the new core data substrate. Lets build kafka, but on s3. Lets build github, but on s3. It feels like we going to see more and more "object-store first" systems in the next few years.
I am excited about this future. Give me stateless servers and a storage bucket over having to manage systems with disks any day.
I do wonder if we will see an expansion of the s3 api to support more of these use cases. S3 added a janky file append operation to their new express-one-zone bucket type, and limited to 10k total file append operations. I wonder what else we will get in the next few years.
Definitely agree that every data system that doesn't need <100ms latency is moving to object storage.
> I do wonder if we will see an expansion of the s3 api to support more of these use cases
This is actually an area where I think we have a big leg up on folks building on top of S3. My team (which built K2) sits next to the R2 team, and we have the opportunity to co-evolve the products in mutually beneficial ways.
> every data system that doesn't need <100ms latency is moving to object storage
I think the opportunity extends below 100ms too, particularly given the existence of faster object storage tiers like S3 express or more recently GCS rapid bucket (both only offering single-zone durability, so still need to do quorum writes to get region-level durability as with standard tiers).
One of the tensions of course is how long to linger before flushing to object storage - you have to trade off directly between latency and cost of your API ops for PUTs.
When building the serverless offering of s2.dev (which is in a similar space, full disclosure!), we designed around stateful backend processes capable of constantly flushing multi-tenant objects (i.e., containing records from many streams), allowing streams to offer low ack latencies (~50ms p99 from same region) without blowing up the unit economics.
From my experience benchmarking S3 express, it is is faster, but not fast enough yet for many use cases.
An obvious disclaimer is that the word "enough" here is carrying quite the weight: I expect it to get better, and each has their own requirements. Do benchmark yourself and don't make expensive decision based on an HN comment.
The range of things you can do with blob storage and a (very simple) auth model are surprisingly broad.
We recently replaced our Docker container registry with S3 using a tiny tool [1] we built in-house. I think that even with current capabilities, we can still model a lot services as a very thin layer over object storage.
From the opposite side of things, I feel the same about OPFS. Finally having something that performant that multiple workers can operate against in a browser is such a massive boon. You could plug something like that in to it locally with markedly less bullshit than you would have had to do previously with the other APIs.
That sounds really cool — I tend to agree with you that S3 and similar are underutilized, but I remember that essentially all of the providers charge for bandwidth measured in gigabytes and I'm like no. My consumer line is measured in megabits per second and if I have to pay for my data usage the way it's paid for in data centers it would be far more expensive. Somehow consumer ISPs, who have to pay for the lines, are cheaper than than the cloud providers.
It's because for providers the bottleneck isn't capacity (which is dirt cheap) but IOPS and bandwidth. To keep transfers fast they need to spread data across a lot of physical disks, meaning they often times sit mostly empty. Egress fees are a way to monetize that empty storage and make sure there is an incentive to minimize egress traffic.
We use R2 (not affiliated) which doesn't have egress fees.
"I remember that essentially all of the providers charge for bandwidth measured in gigabytes" - sure, but the cost per GB is shockingly cheap. If you're doing things well, you can do a lot inside those pricing structures.
Alternatively you can stand up your own object store services, but that's not something I would like to do.
Huh? We're talking about the same S3 right? At list pricing, 1TB is $23/TB to store for one month, and about $90/TB (plus request fees) to send it out to the internet.
While hard drive prices are roughly 3x what they were a year ago, the price per TB of a new hard disk averages around $30/TB -- assume 2x overhead for other hardware and extra space for parity etc, a disk pays for itself in less than 3 months and lasts 5 years or more.
If you assume it takes 1 month to download that 1TB (about 3Mb/s) that's $29.16/Mbps. When I first started buying internet transit in Europe back in 2008 I think I was paying under $10/month. It's now under $0.10/Mbps pretty much anywhere in the US or Europe at the big datacenters.
None of the "big" object storage services are cheap. They're somewhat reasonable if you only access the data from within the same region, but absolutely insane if you ever want to ship that data outside of that cloud vendor (or to another region etc). The pricing of storage and egress has not changed in a decade (I believe AWS last lowered the price of either in 2016) and in fact it costs even more now due to things like NAT Gateways etc.
It's definitely not "shockingly cheap". It's just cheaper than $80/TB of gp3 or $45/TB of st1, and while sc1 is $15/TB it has a baseline performance of only 12 MB/s. There's quite a few companies out there that have $6/TB/month object storage plans with similar performance, rising to about $15-$18/TB/month for SSD backed storage with far lower latency figures.
AWS have found the ideal captive market: computer engineers who don't know how computers work or how much they should cost.
Whatever AWS is selling you - except Deep Archive - I'll figure out a way to sell you for half that price, if you want, and it'll still be 80% profit for me. Your only downside will be that I don't know what I'm doing so it might not be reliable - but neither is AWS.
Not just AWS. IIRC, GCS and ABS are slightly cheaper for storage but more for egress. To be fair to them, their reliability is pretty much second-to-none, whereas many of those at the very cheap end pretty much just ran a Ceph/RADOS cluster and then wondered why they were having lots of problems with reliability -- but with things like RustFS becoming more and more popular, the object storage racket is long overdue a shake-up!
We run Ceph at work. Reliability problems have been exclusively caused by us penny-pinching and trying to have as few drives and servers as possible, causing latency issues. If we had a nice margin instead, I can't see there would be any problems.
Not really. 1TB is 90 USD, so if you are doing anything with video that is gone pretty quickly. Same with software distribution - your 200mb download hits that with 5000 downloads. Convert a 10 usd trial at 1 percent and almost 20% is eaten in bandwidth charges.
At those prices, it only makes sense to use it for data that never leaves that providers cloud (which I assume, but cannot prove, is their intention).
> My consumer line is measured in megabits per second and if I have to pay for my data usage the way it's paid for in data centers it would be far more expensive
Well because your provider assumes you are not using all your bandwidth constantly. Cloud bandwidth is only billed for you actually use
It's still true that "clouds" cost much more than proper internet connections at DCs. E.g. AWS wants you to pay $90/TB, Hetzner $1.50/TB, a good deal on a contract is probably half what Hetzner pays since they need profit too.
And if you have two specific endpoints you need to transfer data between at a high rate, you can get stupidly cheap cost per GB on a leased line in exchange for making all that commitment upfront.
I'm a little surprised the Docker CLI doesn't support this directly, since gcr.io works this way. Maybe there's a thin layer between the bucket and the CLI? Neat that you worked it out for the generic case.
I was surprised too! The OCI layout for storing images is actually pretty simple. But for some weird reason you can't stream it straight into Docker. docker load only accepts tarballs.
So you need something to wrap the layout into something Docker understands. Because S3 is not a server, you have to construct the tarball on the fly as the image is pulled and stream it straight into docker load.
I worked on OCI for quite a while, the very short and cynical answer is that Docker thought the registry was their moat for a long time and fought tooth and nail to keep it at the detriment of almost everything else. (The longer version is a bit more diplomatic.)
Even better, the OCI distribution protocol (which was Docker distribution until a few years ago) is not a static-blob-over-HTTP protocol! The blob bits are and you can route them to blob storage but you need a smart server for a few key bits of the protocol.
Back in the day I made proposals for distribution formats that didn't have these flaws and were more distributed (and previous proposals like AppC's discovery had similar ideas) but they were roundly rejected by the Docker people.
It makes perfect sense: it's a extremely reliable, infinitely-scalable, strongly consistent (in some implementation), cheap, and extremely simple to use key-value store. As such, it transparently solves a lot of the problems distributed systems have to be engineered around.
As long as you can run on a CSP, and can engineer around the high-ish latency (most business cases can), it's extremely expensive to try engineer around it.
People like S3 because they have hard engineering guarantees around bit rot and work well as a high level abstraction of a network filesystem with all of the low level failure recovery built-in. You don't have to worry about doing your own RAID configs. The bigger question is whether non-AWS services can offer the same level of guarantees. I have heard horror stories for example when it comes to downtime on Hetzner's S3 object store.
I am currently using Cloudflare R2 right now and if you see their forums, there's always the occasional post about objects going missing.
If you there's no SLAs to engineer for, just set the target at zero, and drive your cost to zero as well. So you have to design for _some_ defined value of reliability.
Backblaze's famous reports put HD failure rate is ~1.39%, so for the 11 9s you get as a guarantee from S3. Assuming nothing else fails, you'd need at least 6 independent copies to get that, plus all the effort to engineer recovery, and constant upkeep.
Suddenly, S3, even when considering bandwidth costs, seems like a steal.
The usual SLA is not expressed numerically but as a feeling. And a normal server suffices to deliver that feeling. I'd bet 99% of apps are used by under a hundred people and if they have to take a day off due to a head crash, it's annoying but not catastrophic.
If you have two sets of hardware the server can run on (cold standby), and RAID, and are competent at physical IT work, you can have faulty hardware replaced in an hour. Drive fails - replace it. Anything else fails - swap the drives to the other machine, boot it up and then troubleshoot the original.
Most likely you don't even need that. If the app server runs on a standard platform like Windows you can shuffle it over to some spare tower PC. You hear horror stories of dusty towers that nobody knows what they're doing - the horror there isn't from running server software on a tower, but from unmaintained servers no matter the form factor
I find that Ceph is pretty annoying to operate but if you do, it works fine. Keeps redundant copies of data on multiple disks, across multiple racks if your diversity is that wide. Supports erasure coding for effective redundancy less than 2x.
Don't go for the "object gateway" compatibility layer - just use raw Ceph if you're writing your own app.
On one hand the durability and scalability of object storage is amazing. But on the other hand it seems perverse to build so many systems around the very high latency of object storage when we have NVMe SSDs that have higher bandwidth than RAM did not long ago and microsecond latency.
Yup, and if someone is looking for a non-AI startup idea, take a use case with lots of data and a read-heavy workload, and do pretty much what Turbopuffer did for text and vector data. That seems like a very good way to go.
> Give me stateless servers and a storage bucket over having to manage systems with disks any day.
I'm working on something like this for my business, but the server is the bucket. When you shut it off, the VM data is stuffed into a bucket and restored when you want it back on.
Our use case is that we rent out VM's with GPUs in them (on-demand, no-reservation, billed by the minute), and the GPU rental is a lot more expensive than the disk rental. Right now, we don't have a cluster of storage in our DC, so we just delete the data.
We support API/cloud-init, so if they have regular workloads, they can just boot a VM with whatever they need.
It is then... get some storage (in progress) and let people pause their VM and pay less for disk when they don't need the GPU. Fully on-demand GPU compute. Perfect for agentic workloads where you want your agent to be able to spin up larger models on enterprise gear for when it needs it.
I hope to see it become more common to have a fast in-memory proxy layer with the same API in front of object storage, one that can accept writes and respond to metadata requests quickly.
So that object storage can become the default even for regular line of business applications.
You can download objects from S3 from EC2 without traversing the public internet using things like gateway endpoints [0] which avoids s3 egress fees. But doesn't avoid egress fees from EC2 to the end user.
I wish the author provided some latency numbers for this. One issue with tpms is that they are slow relative to performing the same operation on a modern CPU.
Thats the “what it costs” section? Im a bit impressed if they are down to ~3ms per handshake. When I last looked at TPM signing (many years ago) it was more like single digit transactions per second.
That said, even 3ms TPM signatures are going to be for special cases or novelty. Plain old CPU tls will do about 1ms cpu time per request which will scale by cpu core count. One or two orders of magnitude more throughput per host.
There are a bunch of these tools out there. I like curl-impersonate[0] as a quick way to make curl fingerprint as chrome or firefox. There's also CylceTLS[1] for a more programmatic approach.
Often I set up a bunch of automation for a particular service, and then they "upgrade" their security to enforce fingerprint checking, and adding in a proxy is just a config change to my existing tooling.
I got rid of my Alexa devices after they would not shut up. "Alexa what is the weather for today?" "The weather for today will be ... By the way, did you know that you can set alarms? Just say ..."
I do not care about whatever stupid feature you want to build engagement around. Do what I asked you to do and then shut up.
And don't get me started on the Alexa Show that had the audacity to display ads.
I wrote a library[0] to let you concurrently update a sqlite db in s3 safely. It uses the little known sqlite sessions extension plus s3 compare-and-swap on a small metadata file to make this work reasonably efficiently and safely. I have been enjoying it for a bunch of small projects where I want a lambda function to have a db for state but I don't want to pay for a full database instance.
This is exactly what I had dreamed of except without being aware of the session extension... I put it off because I thought I would have to create a layer opening sequential partial sqlite dbs and somehow filtering to the most recent version of each record in-between compactions.
It sounds like sqlite sessions handles the hardest part and you've sanded down some of the rough edges while implementing the glue bits for s3 (generously licensed MIT); thanks for the heads-up!
> If you're using AWS Lambda then you have to work to keep convincing yourself this is better than your own web servers. Keep convincing yourself that using AWS Lambda is not a horrible mistake.
lol ok. I have ~50 lambdas running in my personal aws account. Some of them are webservers running behind an api gateway or using a lambda function url to expose them to the internet. Some are running on a schedule, some are triggered from s3 events. The cost to run these for me is less than the cost of the cheapest vps (my total requests per month stay under the free tier limit). There is also zero maintenance I need to do for these functions (ok, this year I did have to find-replace al2 to al.2023 in my terraform config). I don't have to worry about making sure the os is patched for the latest vulnerabilities. And I don't have to worry about the specific hardware my code is running on at any time. Doing maintenance for old projects sucks. It is great to have servers I deployed years ago continue to chug along without me needing to think about it.
Now, all of my lambdas are written in Go and I suspect if I was using one of the manged runtime libraries I would find the language upgrades to be quite annoying. Go also helps quite a lot with cold start times.
Then again maybe I have just drank the koolaid. In my quest to use lambdas for as much as I can as cheaply as I can, I made a library[0] to use sqlite on top of s3 (not just readonly). It uses the sqlite session extension plus s3 compare-and-swap to allow you to write updates safely to s3, even if you have concurrent writers.
> The cost to run these for me is less than the cost of the cheapest vps (my total requests per month stay under the free tier limit).
I don't think this is a valid argument. Free-tier VPS do exist also.
On the other hand, if you don't trust unattended-upgrades [0], and prefer to spend time poking package manager manually (while at the same time considering that time an expense) - sure, that's a strong argument in favour of using lambda.
As you yourself said. Your load is so light you keep it in free tier. Their entire business model is for them to capture you while your load is light and then when you scale the price goes up.
I have also used lambda at scale in professional environments. I would not use a lambda for a webserver at scale, but having an s3 object trigger processing via a lambda function is a really nice flow.
This "it has to be handwritten" stuff is nonsense. Do that if you enjoy it, but also you should acknowledge the downsides to it.
I started keeping a work journal a few years ago and it has changed how I work for the better. It is just a text file.
The main value of it is that I can search it! When I'm figuring something out for the first time, and I have a lot of trail and error, I write down what I did. And then I might not touch that thing again for 6 months. When I come back to it, it is unlikely that I will remember what I did exactly but because it is written down and searchable I can quickly recover my old state.
I like this so much I also started a personal work journal for my home lab. It really is useful for me. But its primary value is that I can search it.
> This "it has to be handwritten" stuff is nonsense. Do that if you enjoy it, but also you should acknowledge the downsides to it.
From the submission:
"Should you use one?
Maybe! I can't answer that for you. "
There is a middle ground. Write on paper on the spot. Transcribe to digital form later. I've been doing it for many years. Painful, but now I just get an LLM to OCR my handwriting.
And oh, definitely - for me the brain works differently while typing vs handwriting.
I find there's an advantage to writing if I'm trying to memorize something. But if that isn't going to happen because I only did this configuration once and then never needed to reference it again for two years and now I need the exact commands I executed, can't beat a searchable txt file.
Writing things by hand leads to better retention, but if you can't remember it...yeah you'll have a fun time finding it again if you have a nontrivial amount of notes and haven't spent a significant amount of time indexing them.
Yeah I was talking about Google Vision or similar. Though USPS has been doing handwriting recognition since the 80's. I don't it got very good until the late 90's though.
I stopped using my Remarkable because I couldn’t read my own writing on it. I have have no problem reading my writing when I write in a physical paper notebook (well, fewer problems). But the resolution either for capture or display on the remarkable makes my somewhat messy writing impossible to read.
So, it’s not only if you can’t remember it… you also have to be able to read it!
For me the advantage of using digital journalling is it's easier to switch between multiple threads. It means if I have 3 tasks to complete, I can switch tabs to the notes on the new task without thinking about the physical limitations of a notebook.
I handwrite notes because it commits it to memory and I don't have to search it later (most of the time). Important stuff goes in electronic notes, but 90% of things aren't that important.
Most of what I write in my journal I don't want to need to remember (that is why I am writing it down). I also don't know what will be important or not important later, so I want it all to be searchable.
It's not nonsense. At least it hasn't been for me. My memory retention shoots into the stratosphere when I write stuff (that I need to remember) down. Something about it being slower and actual effort needing to be expended to commit the information.
Handwriting is my primary notetaking route, but in situations where it isn't (typing is faster after all), I'll scribe what I typed into my reMarkable later. If the note I created was for work and I wrote a note by hand first, I'll type it into a Google Doc later and upload both versions (since they might differ slightly). I learned this as a study hack while I was in school 15+ years ago and it still works flawlessly for retention.
Handwriting everything has held me back sometimes, though. Ironically, insisting on handwriting everything is the reason why I didn't keep a daily journal for a long time. I started journaling in 2012 and stopped as work and life got busier. I picked it back up in 2021 with a mood tracking app to, well, track my mood and reactions to emotions and fairly quickly regretted not having just typed my journal entries.
it sounds like the author is using an e-ink device for note-taking. i use an ipad app for a lot of hand-written notes and i'm consistently surprised at how well it can search my chicken-scratch.
my biggest issue with handwriting is it just takes so long that i end up leaving out important details. it's a shame because i do enjoy it.
reply