If you exclude the deep work on models themselves and just focus on use — harnesses etc. — then the intellectual community is basically prescientific. Very little meaningful empiricism. danluu's notes on benchmarks and experimental design come to mind: https://danluu.com/exercise-7/
This response confuses me, as I’m pretty sure OP is talking about AI use (i.e. applied use, from solving math problems to vibe coding to etc etc) not something like harness design or benchmarking? I’m also not sure what empiricism has to do with it, are non-scientists not intellectuals in your eyes?
Nope. There's a lot of unscientific intellectualism out there. I don't see a lot of intellectual depth in the communities springing up around AI use — I had vibecoding in particular in mind, and Fernando's blog post directly addresses mathematics.
I don't understand the contrast you're drawing. Harness design and benchmarking are major aspects of vibe coding (and solving math problems with LLMs, for that matter); I would go so far as to say harness design and benchmarking are the more intellectually serious aspects of vibecoding. Fernando is definitely aware of them as well. These just aren't meaningful substitutes for the intellectual work being diminished.
SQLite/Turso probably offers a much better developer experience than Postgres for the overwhelming majority of projects on Supabase (especially for local testing and testing in ephemeral environments).
Didn't mean to dismiss it. I think it's worth reasoning about! e.g. the CI example — in a build cache shared between ephemeral instances, cache poisoning can smuggle malicious behavior from untrusted environments to trusted ones. You're right.
Didn’t necessarily think you were saying that, but everyone’s looking for an easy excuse with security problems, so best we can do is hit them with the ol' “it depends" :)
One thing to consider — one of the reasons I want our Actions CI to be fast is that I do a good deal of agentic coding work in parallel (worktrees) on very, very small VMs, but we have some computationally demanding tests and linters. Delegating the expensive validation work to a separate environment — the beefier Actions runner, with run-isolation — and then watching the check status actually tightens the feedback loop for these agents.
Like with all things CI, your mileage will vary according to where you write your code and what the code does.
> - Allowing actions/setup-go users to specify a cache key prefix so that they can have more than one golang CI job, each with its own cache [...]
I'd actually go further: this may be a sensible default behavior.
"Always update the cache" can get expensive, but it's a neat one; "trim the cache" is definitely necessary if you enable this in a moderately active repository in our experience.
If you want really out-there ideas: rather than storing and loading the full cache monolithically, you could use a GitHub-specific GOCACHEPROG and Go-specific cache service to load only the active items. The pruning problem goes away because accretion is cheap. In theory, parallel jobs could actually share this joint cache. (This may not be a realistic initiative at GitHub.)
If you can raise feedback with your colleagues —
- The docs and settings for Actions Cache limits are really hard to navigate; at some pointed we desperately wanted to pay GitHub more money for more cache, but couldn't figure out why we were capped.
- Bulk-data endpoints for Actions performance would be a boon for optimization projects like this. I wind up either scraping `gh run` (slow) or setting up a GitHub App to collect perf data through webhooks (initially tedious, has to be continuously available).
All this aside — actions/setup-go is a pretty well-considered default and an essential part of writing Go on GitHub; ty for your work maintaining it!
They target different parts of actions/setup-go for optimization:
- WillAbides/setup-go-faster speeds up the Go toolchain setup (literally installing Go)
- cloudx-io/setup-go uses the slower actions/setup-go toolchain setup, but changes cache strategy so your `go test` and `go build` steps do less work
Those strategies are compatible. I hadn't heard of setup-go-faster — thanks for putting me on to it.
If you're deciding between one or the other, it'll probably come down to which inefficiency predominates in your codebase (i.e. how many tests you have, how quickly they run, and how much real churn there is in your test package build graph).
There are some good off-cuts that didn't make the official post, but which might be of interest to HN!
It only gets a brief mention, but the cache-pruning change was an interesting one. Cache accretion happens in the default actions/setup-go too, but dramatically increasing the number of cache-writes for cloudx-io/setup-go made it an actual issue.
As the cache grows, so does the time it takes to load it from GitHub's actions cache... and that grows until it's a significant time-suck in CI. We prune with basic mark-and-sweep.
Digging deeper, the pluggable `GOCACHEPROG` (introduced in Go 1.24) is a really useful tool. Shimming the normal cache logic for measurement, for example. In theory this should also be attractive for remote caching.
The way caches are managed in CI is bonkers (at least in GitLab, but GitHub probably does the same). I guess it's built to conveniently work with the simplest of projects. I had to reimplement the mechanism by hand just to get basic things working, like not downloading the entire cache when only a few entries will be used for each build
I also had fun substantiating the claim "86% of actions/setup-go test runs are unnecessary." We had a general sense things were faster, and we measured impacts immediately after we made changes, but hard to understand long-run performance vs. a counterfactual.
The trick was to run back over our git history and calculate, for each commit,
1. The test package Go cache keys at that point
2. The GitHub actions/cache keys constructed by actions/setup-go and cloudx-io/setup-go respectively
Once you have these mappings, you can
1. Pick some arbitrary HEAD commit
2. Model which prior GitHub cache blob would be loaded under each action
3. Compare the test package Go cache keys in that loaded blob against those for HEAD to determine which test packages would run vs. skip
Might write this up in greater depth sometime soon.
> I’m going to add a recommendation in our docs for pgtestdb, particularly for users aiming to test end-to-end (i.e. job inserted by client → fully completed by worker).
