I have so many use cases for this. I just hope that we'll get a standard like https://www.openresponses.org/, because the idea of coupling an app to a particular vendor doesn't seem appealing at all, especially since it seems that the functionalities they offer are more or less the same. (I'm thinking about the OpenAI's Agent API and Anthropic's Agent SDK as "alternatives.)
It's open source, so yes, though there is no "easy setup" like a Docker image or a Cloudflare template yet.
The artifacts are only kept in your local indexeddb though, so they don't get uploaded. Or well, unless you start a collaboration session, but even then they are encrypted before uploading, and the key stays in your browser (in the url hash), so I can't see any data on my end.
But I definitely understand the concern. Would you prefer it in a Docker image? Or maybe wrapped in an Electron app?
Thanks! I did indeed write it with AI¹. It actually evolved from the approach I was using beforehand: asking every time my coding agent to break down the PR into steps and build an interface for me to go through them (because GitHub PR UI is effectively unusable for me). I got tired of burning tokens building the ~same UI over and over, so I thought "why not define a format for the review and build a decent UI just once?".
¹ Though actually the "harness of guidelines" I've been using recently make Claude not produce _complete_ garbage.
I can see how for a local this system might be better, but as a tourist I strongly prefer the tap in/tap out system, especially if I can just use my credit card.
I used the SBB app a few times, and it's actually very good, but it's still an "app experience". First, I need to set it up (download, login, add payment method), for which I need an internet connection (which I might not have as soon as I enter the country). Then, every time I need to purchase a ticket I need to open the app and figure out how to do what I need to.
Tapping in and out is superior in my view, even though I might end up overpaying a bit (though I'm probably still paying much less than what I'd pay in Switzerland :D ).
Ok, even as a tourist I prefer not to have to fiddle with my card on every ride. I can't comprehend how people can prefer the tap in/tap off. Seems like another unnecessary inconvenience in life to me.
Additionally, SBB offers many different ways of purchasing tickets in the app(week ticket, 24 hour pass, etc, etc).
Author here. I definitely agree that communicating well is a prerequisite to getting decent results. On the other hand:
1. Even if you communicate perfectly, there's no guarantee that the LLM will "behave as instructed" and as you imagined it to. Indeed, the frustration often comes from the fact that you've said something as clear as day, yet the agent takes another path.
2. Part of the value of coding agents is exactly that you don't need to lay it all out perfectly for them. I mean, if I need to give the LLM every little implementation detail, I might as well write the code. Of course, I don't expect it to work off of "I want nice app make money", but I do expect some "intelligence" in figuring out the missing pieces.
> Even if you communicate perfectly, there's no guarantee that the LLM will "behave as instructed" and as you imagined it to. Indeed, the frustration often comes from the fact that you've said something as clear as day, yet the agent takes another path.
People forget. People misunderstand clear things. Teach yourself to not judge people for being human. You'll have easier time with AI. You are not gonna be angry at 5 year old because it occasionally can't follow your instructions. AI is a 5 year old that accidentally ate all the encyclopedias in the world and is super eager to help. Be a more charitable, generous, understanding person, even in the absence of actual people.
Also try a stronger model. There is a difference. I have very good results with Codex but don't get fixated on any one, they are all "state of the art" or close but they are different and state of the art is moving ahead faster and faster.
I don't extend the same grace to machines as I do to humans. This is working as intended. I have patience for people that mistake. But much less for machines. Why should I? These are things created by trillion dollar for-profit corporations. Extending them any benefit of any doubt is a vulnerability waiting to be exploited.
That's the issue. You frame your relationships with people using words like "grace" and "patience".
If you were "good natured", "kind hearted", "understanding" and "nurturing" instead you wouldn't have to "extend" it to "machines" because those are not limited resources you can run out of. Politeness towards AI would come naturally because of who you are. The only thing you could run out of is energy and that's easy to replenish with a bit of sleep and a meal.
And it would lead to better results. Not because there'd be some genuine psychological interaction between you and LLM. LLMs are not machines. They are just a finetuned math equation that tends to give better results when people act nice.
Also it would be better for your own well being because it would reduce your frustration. Both with LLMs and people.
You're making a lot of assertions about the nature of relationships and the definition of machines. I'll assume this is based on a philosophy that works for you. But now you have set up a system of definitions where, for example, a good natured person, by definition, cannot be frustrated. It kind of sounds like sophistry to me.
I genuinely don't understand your argument about machines. It's not a machine it's just a system of levers and pulleys.
I have developed some of my own systems for minimizing frustration, and they generally work reasonably well, despite being different from yours.
> You're making a lot of assertions about the nature of relationships and the definition of machines. I'll assume this is based on a philosophy that works for you
Always.
> But now you have set up a system of definitions where, for example, a good natured person, by definition, cannot be frustrated.
> It kind of sounds like sophistry to me.
It's just my lived experience. When I get frustrated I don't look for guilty and their flaws or seek retribution. I'm feeling something more along the lines "I'm not mad. I'm just disappointed." and tired that the world is not in the state I would like it to be.
Do you think Mr. Rogers would verbally abuse AI if it wasn't giving him answers he expected because he ran out of "patience" or "grace"?
> I genuinely don't understand your argument about machines. It's not a machine it's just a system of levers and pulleys.
Think for a moment about a "machine". List all the things that you expect of machines, that you think they ought to be, that you know they sometimes are. Think about their behaviors both intended and unintended. Think about would you evaluate what is a good and a bad machine. Now you have a rough picture of what a machine is for you.
Now try to imagine something else like a fast food worker, or a cat or a dining experience. Try to imagine that whatever you came up with is a machine. You expect them to have machine qualities, machine flaws and you expect of them what you would expect of a machine. Inspect how many mistakes would you make if you navigated the world with assumption that they belong to a "machine" class. How many frustrations and of what kinds would that bring.
Regardless of LLMs implementation details, based only on the results and frustrations, yours and reported by other people, can you recognize that thinking about them as of "machines" is a misclassification error?
Something like a ChatGPT app is more of a machine, but nearly its entire value comes from the main component that has almost nothing to do with the concept of a machine.
There is overlap in the things that humans and machines can do. But to me, humans enjoy a special position regardless of what they can do or are doing.
If I'm using a ball-point pen (a machine) that's leaking ink, I'm just going to throw it in the trash.
If I'm using a hand-made slingshot made as a gift, and one of the straps breaks, I will endeavor to fix it.
To me, the chief difference between a fast food worker and a machine is that one is a human. To me, humans deserve more respect than machines simply because they are humans. If a human successfully tricks me into thinking they are not a human, then I will also erroneously afford them less respect than I intended to.
What you call the biggest value of ChatGPT is what I'd call its biggest threat.
I value people for being people but it's completely orthogonal with anything else. I might at the same time value fast-food worker for being human with full kindheartedness I have for any stranger and find them completely useless for taking my order in comparison to the self-checkout kiosk. One doesn't influence the other and the other way around. I won't think worse of them as humans, because they don't fit my needs when it comes to taking order. And I won't think they are better at taking my order just because they are human.
> What you call the biggest value of ChatGPT is what I'd call its biggest threat.
I didn't mean LLM is a biggest value of ChatGPT system in terms of appreciation. I was just stating a fact that without LLM ChatGPT is just a lame chat app with some text filters that we pretty much had for the last 3 decades, so not very valuable. And that part of little value is the machine part of ChatGPT system. LLM itself is not a machine. It doesn't have machine flaws any more than a fast-food worker, or a dining experience or literal walk in a park could have machine flaws. LLMs are math. If you want to look for their flaws, you need to look for math flaws. You can be angry at them not in a way you are angry at machines (or people, or park walks, or dining experiences), only in a way you could be angry at math.
I can't really remember if I was ever angry at math.
> Teach yourself to not judge people for being human. You'll have easier time with AI.
LLM is not a human. This implication that OP or someone else is impatient against people when they get frustrated with effin machine is completely absurd.
> AI is a 5 year old that accidentally ate all the encyclopedias in the world and is super eager to help
LLM is not 5 years old kid. It is an expensive tool.
LLM literally is neither 5 year old kid, nor a machine. It's just a large math equation that was tuned to produce outputs similar to what humans produce.
If you choose to think about LLM as a machine you are biased. You expect from it things you'd expect from a machine. Reliability, repeatability, utility, indifference, etc. You pass your past frustrations with other machines onto LLMs.
If you choose to think about LLM as well read 5 year old you are also biased to expect from it different things. You don't have the same frustrations. You have different behaviors towards it. Behaviors that accidentally are more conducive to drawing value from the tool that is LLM.
We perceive the world through metaphors. If you pick wrong (counterproductive) metaphors you end up being constantly upset by something that's just a math result. By no ones fault but your own.
"Teach yourself to not judge people for being human."
"Be a more charitable, generous, understanding person."
Anyone making such blatantly judgemental and egotistical comments to a complete stranger has absolutely no idea what is frustrating to people. And is not being anything like a charitable or understanding person.
Thank you for illustrating perfectly, the mindset that gets you worst possible results from AI. I'm, in your opinion, wrong and you chose to react with indignant fury. What kind of result will that get you?
> Every sentence says something. Every sentence is important and holds its weight. [...] Also, it makes the post harder (and more boring) to read.
I actually prefer that style of writing! (When it's not AI-generated ofc.) And I also try to use it in my technical blog posts. I usually re-read my drafts asking myself: "Does the reader actually care about this? Is this sentence adding something or is it just fluff?"
And actually I feel like AI text usually produces more fluff, or anyway I notice it more, but I see how it can make the result "robotic and boring".
We sorely need a way to reliably detect AI slop, but unfortunately it doesn't seem possible and it's just getting harder and harder.
Last month I tried my hand at finding a way to tell whether an OSS project is slop or not, based on the amount of "human attention" it received vs the amount of code it contains. The idea is that a 100k LOC project which received 3 days' worth of attention from a human is most certainly slop.
The approach doesn't work very well, though¹, mostly because it's hard to gauge the amount of attention that was given. If I see one commit with +3000 LOC, I can assume it's AI-generated, but maybe you're just the type of dev that commits infrequently.
Maybe we need some sort of "proof of human attention" for digital artifacts, that guarantees that a human spent X time working on it.
I suspect that it will be impossible, soon. People will just train LLMs to "act human," and pass the various turing tests we throw at them.
I stay pretty busy[0], and have been accused of "gaming" my GH repos.
That's not the case. I'm retired, experienced, and working on software all day, every day. I just don't get paid for it.
I also don't especially care, whether or not anyone thinks I'm a bot. I eat my own dogfood. Most of my work is on modules that I use in my own projects.
There's no reason to care that a human spent time on it.
Humans are bad at writing code. Garbage PRs and slop have been a problem in open source and bug bounty programs since long before AI came on the scene.
We need better AI so that there's no need to solicit external bug fixes, and better AI so other contributions can be evaluated for usefulness and quality.
What do you care if a human ever looked at it at all? It implies that humans are adding value to the process. It's possible for a human to add value. The right human can add tremendous value. But I'll take a completely autonomous AI over 99% of the human software engineers and 99% of the people contributing PRs and bugfixes.
It was hard to keep up with slop before. It's a lot harder now. AI will help weed through the garbage.
If AI is already mass-producing garbage PRs and other unreliable crap, what makes AI (established as producing unreliable crap) the solution for review? What makes the reviewing AI not produce unreliable crap with regards to the review?
A magical, hypothetical AI that always gets it right and will make all these problems go away is neither a solution nor a plan. It's wishful thinking.
AI in the hands of the right people is incredibly powerful. A good team of engineers with AI doing their own bug-hunting on their own code is already far better than any outsider—human, AI, or human-assisted AI—could ever do. A good internal AI-assisted team is also the only thing that can vet all other contributions. It doesn't matter if those contributions are 100% human-written, 100% AI-written, or a combination. The problem is the same.
Unless you stop accepting outside contributions at all, there's simply no way to determine if a human was involved in the process. Any mandate that all contributions come from humans will fail because there's no detection or enforcement mechanism. You have to assume it's slop either way, and improve your ability to vet it. Only another AI can do that, because we don't have enough qualified humans to keep up.
I did address it: AI in the hands of the right people.
Random contributions to bug bounty programs or random PRs for new features come from all corners: expert engineers producing fantastic code; intermediate engineers trying their hardest but producing mediocre code; junior engineers wasting everyone's time with ill-conceived poorly-written code; and all of the above with varying amounts of AI assistance. And now also purely-automated AI, where the only human involved is pointing their AI at GitHub with no guidance.
You can't stop it on the inbox side. Either you turn the inbox off, or you leverage AI to help you separate the wheat from the chaff.
You by default assume all AI code is slop?
This seems like an approach that won't be beneficial to the OSS overall.
What I found is
1. With LLMs, I was finally able to find time and confidence to contribute a patch. Before, contributing to established project seemed impossible with an amount of guidelines to follow and insider knowledge to have.
2. Many small isolated parts do not require some great code. Just glue, tying libraries together allows producing new features in existing apps. With some cars, it is possible even if I don't know the languages and libraries used.
I recently ran an experiment where I tried to use _quantitative signals_ (and not _qualitative_ ones) to tell whether something is vibe-coded or not.
My idea was that, if I see that your project is growing 10k LOC per week and you're the only developer working on it, it's most likely vibe-coded.
I analyzed some open-source projects, but unfortunately it turns out not to be so clear cut. It's relatively easy to estimate the growth rate of a project, but figuring out how much time developers worked on it is very error prone, which results in both false positives and false negatives.
I just finished¹ building an experimental tool that tries to figure out if a repo is slopware or not just by looking at it's git history (plus some GitHub activity data).
The takeaway from my experiment is that you can really tell a lot by how / when / what people commit, but conclusions are very hard to generalize.
For example, I've also stumbled upon the "merge vs squash" issue, where squashes compress and mostly hide big chunks of history, so drawing conclusions from a squashed commit is basically just wild guessing.
(The author of course has also flagged this. But I just wanted to add my voice: yeah, careful to generalize.)
reply