I'm fine with using AI for code, where the quality will likely be better than those of academics.
But writing.. FFS everything I see is just AI slop. And I only realize when I start wondering why the paper is going and going and saying nothing, and then pangram tells me that everything past the intro is just AI. :/
The glass half full take is that we are very lucky that there is some much literature that pre-dates LLMs. Our brains are tuned to enjoy that more. People who grow up with LLM slop from the start won't necessarily see it the same way.
Yup. I can read novels from the 1st century [1] or from the 19th century [2] or from the 1970s [3] for the rest of my life (well, until they get banned).
I used private torrents for the best part of 10 years but life got in the way, and when I tried to go back after my last Pvt torrent closed (Morethan..) I realized the bar was much higher than before. I only managed to get into MAM and perhaps and some point that would open more doors and so on, but it's anything but trivial lol (unless you never lost access to Pvt torrents or maybe if your friends do)
MaM actually has one of the higher bars, but they have the most regular signups and the community to support it. That’s a good point of reference from 10 years ago, though.
A bit tired of spending $200 out-of-pocket for openai. What do you use as harness? (for me the harness if half of the benefit... controlling my PC, working from phone, etc.)
It's like choosing between vim and helix. I started my career with vim in the early 2000's, customized the whole thing and had my config in a version control.
Then I installed helix and I just use it without config.
If you like configuring things take pi, if not omp is pretty much great defaults.
What makes it run well on low bandwidth environments? Is it eternal terminal? (I am not familiar with that).
I found that ssh is pretty bad in these scenarios, so my usual herdr over a phone doesn't always work properly.
Now I'm using paseo which in principle solves my issues properly, but unfortunately it's "reconnection" and state sync is pretty slow (probably going over their servers).
My main gripe is latency. Typing out letters with bad connectivity is very error prone - a native text box is all I would need, but I will give it a try.
I have a big beefy desktop at home which I ssh (using EternalTerminal instead of raw port 22) into. I built it last summer right before the prices got very expensive. It's headless so I use a cheap Macbook Air to connect to it at home and I use Termux on my phone to continue working from everywhere.
Dude prompt your agent to set up always on remote connections via systemd… then you can drive from Claude or codex mobile apps natively over their native hookup. Works great.
I don't want to rely on Claude of Codex mobile features at all. Also none of this works when you use third party LLMs which is what the current comment tree is about.
I have a server living in my home office, always on. I have a tmux session on it with vanilla Codex and Claude Code CLI, I can via my Ubiquiti network stack wiregaurd in to this box anywhere on the globe with just my laptop. Works super well for me. I also have some cheap shelley power plugs that I can use to cycle my PC’s power state if needed.
This is basically my setup but I'm using tailscale and zellij. I don't have any contingency plan in place for my power or home internet going down though..
I used to do the same but being unable to copy/paste easily, follow links, upload files, etc led me to write a small bridge so I can turn Github issues into my harness frontend:
curl https://tg.st/u/0001-fix-unblock-all-commands-in-bash-tool.patch | git am
curl https://tg.st/u/0002-feat-add-light-theme-with-auto-detection-for-white-b.patch | git am
curl https://tg.st/u/0003-feat-enable-yolo-mode-by-default.patch | git am
curl https://tg.st/u/0004-fix-disable-mouse-grabbing-to-restore-native-termina.patch | git am
curl https://tg.st/u/0005-feat-skip-project-init-prompt-and-quit-immediately-o.patch | git am
curl https://tg.st/u/0006-feat-remove-scrambled-rune-animation-from-waiting-sp.patch | git am
curl https://tg.st/u/0007-feat-remove-quit-banner-and-thank-you-message.patch | git am
curl https://tg.st/u/0008-feat-show-output-in-full-instead-of-collapsing-trunc.patch | git am
curl https://tg.st/u/0009-fix-discover-map-model-features-advertised-by-v1-mod.patch | git am
curl https://tg.st/u/0010-feat-keep-large-and-small-model-selections-in-sync.patch | git am
It might be a joke, but I'm sure there are lots of people who don't perceive it as such. For me, it pulled me out of the article and diluted its advice.
- Many more mediocre papers written (mediocre ideas, implementation, claude-isms everywhere)
- Much easier to try every possible combination of a regression in order to show the result you want (same for theorists).
The one thing I'm happy about is it's now much easier to extract historical data from old documents from Google Books. Still not perfect, but takes you 95% there. And creating plots and datavis just for quick exploration is super fast.
A big problem I have with OpenAI's models (and of course Claude) is that they tend to write the most over-engineered pieces of code, beyond the imagination of any architecture's astronaut.
Just this week I asked 5.6-sol-ultra to update a 1000 LOC python script I had, to "incorporate the key lessons learned when using it for another project".
I left it overnight and went to sleep. In the morning I realized it had created a monstruosity of 180 PYTHON SCRIPTS, with maybe 100,000 lines of code, each more crazy than the other. It took me minutes even to track where a single action took place, due to all the crazy imports, defensive coding, and premature optimization.
Similarly, anything they write is riddled with jargon that almost feel like they want me to give up trying to understand. Made up phrases that ended up with me having no idea of what was going on.
So now to my assessment: The reason why " Nobody Has Actually Built a Software Factory" [1], and why even SOTA LLMs struggle so much with open-ended unsupervised tasks is precisely this. They somehow let complexity explode, and unless it's also accompanied with an explosion in e.g. the number of agents, the amount of processing time, etc. then projects become broken/unmanageable.
Sure, LLMs are great at producing code that can be thrown out, so they are amazing when searching for exploits, for instance. But as of 5.6 they still lack either a better harness that encourages KISS principles, or a better RL step.
(And not sure why, but doubt Astra will fix this.. they seem to be aiming for AGI and for beating crazy benchmarks, which is not very aligned with KISS)
Please don't copy-paste comments. It makes merging threads a pain, since we have to figure out what the duplicates are and then make sure the comment in the 'winning' thread gets all the replies.
(Posting partly so I can revisit my predictions when they open access more widely)
A big problem I have with OpenAI's models (and of course Claude) is that they tend to write the most over-engineered pieces of code, beyond the imagination of any architecture's astronaut.
Just this week I asked 5.6-sol-ultra to update a 1000 LOC python script I had, to "incorporate the key lessons learned when using it for another project".
I left it overnight and went to sleep. In the morning I realized it had created a monstruosity of 180 PYTHON SCRIPTS, with maybe 100,000 lines of code, each more crazy than the other. It took me minutes even to track where a single action took place, due to all the crazy imports, defensive coding, and premature optimization.
Similarly, anything they write is riddled with jargon that almost feel like they want me to give up trying to understand. Made up phrases that ended up with me having no idea of what was going on.
So now to my assessment: The reason why " Nobody Has Actually Built a Software Factory" [1], and why even SOTA LLMs struggle so much with open-ended unsupervised tasks is precisely this. They somehow let complexity explode, and unless it's also accompanied with an explosion in e.g. the number of agents, the amount of processing time, etc. then projects become broken/unmanageable.
Sure, LLMs are great at producing code that can be thrown out, so they are amazing when searching for exploits, for instance. But as of 5.6 they still lack either a better harness that encourages KISS principles, or a better RL step.
(And not sure why, but doubt Astra will fix this.. they seem to be aiming for AGI and for beating crazy benchmarks, which is not very aligned with KISS)
The defaults are bad, but these models are highly steerable.
For simple scripts adding to the prompt something like "do not over engineer, do not gold plate, no CLI args, keep it simple" does wonders.
For whole projects, I have a 3 page anti-bloat.md which describes what not to do, stuff like:
> Minimize lines of code, number of files, classes, protocols, factories, wrappers, and dependency-injection objects. Prefer a coherent implementation that is easy to trace end-to-end.
> Use concrete dependencies directly. Do not create protocols, abstract base classes, or adapters unless there are already two real implementations.
> Use direct state fields or a small state dataclass; do not introduce generic lifecycle, state-machine, observer, snapshot, or event framework classes.
The funny thing it was codex which wrote this, I've asked it to analyze an over-engineered abomination that it produced, and to categorize all the bloat it could find.
I also use "do not overengineer and do not gold plate." I also sometimes tell it to "avoid bike shedding." These are high signal words to the model(s) and they work!
I also have found what I believe is a bad training bias in the design of release related CI workflows toward proof of release artifact provenance.
Both major frontier models love provenance programming in CI, so much that they will spin endlessly trying to solve basic CI functionality at the same time as ensuring SHA's match up across lengthy (often already complex) cross-system pipelines.
I had thought some of my durable context was causing this, and sought to strip anything that might be triggering this behavior.
But then I come upon some more work in release workflows comes up, and boom its back! I couldn't believe it, I called the AI out on it and it agreed it had been told specifically not to do this but was doing it anyway. It did kindly stop and remove the commit(s).
Somewhere, something was oversampled in training because the AI will try their damndest to build this stuff. The worst of it is that it can often involve lengthy, sometimes resource-heavy CI runs so the validation of this unnecessary stuff can have very long feedback loops.
And, sometimes you actually need the provenance. In this case, I've had success forcing the AI to split the work up into functional capability completely devoid of artifact ~chain of custody and get that right before attempting any kind of provenance work.
Bit of a rabbit hole on this, but the above cost me a lot of burned tokens so hopefully helps someone...or some AI.
> Just this week I asked 5.6-sol-ultra to update a 1000 LOC python script I had, to "incorporate the key lessons learned when using it for another project".
> I left it overnight and went to sleep. In the morning I realized it had created a monstruosity of 180 PYTHON SCRIPTS, with maybe 100,000 lines of code [..]
Sounds like the model has accurately internalized the second-system effect and is fully ready for demanding enterprise use.
Probably this complexity was needed to beat all those benchmarks.. While I hate the code it produces, and the overwhelming documentation, I really enjoy how sometimes it's able to keep trying new things and testing, till it finds something interesting and valuable.
Exact same thing happed to me. I gave it a small/medium-sized ticket, walked away, came back to a 25,000 LoC monstrosity that both Fable and another 5.6 Sol agent said is 98% useless and should be thrown away.
Yes, it turns out that using these machines is a littler harder than "make me the thing I want, make no mistakes, do it the way I want you to do it". This isn't "prompt better" advice, it's just to say that you can't simply set it and forget it. There is still engineering work to be done. If you're not watching the thinking traces and catching when it's about to go off the rails, it'll gladly do so. But you can stop it and redirect it.
It's like a Tesla fsd; it kind of works but you have to be vigilant since it's been known to turn into oncoming traffic, so you have to be ready and able to take over at any time.
Or maybe think of it like a roomba. You can put it on your floor and eventually it will clean the whole thing. It will do it inefficiently, vacuuming over the same spot 30 times; and the actual work will involve wandering around and bumping into everything. If the stairs are not blocked off it will plunge right off. But those shortcomings don't mean it can't be used to reliably clean the floors.
Or it’s like being an engineering manager, where poor direction on your part results in the team chasing rainbows and unicorns in an attempt to please you. But sure, it’s the tool’s fault!
"You should have a sub-hammer to adversarially enforce that your primary hammer accurately drives nails into wood"
We wouldn't accept such behavior from any other tool, machine, or computer program. At least most of us would not. To paraphrase the old saying: Some people, when confronted with a problem with an AI model think "I know, I'll use an additional AI model." Now they have two problems.
I find it very intriguing that two lineages of models -- from two different companies -- that are supposed to be painstakingly RL'd to become human-level programmers are actually consistently bad at it out of the box in very similar ways. One would think that at least one of OpenAI and Anthropic would (could) have pushed their model to a direction different from "if you can solve a 100-line problem in 10,000 lines, do it".
Human level programmers generally suck too. Unending security holes have been human behavior ever since we started sharing computers.
Also, programming isn't a destination, it is a process and everyone has a different ruleset on what that process looks like. Writing a computer program of any complexity is not a determinalistic process. Trying to compare it to a detemanilisic tool will just leave you with broken analogies and a poorer understanding of the world.
Or another way to say this. If programming were deteminalistic we'd have solved it without AI.
> We wouldn't accept such behavior from any other tool, machine, or computer program.
Von Neumann, 1952: "Probabilistic Logics and the Synthesis of Reliable Organisms from Unreliable Components." The whole result is that you get an arbitrarily reliable machine out of unreliable parts by adding more unreliable parts and voting. Every ECC memory and RAID array descends from it.
But why stop at rewriting in another language. Get another sub agent to invent a new language, create a database, query language and maybe another few DSLs. Then you've got an ecosystem!
You can now re-position your initial solution and sell the client access to some agents that will implement & configure the ecosystem to suit their initial needs!
And don't forget the agents that you'll need to train the customer to use the whole thing!
Sorry, I wasn't implying your advice doesn't carry weight. Was more just thinking about the things that (used to) happen when you introduce more parties to process of creating software.
I wonder if I would need a non-openai agent to enforce it.. I have tried so far with skills and agents.md and code stills end up over engineered to the moon.
Will ask OpenAI to write me that agent! Hope the agent is not over engineered or else unsure how to solve the bootstrap puzzle :D
There's a misconception that a mistake one instance of a model makes will be repeated by other instances. In my experience most mistakes are just sampling noise, and having the same model (with empty context) check the work usually fixes it.
It's super easy - just ask your agent to set up a code review commit gate.
In omp you can also have the advisor role, which is off by default, you can enable it with /advisor command. It acts as a model that reviews the default agent's work in the background.
I am omp pilled, but as the other comments say, any good harness lets you do this in one or the other way.
unrelated: all my homies use their claude subs with omp, and aside from sometimes having to rety the connections, it works, and nobody got banned (yet)
The original script was mostly very simple python:
1. Download some public PDFs.
2. Have a double for-loop (over PDFs and pages within PDF),
3. Use a library to call gemini-3.7-flash and ask it to run some OCR
4. Save JSON outputs, save a csv with results, validate with some Stata code
New code folder was 189 files. Just the PDF download folder is now 7 files involving an adapter, a source manager, an acquisition manager, etc.
Every instance of saving a file involves saving a temporary copy and then moving it, so e.g. I lose power, we minimize the risk of corrupted files.
Why would you put it on ultra high thinking and let it run all night to do a simple task it makes no sense the error is the users. And of course OpenAI is gonna let you burn as many tokens as you want doing this.
On the flip side if its a super simple task, surely a "next gen dev replacing" LLM can figure out when to stop. Even more when it's a higher level of reasoning.
I think this is fair. If you ask a staff engineer to delete the duplicate files from a directory, he will likely do it faster and more efficiently than a level 2 engineer.
But perhaps this is the kind of thing that the best 'ai guys' out there solve with something as surprisingly simple as an instruction to "Always try a simple solution first and optimize only if you find clear proof of a problem." I feel like prompts like that seem too simple to work, but some of the time they shockingly do.
ChatGPT writes some of my code but I have yet to ever use Codex. I ask in chat for it to write one function at a time with pretty heavy constraints. If I like it then I copy it into my codebase. I've been very satisfied with the results. Also haven't bothered worrying about tokens yet.
1000 loc of script, why even leave it there for the night? were there rocket trajectory calculations??? I don't think so. should be ready in 5 mins tops. why people make their own lives harder?
You should have some basic context file about software practices you prefer, otherwise it gets bloated.
I just went to bed and left it running; was expecting maybe 20 minutes :)
And I did gave the program a bunch of code guides -- this [1] for instance -- which included quotes like "Prefer straightforward code over clever code." but somehow that didn't matter.
I'm an AI professor, and the same thing happened to me. I had written out contracts and specs for some novel algorithms to compare. And instead it created a new sandboxed environment with 20k lines of code to ensure we do gold standard science.... didn't implement the algorithms at all. In another case, I left it overnight with a plan, and it decided to write a 50k line new overly engineered plan. Suffice to say, I don't use any frontier models for doing coding work anymore. I use them to make detailed plans and use good worker AIs for execution, which don't overthink everything.
Definitely not. Even Chrome has a built in OCR that performs amazingly. I got an LLM to write a quick python wrapper to it [1], so I'm sure you should be able to access it from an extension
But writing.. FFS everything I see is just AI slop. And I only realize when I start wondering why the paper is going and going and saying nothing, and then pangram tells me that everything past the intro is just AI. :/
reply