It's been my experience people invested in static analysis think of it like some wholly additive extension.
Compiler-independent static analysis is code applying rules. Someone has to write all the rules, align them with the language and the compiler. The user rarely reads the static analyzer's code or full rule definitions and is blissfully unaware of the full set of disconnects, inconsistences, and gaps between the coverage of the two.
They only notice divergence in the analysis when it generates a false-positive warning or error.
Static analysis adds compilation overhead. Sure, but I'd hazard that re-reading and in to some degree repeating the parsing/translation process that the compiler is going to do also introduces overhead.
There are some languages that demonstrate sta-by-compiler capabilities with heinous compile times, but it doesn't have to be that way. We can engineer better, but at some point it requires a price. Better comments, boilerplate of some kind or another, annotating intent over idiom.
A complex type system isn't ideal; with the right set of primitives you can achieve sophistication without complexity.
```
Freeable buf = malloc(SIZE);
free(buf); // compiler error, you didn't check buf isn't valid.
free(buf); // compiler error still if you fixed the above, free makes buf invalid.
```
"Making the compiler do STA makes it slow". I don't think that's proven one way or the other. There are examples both ways. "complex" type systems frequently have slow compilers, but if you look more closely that's usually because they're trying to compensate for the disconnect between organically emerged complexity in their under-designed type systems.
Can't is a strong word. You could 100% put it in the compiler. Yet you don't (for the reasonable reasons you give). The line between compiler static analysis and non-compiler static analysis is a design decision. "defeating the point" is a subjective and pragmatic decision, not one rooted in safety absolutism.
I didn't really phrase this particularly well: I meant that you cannot determine this stuff at compile time because it is inherently a runtime analysis. Of course the compiler could emit the Miri interpreter into a binary if people chose to put the pieces togehter that way.
LLMs have great potential. So, it turned out, did uranium, just not as chewing gum or a hair pomade.
There are good ways to leverage LLMs, but there's a lot more load bearing wait on that word 'leverage'. Something needs to do the leveraging, and do it well.
I'm experimenting with my own harness at the moment, currently codenamed Murder because I call the individual contexts/agents 'crow's.
The fundamental unit of it is what I call 'intrusive harnessing', where the harness actively manipulates the token stream so that significant quantities of tokens are only ever exposed to Layer0 when it's useful for them to be present.
For example: the full instructions for shell-tool calling aren't in the system prompt diluting attention while the model is reasoning/discussing what kinds of cat picture you want to put in your app.
My approach is more like dev-branching, and it seems to be working way more effectively than compaction or simple aggressive sub-agenting.
As soon as the harness sees the model is inferring a shell tool call, I stop the inference, mutate the context so that the full set of instructions/examples/guidance for shell tool use are inserted. Once the model has inferred the tool call, I curate the output it gets back. I ask the model to evaluate the output - good or bad - and give it a chance to accept/retry, before allowing the tool-call and output into the original context.
Does it use more tokens? Yes, although we're only mutating at head, so in a long-horizon context, it leans heavily into cache, just not the way anthropic/openai want you to realize you can.
It sounds like compaction but it doesn't come with the nasty brainwash experience where you just need the agent to fix that one last thing, it compacts and the agent comes back a paranoid delusional mad max.
```
<|system|>You're an AI agent. You do agent things.
<|system|> ... there's a list-dir tool and a shell-call tool ...
<|system|> ... memories
...
<|user|>It doesn't look like it ran.
<|reason|>I should look and see if there are any errors in the log file.<|agent|>I'm going to read the log file to see if there are any errors.
<|tool-call tool=shell-tool
```
We stop there, and splice in the detailed instructions for the tool the model was about to predict. I'll use <|ALLCAPS|> to denote harness-generated pseudo turns.
```
... as before ...
<|agent|>I'm going to read the log file to see if there are any errors.
<|SYSTEM|>Shell Tool: ... shell-type=bash, zsh, fish, pwsh on this system. Preferred shell is ... Additional arguments ... Pagination ...
<|tool-call tool=shell-tool
```
the model finishes out the call. On windows, with a typical harness, this frequently goes like this:
```
<|tool-call tool=shell-tool|>Get-EventLog ... | head<|tool-call|>
'''tool-result
error: unknown command: head
'''
<|agent|>Ah, windows doesn't have head. Let me just read the whole log.
<|tool-call ...|>
'''tool-result
... 500k tokens ...
<|agent|>I see some windows log events but you didn't ask me a question. Daisy, daisy?
```
With Murder it goes like this:
Rev 1
```
... prefix as before ...
<|tool-call tool=shell-tool
```
Rev 2
```
... prefix as before ...
<|SYSTEM|> ... how to use shell tool; shell-related memories and rules ...
<|tool-call tool=shell-tool shell=pwsh fence-vs-escape=true|>
'''pwsh
Get-EventLog ... | head
'''
'''tool-result
error: unknown command: head
<RESULT>Your tool call terminated with an error, ...
... structured response required ... options
<ACCEPT /> or <ACCEPT> <WITH> annotation </WITH> </ACCEPT>,
<REDO> ... </REDO>
<RETHINK> ...
<|reason|> windows doesn't have the head command. Let me try reading the whole log.
<REDO><TOOL-CALL> ... replacement tool call ... </TOOL-CALL> <WITH> ... model note ... </WIDTH></REDO>
```
I take that feedback and loop it, so, Rev 3:
```
<|system|> ... how to use shell tool; shell-related memories and rules ...
<|agent|>
... prefix as before ...
<|SYSTEM|> ... as before ...
<|agent|>{prev_cmd} failed, because windows does not have a head command. Let me try reading the whole log.
<|tool-call ... no head ...|>
'''tool-result
... first few lines of result ...
'''
<|system|>Your tool call succeeded but generated 446,219 lines of output. Only the first 5 were listed.
... structured pagination / retry / rephrase options ...
```
It then repeats while the model figures out the right command, figures out which filters to use, but the harness effectively immediately guides the model to do an immediate [optionally self-adversarial] review of the command against the output until the model concludes that the result is useful by various criteria. That doesn't mean successful - sometimes what is superficially an error (no such file or directory) is the answer you were looking for.
Let's say it takes the model 3 more turns to figure out how to use event viewer, and finally it <ACCEPT>s.
Here's the win, the outer main context - the one we're going to keep growing as you work with the agent, looks like this:
```
<|system|>You're an AI agent. You do agent things.
<|system|> ... there's a list-dir tool and a shell-call tool ...
<|system|> ... memories
...
<|user|>It doesn't look like it ran.
<|reason|>I should look and see if there are any errors in the log file.<|agent|>I'm going to read the log file to see if there are any errors.
<|tool-call tool=shell-tool shell=pwsh|>Get-EventLog ... | ... | ...
'''tool-result (use ref-tool id=A401U8X593 for full transcript)
Event ID | Last Occurred
1010111 | 3 weeks ago
'''
```
We used a lot more tokens. How can that possibly be good?
It's happening at the end of the context, so the cache comes into play very effectively.
But if we'd let all that derp into the context, it would be a potential attention sink degrading the value/worth of every subsequent token.
The pattern of try-thing-fail-try-solution-fail-try-win appears to be an incredibly strong pattern for most agents.
Fundamentally: When you're 3 prompts down the line and there's the imprint of the model doing "somewindows command | head" in the context with the model litigating it and fixing it -- that meta-pattern will drive the model to predict more of these patterns. It's going to repeatedly eff-up the exact way it saw in its training material.
When I try to get Claude/Copilot to work on this codebase, they freak out. The hyperbole/marketing pitch the agents were trained on and is built into their inner prompts cannot seem abide the idea of stopping an LLM mid inference. They seem driven to perceive an LLM endpoint like a 911 call you can't just go quiet on.
I have a mechanism for non-parallel sub-agents ('maggots', their job is to curate a large body of work whose full text is irrelevant to the main context). Basically just a tool call, but every time Claude or GPT have been near it, they've broken it, forcing it back parallel so they can send the invoking model a notification that it's child has been spawned and the parent should call the 'check-result' or 'wait-result' tool when they're ready to receive the results.
One of my test architectures is running against a solo Unsloth Studio instance that can only load one model at a time. It really doesn't react well to having you load the coding model to start your sub-agent work and unload before the model has generated its first token... :)
Also exploring mechanisms that try to pre-emptively keep attention-draining distractions/anti-patterns out of the context, things like when a model edits a file, we take the cache hit of removing the stale versions it read to make the modifications, replacing them with a reference syntax that the model can access in a sort of sandboxed auto-fork of the context.
That's going a little slowly because I'm trying to strike a balance between working 'reasonably' with extant models, and providing a mechanism to SFT/lorafy a model to make best use of it.
The streaming detection piece is what I'd want to understand better. How are you catching that the model is committing to a tool call before generation finishes? Pattern-matching on partial token sequences, or something at the structured output layer that gives you earlier signal?
On the caching claim, I want to make sure I'm reading this right.. Mutating at the head of a long context should bust prefix cache for everything after the insertion point. The win would come if the stable prefix before the insertion is long enough that most of the cache value lives there anyway, and tool invocations tend to cluster toward the end of a conversation. Is that the workload shape this is designed around?
There's a reason engineers are prissy about people coming along and saying "I write code, I'm an engineer" that people periodically try to sand-paper away.
Engineers don't just tie a sheet to a rock and throw it off a cliff and call themselves aerospace engineers.
They do full diligence on the theory, math, physics, material science, fluid dynamics, etc, and plan a controlled series of tests specifically designed to verify/challenge/disprove their concept and the theories behind it.
Sure, there's a team member ultimately responsible throwing half a dozen rocks off a cliff in the first test.
A technician.
The guy who throws the rock off the cliff is a technician.
The other glossed over part is that the above sounds like science.
Engineering often continues until the concepts and theories are developed into safe, practical methods. "If you stay within these parameters, you can confidently expect these results." The reliability can be codified and reproduced without going from first principles on every application of it.
It's not clear to me that the current AI fad is really developing such reproducible, safe methods. "If you stay within these parameters, you might get these results. Or a teapot. Or some subtly misleading fabrication."
You have to do full due diligence to validate every result. There is safe usage where the hard work was done up front so that day to day practice can skip to boring and reliable application.
To a software engineer, a (current) LLM is a stateless algorithm that performs an idempotent transformation on a large numeric input.
People who think it's a system that thinks and reasons have confused the agentic harness, perhaps forgotten(?) layer0[0] is a seed, the inference engine sets to a concrete value when the caller leaves it as 0.
They probably work on (current) AI software by repeatedly writing prompts like "DON'T READ THE FILES IN /tmp. SOME OF THE FILES IN /tmp ARE VERY LARGE. DUE TO THEIR SIZE, YOU ARE NOT TO READ THE FILES IN /tmp." and wondering why the model becomes obsessed with files in /tmp 100k tokens into every conversation.
I was the kind of shell guru whose teeth itched when they saw someone doing `grep | awk`. Then I had to try and bring Windows into a Mac+Linux CI system under Jenkins groovy files that were full of `"""sh` fragments, shell scriptlets wrapping python, etc, etc, and I don't bat (I don't groovy either).
Option 1: Learn to bat and try to translate. Hmm. No. Just no
Option 2: ?
Pwsh core had just come out. My immediate thought was that it would be great material for an anti-MS-ragging blog post, but then a line leapt out at me from one article I was glossing over: "... POSIX Terminal Shell Spec ...".
I still wanted that anti-MS-ragging blog material, so I decided to try and use Pwsh as a Rosetta stone until I got to a point I could convert to a real language.
But things just began to click for me. It was like going from Perl to Python - suddenly everything is an object and you can interact with everything* that way.
There's no need for grep or awk or sed in pwsh, because the output of a shell command is an object -- a string (or []byte). It has methods.
(netstat -an).replace("192.168.86.", "10.0.100.")
6 years later, pwsh is the default shell on my Mac, Ubuntu boxes, lab vms, ... everything but my docker containers unless I'm feeling feisty.
ideally containers run a single process. if you're spawning shells you're doing something wrong. and idealier, you should use just bare processes with namespaces and abandon docker.
Careful - you start from the POSIX terminal spec, you think maybe it would be interesting having objects instead of raw text streams, and next thing you've reinvented powershell...
(*35 years living and loving sh/ksh/bash/dash etc, only tried pwsh so I could write some comparisons and slag off MS a bit; now it's my default shell on everything)
It doesn't sound like you were trying to launch a product, but doing an experiment and someone threw you under the HN-spotlight-bus :) Is this a "see what I can achieve with LLM coding" or is this "build this and see how much of the coding can be accepted from LLMs"?
I thought that was where I was going to end up, but that's where Claude's remote control or copilot's agents tab spared me.
The hitching is compounded for me because I tend to run the agents in squads: the agent I talk to operates a strict no-coding 'producer' mode, it tasks a sub-agent to do the research or coding, then the results go via a file to a critic or review agent; keeps the producer context very minimal and lean. Not convinced it's as necessary as just starting new contexts frequently with Fable etc.
My general rule is that I won't commit code a human hasn't seen/reviewed to production codebases, and I know I won't maintain that rule if I have to read all the slop that gets generated first time round without an AI reviewer pass.
So far my producer skill has survived 4.6 thru fable in succeeding to treat the review/critic output skeptically, as a likely yes-man or team player.
The key is to remember that, as of Fable, the size of the training corpus segment representing people responding to AI-generated content is still relatively tiny. Telling Sonnet 4.6 "this is code an agent produced" has a near decorative effect with no apparent significance, Sonnet 4.8 shows some misgivings, and when I experimented with Fable it seemed to do well at anticipating the kind of slop 4.6 would throw you.
Interestingly, to me, telling Fable that code a previous Fable agent wrote was AI generated seemed to raise some kind of "I'm being benchmarked" flag; expanding the reasoning finds it being evasive and mistrusting; look past the null derefence because this must be a trick question type thing.
I wish Claude's remote control sessions were longer running. If I forget about a conversation it's been cognitively difficult for me to track down which tmux tab = the correct remote control session in the Claude GUI. Remote control does come in super handy for pasting images in when some shell environments where pasting doesn't work for...reasons. The output in the GUI is also so much easier to read.
> On an early checkpoint of LFM2.5-2.6B, 10.2% of completions on hard math and coding prompts produced repetitive loops. After Antidoom training, that rate fell to 1.4%, with eval scores improving across the board as a direct result of reduced looping.
That's a 9% reduction, and it doesn't really discuss applicability over longer contexts, different language/language familes/programming, or context/rope length aspects.
Several families - esp Llama and gemma - seem to be easily doom-inclined during expansions (I realize this is a pathological misuse of the tooling, but it's what people do): prompt: "add comments to this file", +1000t file read, [+3k reasoning], +800t edits, +1100t file re-read promoted by tool, prompt:"pep8 format it", [+2kt reasoning], +700t edits, +950t file re-read.
The unmodified/repeating sequences in those file reads, the differences between them, is a honeypot for attention.
I'd non-scientifically concluded for myself that this was tied to an anti-pattern for rope. In the first portion of the context inference needs attention to fall in certain places to produce the correct suggestions for the first round of edits, and somehow largely refocus attention to the subsequent version of the file in the context to correctly apply the subsequent edits, not getting distracted by the tantalizing pattern of repeated sequences and disrupted sequences
Using Gemma4 e2b I can repro this in LM-Studio, and fix it by hand-editing the context so that there's only the latest version of the file being worked on up at the start of the context, or replacing the original reads with something like:
"
[tool name='read-file' path=...']... elided, see file.v1.py if you genuinely need to revisit the original state...[/tool]
"
The handful of times I've tried this over the last 3 years it hasn't cared to discover file.v1.py didnt exist, limited samples = limited confidence.
``` ((void()(void) free)(buf); ((void()(void) free)(buf); ```
It's been my experience people invested in static analysis think of it like some wholly additive extension.
Compiler-independent static analysis is code applying rules. Someone has to write all the rules, align them with the language and the compiler. The user rarely reads the static analyzer's code or full rule definitions and is blissfully unaware of the full set of disconnects, inconsistences, and gaps between the coverage of the two.
They only notice divergence in the analysis when it generates a false-positive warning or error.
Static analysis adds compilation overhead. Sure, but I'd hazard that re-reading and in to some degree repeating the parsing/translation process that the compiler is going to do also introduces overhead.
There are some languages that demonstrate sta-by-compiler capabilities with heinous compile times, but it doesn't have to be that way. We can engineer better, but at some point it requires a price. Better comments, boilerplate of some kind or another, annotating intent over idiom.
A complex type system isn't ideal; with the right set of primitives you can achieve sophistication without complexity.
``` Freeable buf = malloc(SIZE); free(buf); // compiler error, you didn't check buf isn't valid. free(buf); // compiler error still if you fixed the above, free makes buf invalid. ```
"Making the compiler do STA makes it slow". I don't think that's proven one way or the other. There are examples both ways. "complex" type systems frequently have slow compilers, but if you look more closely that's usually because they're trying to compensate for the disconnect between organically emerged complexity in their under-designed type systems.