Rendered at 20:54:48 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
OleksandrC 14 hours ago [-]
If you're looking for a coding agent that would fit nicely into resource-constrained environments (such as laptops, or tiny VPS servers, or tiny single-board computers, etc), and would also work great with local models - you might also like hax (https://usehax.dev/). 0.7 MB dynamically linked native C binary, few MBs of RAM usage when running, auto-discovers config from running local llama-server, and uses minimalist system prompt and tools for lean context usage.
crossroadsguy 5 hours ago [-]
This looks like such a nice cli/tui talking/chat harness - an agent with which I can work on non-coding tasks (where it literally doesn't need any tool - maybe a read tool for few files in that folder, or not even that) - writing critique (w/ strict strict for on generation/suggestion), analysis, summary etc.
I've been so far using pi like this:
> pi --offline --no-extensions --no-skills --no-prompt-templates -nc -nt --thinking low --system-prompt "$(cat <custom purpose path for system/role prompt>)
Purpose was to reduce token usage to an absolute minimum (zero extra token) as often I use this per use API keys (and not my GLM key, which let's say, is a bit "different" when it comes to conversations).
Are there any other tools specifically designed for such tasks? Even though hax looks like the absolute bare minimum one can go while still being usable.
I wish there was also tools that would refuse and reject (or prevent) models from inserting "thinking" messages into responses. They keep appending those and that very soon that part starts snowballing on steroids.
it grew out of annoyance of dependencies on js runtimes, probably similar to you. mine additionally works on solaris and esp32.
could be interesting to collaborate!
d3Xt3r 11 hours ago [-]
I'm using fourlexboehm's minimal agent[1], it's only 674 lines of C (with comments), which results in a tiny ~10KB binary (after various optimizations). Dynamically linked, with only libcurl and libc as direct dependencies. Could probably shave a couple of KBs further if I UPX'd it.
If anyone's aware of a smaller agent than this, hit me up!
It's absolutely amazing to see this harness. I've been on the lookout for something like this for a while now. I've used both Pi and Maki in the past but was unhappy with certain aspects for both. Pi is not respecting XDG and the author refuses to change and maki you curl an install script into bash.
So the points about it being a well behaved unix tool, installing it via brew and it not being react are points I - love - to see.
Thank you for making it, I will definitely try it out.
sejje 6 hours ago [-]
You could always curl the install script to your disk instead of into bash.
It's a command you copy/paste, but you are free to edit it to suit your whims.
SSLy 6 hours ago [-]
try also goose, 70-80 MiB RSS.
altruios 5 hours ago [-]
Digging down through the source code, I see "const char *system_prompt_append;" in the config.
A lot of people have multiple files for their injected system prompts (a.la openclaw or hermes), I think it would be a good idea to either add or modify this point to be able to handle multiple file injections (system_prompt_append_folder or the like). Fitting that shape would make it easy to compare to those systems and make it easier for people to transition from those systems to yours.
mring33621 8 hours ago [-]
Happy Hax user here! Thanks again, Oleksandr!
julesrms 13 hours ago [-]
HN seems to have had a stream of agent harness benchmarks floating past. And every time I wonder where the people who create these tests are looking when they're deciding which harnesses to test? Because right now nobody seems to bother testing mine! (https://juggler.studio)
I know Juggler's very new, but there's so much churn going on in this area that it's hard to know where I should be pushing it. It's hard to guess whether juggler's strengths would played well with a particular test like this, or made it look bad, all feedback about the kind of parameters people are interested in is useful to know when I'm deciding what to optimise.
kouteiheika 10 hours ago [-]
> And every time I wonder where the people who create these tests are looking when they're deciding which harnesses to test?
Well, everyone and their dog has a custom harness now (including myself!), and more often than not they're just... pretty much the same, just with a slightly tweaked UI and different defaults? The harness needs to either be one of the popular ones, or have some sort of a unique feature which actually differentiates how it runs, not just how it looks/controls like.
For example, one of the unique features of my harness is that the filesystem is completely virtualized and all filesystem I/O is part of the session. This means that any files the agent modifies aren't actually modified on disk until the user uses `/apply`, and rewinding the session also rewinds the state of the disk, and this works for all I/O and all tools (including anything spawned by the `bash` tool), and also supports at-will forking of the session including the filesystem state. This means that the agent can run fully YOLO and won't be able to nuke your disk, and it also means that there's no need to use git worktrees and such because multiple agents can just run in the same directory at the same time without an issue.
Essentially, what I'm trying to say: I'd love to see more experimentation in the harness space which doesn't just involve "a different GUI".
aktau 8 hours ago [-]
> For example, one of the unique features of my harness is that the filesystem is completely virtualized and all filesystem I/O is part of the session...
What are the mechanics of this? You mentioned bash so I asssume you don't "control" all the tooling. Is it running in a VM with an overlay filesystem that takes snapshots at every file interaction? Or perhaps outside a VM but containerized with a FUSE-based overlay filesystem?
kouteiheika 8 hours ago [-]
It's not a VM; it's containerized with a FUSE-based custom overlay, and the agent never touches a real mount (with two exceptions: passthrough read-only mounts for things like /bin, and ephemeral tmpfs mount at /tmp).
julesrms 9 hours ago [-]
What I've found difficult with harness design is deciding which features are actually going to be useful, and which ones are cool-sounding ideas that get people talking, but nobody actually uses in practice. It should be easy to tell the difference, but it isn't!
I've had the same "virtualised filesystem" in my TODO list for a long time, but can't decide whether it's a game-changing workflow that everyone will want, or a maintenance burden that I'd regret adding... Could go either way!
nasutton12 2 minutes ago [-]
There are a comical amount of harness design features in the early 1.x release. A full LSP integration, find symbols, batch edits, etc. Terminal bench isn't everything but I spent a couple of weeks testing different combinations to no statistical effect greater than a bare loop. It was like running uphill against what the underlying LLM wanted to do.
kouteiheika 8 hours ago [-]
> What I've found difficult with harness design is deciding which features are actually going to be useful, and which ones are cool-sounding ideas that get people talking, but nobody actually uses in practice. It should be easy to tell the difference, but it isn't!
Same. But there's a trick to it, in a way: vibe code it first, test it out yourself, and then decide whether it should be reverted or kept and deslopped/properly shipped.
> I've had the same "virtualised filesystem" in my TODO list for a long time, but can't decide whether it's a game-changing workflow that everyone will want, or a maintenance burden that I'd regret adding... Could go either way!
For me personally it's a game-changing workflow, because it means I don't have to sandbox (e.g. with a Docker) my agents while still being able to run full YOLO. Essentially have my cake and eat it too. If the agent does something stupid I can just rewind. And I don't need to spend as much effort continuously reviewing what it does, because I can just do it right at the end on the `/apply` screen.
(But this only applies if the filesystem is actually fully virtualized, as opposed to the git-based checkpointing that so many harnesses have, which to me is pretty much useless because it doesn't actually give me any guarantees. That is: the agent running `rm -Rf /*` should be a non-event that doesn't do anything to my filesystem, and can only nuke the agent's view of the filesystem.)
Of course, this could just be me, and other people may not really care. But then, is your objective to make the best possible harness for you and people like you, or for everyone? In my case it's the former.
sejje 6 hours ago [-]
prime-agent tried something new fwiw
i think headlong as well
blurbleblurble 5 hours ago [-]
open-jarvis seems cool too
nasutton12 10 hours ago [-]
I'll try it.
alex_john_m 15 hours ago [-]
What is this supposed to mean?
"it spreads up to 50% between nights, so nothing between the lean arms is a finding."
entrope 15 hours ago [-]
The same thing as the last word of "That is the difference between 22 and 226 seconds, measured." Techies I know would mostly omit "measured"; the rest would show, not tell.
arjie 13 hours ago [-]
Pangram fires as usual. Human opening, 75% machine.
nxobject 12 hours ago [-]
Whatever it is, it’s just as hilarious as Engrish…
stavros 14 hours ago [-]
Means Claude can't write for shit.
ramon156 13 hours ago [-]
Or chad
toasty228 13 hours ago [-]
A bit off topic because I'm not using local models, but I recently benchmarked codex vs pi vs omp with my workload and found codex to be both faster and more token efficient than pi/omp. There was not a single case for which pi was faster/cheaper
crossroadsguy 5 hours ago [-]
The trick to use Pi is you use it in N ways if you have N use-cases. You need N set of shortcuts/plugins/aliases etc for that. Can get tiring at times.
So I keep Pi for just one case - when I have to easily strip everything out for some work. Anything heavier and OpenCode or Claude are ones. I am sure I can make Pi behave as I've suggested above the "N harnesses within 1 harness" and I even tried but it simply started getting out of hand and using the harness started becoming the frustrating hobby.
As for OMP, I just don't understand why would anyone use that not Pi or other "full-fledged" harnesses.
weiran 13 hours ago [-]
Pi is a very basic harness by design. On the other hand OMP is a bloated mess of other people’s workflows.
The trick with pi is to extend it yourself as you use it. It’s pretty easy to do.
toasty228 12 hours ago [-]
Every single article and banchmark say pi saves token by default, the more I add extensions the more token hungry it gets.
pi used 2-3x the tokens of codex. pi with subagent pkg used 8x-10x the tokens of codex.
I don't see how adding bloat to pi would make it more token efficient if the baseline is so poor to start with
kadoban 6 hours ago [-]
You'd have to look where the extra tokens are coming from. If it's using extra turns or doing extra work because a tool it ~wants is missing, then adding extra things will help. Otherwise, it won't.
If it's missing guidance that would help, system prompt additions might help.
8x-10x the tokens is wild, is this for some tiny artificial benchmark? That's just too much extra for something not to be just broken.
toasty228 4 hours ago [-]
It was a real workload from my day job, a modest merge request, I asked for a review and for an implementation plan to fix the findings, nothing crazy. It's the most simple thing I could think of that wasn't an artificial benchmark, self contained, no need to look for extra documentation, web searches, etc.
kadoban 4 hours ago [-]
Yeah something must be actively broken or just an awful extension for it to have that much effect.
Most things you can mess with the big effect is like, oh a thousand tokens ended up in the ~system prompt, or 10% extra or fewer work based on extra tool calls or churning through thinking or whatever.
Harness stuff if it's 8x worse that's like, it's fucked and broken, something went _wrong_.
crossroadsguy 5 hours ago [-]
I don't know what you tried but I tried to do something in Pi what I'd have usually done in Claude and it devoured tokens (much more than Claude) and I wasn't even close to finishing the task. Mostly because the barebones Pi was/is very inefficient at anything with any weight.
Then I began to customise it to be as good as Claude but eat less taken. I got tired and I had not even scratched the surface. Gave up.
I finally realised, at least for me, Pi's best use case is - strip even the little "extra" Pi comes/starts with and then use it just like that if you have a task/work that is appropriate for that bareness.
Isn't using it with GitHub Copilot a violation of GitHub terms of usage, due to usage of internal API? I read that they have an official partnership with OpenCode, and I guess it is fine to use it with OpenAI Codex, but what about other coding agents like maki?
tontinton 5 hours ago [-]
Is it really? I'll need to check it out
nopurpose 11 hours ago [-]
Do I understand correctly that in your harness models don't call tools and pass output of one to another via context, but instead code whole pipeline as small on-demand tools and see only final output?
tontinton 5 hours ago [-]
They can do both, the model decides when to use code_execution like any other tool
larodi 12 hours ago [-]
I can see this pattern of many people using Qwen 3.8 27B for local inference both on Apple Silicon and x86. This implies the model must be very good, given all these peoples' opinion converges on it.
suprjami 8 hours ago [-]
Qwen 3.6 27B and 3.8 27B are the darlings of local inference at the moment.
The only other thing anyone is using is Qwen 3.8 Flash Next, only by memory-rich people.
Depending on which benchmarks you believe, these models (and the Ornith 1.5 finetune of Qwen 35B-A3B) are competitive at about Opus 4.5 to 4.7 level. That matches my experience in real tasks over the last few months.
Not bad for something you can run at home for a couple of thousand dollars.
noir_lord 9 hours ago [-]
I've played with a fair few local models, Qwen 3.8 27B is the only one I've currently kept around (Muse Glimmer came the closest but will vacillate wildly if pushed, Qwen doesn't seem to do that (as much), I don't use it to generate code, it's essentially a glorified linter, it's pretty good at spotting dumb stuff you miss when you proofcheck your own code, especially things like comment drift where I alter a function and it makes the comment slightly (or sometimes wildly) out of sync with behaviour.
It's fine for that (and I happen to have a 24GB VRAM GPU anyway since I game on the same PC).
It's neat but for me not world changing.
It's also just fun to be able to poke stuff and see what it can and can't do (but I could see how it could also become a time trap in cases where it gets kinda close and you want to fix that).
Scarjit 11 hours ago [-]
For it's size it's pretty good.
I run it and the older Qwen 3.6 35B A3B, which is significantly faster, but also a bit dumber, on both my PC and my MacBook.
Wouldn't trust it for long form coding, but for shorter stuff it's really good.
snek_case 9 hours ago [-]
How far can you get with Qwen 3.6 35B A3B? What scope of task is it able to do?
lrvick 8 hours ago [-]
Really surprised to not see jcode in there which beats everything else in efficiency by miles with one tiny rust binary.
humbleferret 13 hours ago [-]
Nice writeup! I imagine these results change as harnesses are updated, so you'd need to frequently rereview.
I'd love to see a tiny, reproducible benchmark repo that anyone can drop on their own hardware and then run against all harnesses at once to compare the per turn prefix token count, time to the first token, experienced tokens/sec (and prefill), cache reuse % and a pass rate on a deterministic set of small tasks. I think it could also be useful to have some way to share results and hardware for others to compare.
montyanne 15 hours ago [-]
Neat article.
“Chad” initially looked interesting but the minute I saw the ai-written markdown and giant commit I just left. I just can’t bring myself to read someone elses’ slop, regardless of performance.
If all a developer hand writes is a truthy and readable markdown document, I really don’t care if the rest of the project is vibe coded, but I struggle to get interested in AI generated summaries and docs.
CGamesPlay 15 hours ago [-]
I for one am excited to learn more about how it spreads up to 50% between nights, and how nothing between the lean arms is a finding.
ramon156 13 hours ago [-]
Don't forget it's, measured.
imtringued 10 hours ago [-]
Not to mention it's yet another bash does everything harness which compares itself against claude code when the closest competitor is pi.dev?
montyanne 15 hours ago [-]
[dead]
asdfsa32 12 hours ago [-]
What is with the website though? Rubbish scrolling. Junky rendering with artifacts if you scroll fast.
embedding-shape 11 hours ago [-]
Notion is a "knowledge database" with awful performance and jank, that some people have decided sounded like a perfect place to host their blog for whatever reason. But these shared pages been as buggy as the first time I saw them years ago, not sure what they're doing.
8 hours ago [-]
gramie 5 hours ago [-]
Yes, scrolling with the arrow keys does nothing for several presses, then jumps, then acts normally, then jumps again. Really stupid and annoying.
Also, Alt-Left Arrow doesn't return me to the previous page (Hacker News) and I have to use my mouse.
nasutton12 10 hours ago [-]
sorry, its a notion not a real site.
teekert 15 hours ago [-]
Fun reference I tested on 32 GB ram laptop with no extra GPU: llama.cpp: “what is ls”, almost immediate starts answering at one ~word/sec. Ask opencode with same model (some gwen e4b or something) to check what’s in its working directory: 20 min to response.
throwa356262 14 hours ago [-]
Opencode system prompt contains a lot of stuff but even worse is oh-my-pi where their long prompt looks like random garbage hallucinated by a 2023 LLM:
I'd be interested to see how Reasonix stacks up here - they seem to have spent a lot of effort on tuning prefix cache reuse
bozhark 6 hours ago [-]
Where is the hugging face of agents and harnesses?
grigio 10 hours ago [-]
jcode is the best for RAM usage and browser interactions
nottorp 11 hours ago [-]
Uh I have this feature I'm doing where I used a context to 600k tokens, then told Claude to make itself a summary and started a new context with it, and it's at 500k already.
Mind, this is more of an exception, at least for my work, but still, what do you do with 32k tokens?
Edit: i resumed a quickie that basically added a 10 line function in one spot, and even that is 50k tokens...
I've been so far using pi like this:
> pi --offline --no-extensions --no-skills --no-prompt-templates -nc -nt --thinking low --system-prompt "$(cat <custom purpose path for system/role prompt>)
Purpose was to reduce token usage to an absolute minimum (zero extra token) as often I use this per use API keys (and not my GLM key, which let's say, is a bit "different" when it comes to conversations).
Are there any other tools specifically designed for such tasks? Even though hax looks like the absolute bare minimum one can go while still being usable.
I wish there was also tools that would refuse and reject (or prevent) models from inserting "thinking" messages into responses. They keep appending those and that very soon that part starts snowballing on steroids.
it grew out of annoyance of dependencies on js runtimes, probably similar to you. mine additionally works on solaris and esp32.
could be interesting to collaborate!
If anyone's aware of a smaller agent than this, hit me up!
[1] https://gist.github.com/fourlexboehm/a60e4ef9306744483731cd1...
So the points about it being a well behaved unix tool, installing it via brew and it not being react are points I - love - to see.
Thank you for making it, I will definitely try it out.
It's a command you copy/paste, but you are free to edit it to suit your whims.
A lot of people have multiple files for their injected system prompts (a.la openclaw or hermes), I think it would be a good idea to either add or modify this point to be able to handle multiple file injections (system_prompt_append_folder or the like). Fitting that shape would make it easy to compare to those systems and make it easier for people to transition from those systems to yours.
I know Juggler's very new, but there's so much churn going on in this area that it's hard to know where I should be pushing it. It's hard to guess whether juggler's strengths would played well with a particular test like this, or made it look bad, all feedback about the kind of parameters people are interested in is useful to know when I'm deciding what to optimise.
Well, everyone and their dog has a custom harness now (including myself!), and more often than not they're just... pretty much the same, just with a slightly tweaked UI and different defaults? The harness needs to either be one of the popular ones, or have some sort of a unique feature which actually differentiates how it runs, not just how it looks/controls like.
For example, one of the unique features of my harness is that the filesystem is completely virtualized and all filesystem I/O is part of the session. This means that any files the agent modifies aren't actually modified on disk until the user uses `/apply`, and rewinding the session also rewinds the state of the disk, and this works for all I/O and all tools (including anything spawned by the `bash` tool), and also supports at-will forking of the session including the filesystem state. This means that the agent can run fully YOLO and won't be able to nuke your disk, and it also means that there's no need to use git worktrees and such because multiple agents can just run in the same directory at the same time without an issue.
Essentially, what I'm trying to say: I'd love to see more experimentation in the harness space which doesn't just involve "a different GUI".
What are the mechanics of this? You mentioned bash so I asssume you don't "control" all the tooling. Is it running in a VM with an overlay filesystem that takes snapshots at every file interaction? Or perhaps outside a VM but containerized with a FUSE-based overlay filesystem?
I've had the same "virtualised filesystem" in my TODO list for a long time, but can't decide whether it's a game-changing workflow that everyone will want, or a maintenance burden that I'd regret adding... Could go either way!
Same. But there's a trick to it, in a way: vibe code it first, test it out yourself, and then decide whether it should be reverted or kept and deslopped/properly shipped.
> I've had the same "virtualised filesystem" in my TODO list for a long time, but can't decide whether it's a game-changing workflow that everyone will want, or a maintenance burden that I'd regret adding... Could go either way!
For me personally it's a game-changing workflow, because it means I don't have to sandbox (e.g. with a Docker) my agents while still being able to run full YOLO. Essentially have my cake and eat it too. If the agent does something stupid I can just rewind. And I don't need to spend as much effort continuously reviewing what it does, because I can just do it right at the end on the `/apply` screen.
(But this only applies if the filesystem is actually fully virtualized, as opposed to the git-based checkpointing that so many harnesses have, which to me is pretty much useless because it doesn't actually give me any guarantees. That is: the agent running `rm -Rf /*` should be a non-event that doesn't do anything to my filesystem, and can only nuke the agent's view of the filesystem.)
Of course, this could just be me, and other people may not really care. But then, is your objective to make the best possible harness for you and people like you, or for everyone? In my case it's the former.
i think headlong as well
"it spreads up to 50% between nights, so nothing between the lean arms is a finding."
So I keep Pi for just one case - when I have to easily strip everything out for some work. Anything heavier and OpenCode or Claude are ones. I am sure I can make Pi behave as I've suggested above the "N harnesses within 1 harness" and I even tried but it simply started getting out of hand and using the harness started becoming the frustrating hobby.
As for OMP, I just don't understand why would anyone use that not Pi or other "full-fledged" harnesses.
The trick with pi is to extend it yourself as you use it. It’s pretty easy to do.
pi used 2-3x the tokens of codex. pi with subagent pkg used 8x-10x the tokens of codex.
I don't see how adding bloat to pi would make it more token efficient if the baseline is so poor to start with
If it's missing guidance that would help, system prompt additions might help.
8x-10x the tokens is wild, is this for some tiny artificial benchmark? That's just too much extra for something not to be just broken.
Most things you can mess with the big effect is like, oh a thousand tokens ended up in the ~system prompt, or 10% extra or fewer work based on extra tool calls or churning through thinking or whatever.
Harness stuff if it's 8x worse that's like, it's fucked and broken, something went _wrong_.
Then I began to customise it to be as good as Claude but eat less taken. I got tired and I had not even scratched the surface. Gave up.
I finally realised, at least for me, Pi's best use case is - strip even the little "extra" Pi comes/starts with and then use it just like that if you have a task/work that is appropriate for that bareness.
The only other thing anyone is using is Qwen 3.8 Flash Next, only by memory-rich people.
Depending on which benchmarks you believe, these models (and the Ornith 1.5 finetune of Qwen 35B-A3B) are competitive at about Opus 4.5 to 4.7 level. That matches my experience in real tasks over the last few months.
Not bad for something you can run at home for a couple of thousand dollars.
It's fine for that (and I happen to have a 24GB VRAM GPU anyway since I game on the same PC).
It's neat but for me not world changing.
It's also just fun to be able to poke stuff and see what it can and can't do (but I could see how it could also become a time trap in cases where it gets kinda close and you want to fix that).
Wouldn't trust it for long form coding, but for shorter stuff it's really good.
I'd love to see a tiny, reproducible benchmark repo that anyone can drop on their own hardware and then run against all harnesses at once to compare the per turn prefix token count, time to the first token, experienced tokens/sec (and prefill), cache reuse % and a pass rate on a deterministic set of small tasks. I think it could also be useful to have some way to share results and hardware for others to compare.
“Chad” initially looked interesting but the minute I saw the ai-written markdown and giant commit I just left. I just can’t bring myself to read someone elses’ slop, regardless of performance.
If all a developer hand writes is a truthy and readable markdown document, I really don’t care if the rest of the project is vibe coded, but I struggle to get interested in AI generated summaries and docs.
Also, Alt-Left Arrow doesn't return me to the previous page (Hacker News) and I have to use my mouse.
https://m.youtube.com/watch?v=c_fQoDkULl0 (see around 8:00)
Mind, this is more of an exception, at least for my work, but still, what do you do with 32k tokens?
Edit: i resumed a quickie that basically added a 10 line function in one spot, and even that is 50k tokens...