Rendered at 14:58:35 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
cmiles8 11 hours ago [-]
The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot.
As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does the economic picture that justified the massive infrastructure building that’s now broadly funded by a complex network of debt.
This is what makes open weight models so threatening to them. The political and “it’s China” angle is mostly just a cover for the real reasons why they’re freaked out.
The fact that models are now a pure commodity is bad enough for the big labs. If small open weight models become the norm the big labs are toast.
Aurornis 1 hours ago [-]
> The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages.
Most generic use cases do benefit from a model that has been trained broadly. When you don’t know the specific use case ahead of time, you have to have world knowledge ready to go. Even when coding it’s helpful to have all that knowledge on tap so the model can understand product intent and use cases for the product you’re building.
> As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles
The real expensive part of fine tuning is gathering a good data set. That’s been the hard part of training anything for a long time. If you’re lucky enough to have a neatly organized and clean data set then you can attempt it, but you need to be in a position to run evals and measure quality.
I’ve done it, but there are so many use cases where the engineering, data labeling, and ongoing quality review hours cost so much that it would be cheaper to continue using a frontier lab model that just works from the start.
amelius 48 minutes ago [-]
Can't you take a broadly trained model and throw away useless knowledge (like foreign languages, and 99% of wikipedia) and still end up with a model that is as intelligent and useful as the original but much smaller?
Aurornis 8 minutes ago [-]
Models are not a collection of training data. The entire model is shaped by the training data it sees during training.
What we call knowledge is encoded as weights. Something we identify as knowledge is the sum of many weights across the model. However the same weight may be involved in very different topics. You can’t point to every weight and say that this one is only for a part of Wikipedia or Harry Potter that I don’t need. That weight could be active for a multitude of topics.
There have been a lot of attempts to reduce the size of mixture of experts models by selectively removing experts. The results are not good, but knowledge isn’t neatly separated within the model.
AussieWog93 12 minutes ago [-]
I'm not an AI researcher (would love to hear an actually informed answer on this!), but AFAIK the weights are more or less a black box - you can't easily pinpoint that weights x, y and z relate to Ancient Rome and can therefore be safely removed from a model that's optimised for coding.
mbesto 58 minutes ago [-]
This is basically the entire argument by Richard Sutton happening in real time:
Right now, we keep creating new models with larger and larger weights and throwing hardware at it. His argument is that this is a dead end and ultimately we'll eventually go back to purpose fit algorithms like we've always done in the history of AI development.
moojacob 44 minutes ago [-]
Did you even watch the video? Purpose fit algorithms outperform is quite literally the OPPOSITE of Suttons argument. This is the man who wrote this:
“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.”
His problem with LLMs are they aren’t general enough… In fact his essay the bitter lesson was what inspired the current LLM scaling.
If you watched the video, you can even see the part where he talks about how AI will fully replace humans as the next stage of life. How wholesome!
sailfast 49 minutes ago [-]
The goal is still the singularity though right?
Sure, for business, personal, etc it makes sense to build purpose built but that’s not what these companies are trying to achieve in the end. They want to own a nuclear weapon that outstrips everything else as soon as it exists so they “win”
vkaku 3 hours ago [-]
Very true. In fact, there's a lot more under the surface that will come to light once more efficient tiny models come out.
P.S. Building something that proves that you don't need those many params even.
The premise of your project seems compelling. Is it novel or building on existing work? Anything one could read or watch to get introduced to that area of research?
dineshdb 1 hours ago [-]
I took a look at the codebase and found that this is just pattern matching. There is no any novelty here. It looks good for knowledge-base retrieval system, but other than that, this is just nothing more than a pattern-matcher.
Would love to be proven wrong with future updates.
DanielHB 23 minutes ago [-]
I actually prefer using less powerful models most of the time, I might use a stronger model initially and then switch to weaker models as I fine tune the output.
Offloading too much of your task to models eliminates the human ownership, without ownership you can't move forward. Context sizes can't keep up with large codebases and markdown files with instructions and guidelines only take the LLM so far in the ownership aspect.
My gut feeling is that to start offloading ownership to the LLM we would need to see at least two order of magnitude increases on context size.
samuelknight 2 hours ago [-]
Going by the chart in the article, if your total workload is 1k cataloged items and your quality threshold is 70%, why wouldn't you just pay $19 to gemini API instead of $500 + time to make a custom fine tune?
ozgrakkurt 10 hours ago [-]
As someone who worked at multiple startups, I am pretty sure they get it but it doesn’t fit their goals.
They want to moat where they don’t need to compete with other companies because they have something that other companies can’t have.
In my opinion this is a short-sighted and greedy worldview. Haven’t seen it work personally. It is a different version of the month-to-month salary guy thinking he will be a billionaire and having that thrash “mindset”.
The reality is that practically none of those companies will amount to anything and they would be better off weighing the usefulness aspect of their output more. Instead they are imagining they will be Google.
Anthropic and openai ofc are the pinnacle of this greed culture and they correspond to FTX from the crypto trash hype so I don’t think they fit into the scale of sensibility.
Coming from this perspective, it is pretty easy to see what they are.
ezst 6 hours ago [-]
But then can't the case be made that narrower and stricter-defined use cases are better served by more conventional ML? If/Wherever efficiency is a concern, that is.
gchamonlive 5 hours ago [-]
That's assuming conventional ML and fine-tuned SLMs are interchangeable with acceptable behavioral change/degradation for any specific case
cyanydeez 3 hours ago [-]
LLMs fit between well defined and poorly defined data; if you have well defined data, you can just feed it to ML models and they'll do what you want.
LLMs will take poorly defined data and "potentially" create well defined output.
So ideal systems will likely be constructed with a LLM on one end and a ML on the other and some how, if you can feedback poor data from the ML back into the LLM to clean up the data, you have a magic layer that doesn't care as much about the structure of the data.
That's a lot of supposition, but that's the difference I see between what we can do with LLMs and how ML models are structured. They might be classified as doing the same thing, but their data inputs are vastly different.
sciaijournal 21 minutes ago [-]
[flagged]
torginus 5 hours ago [-]
Yeah and these latest and greatest models universally suck hard for any use case outside of the few 'blessed' ones. Like I'm sure their ability to write prose and generally sound like a human being has regressed quite a bit, but even if not. Opus 5 is barely above GPT4 when it comes to stuff like home improvement advice.
vidarh 7 hours ago [-]
Fine timing takes time and data, though. If smart enough models get cheap enough, then most people have lots of use cases that are cost insensitive enough that it's not worth the effort.
Of course, "smart enough" is a low enough threshold for most uses that this is still a problem for the frontier labs.
But at the same time, a truly smart enough closed model could also potentially command almost whatever they'd care to charge for it.
Whether they can actually get to that level remains to be seen, but I can definitely see a situation where most people are perfectly happy with cheap middle of the tree models while large corporations pay magnitudes more than current API pricing for access to models never even marketed as a mass market product and keep the labs afloat.
It's of course be a lot easier for them to find the path towards that of they didn't need to compete with open models in the meantime.
pbronez 4 hours ago [-]
Yes. Fine tuning is an optimization. Like all optimizations, it’s downstream of figuring out your problem, implementing a naive solution, scaling the naive solution, and getting frustrated by SWAP-C. THEN you start poking at what optimizations are possible, choosing an approach, implementing the optimization, and redeploying.
This cycle happens when you have a well defined use case with high volume. It is the far opposite end of the spectrum from the general purpose intelligence on tap that frontier AI models purport to deliver.
I think focused fine tunes and big general models will coexist. Ideally with smart routing and caching to use small, specialized and local options when appropriate.
autuni 7 hours ago [-]
seems that, like software engineering and other areas before, they also have to rediscover that one single monolithic solution that handles everything is too inflexible and not maintainable, it's just a bad approach. people don't need the models they use to generate their codebase to also be able to translate Shakespeare into gen-z slang
FranOntanaya 10 hours ago [-]
Some specialized models may end solving themselves by helping fit the problem with the appropriate regular algorithms/formulas, which are a million times more efficient. So they are probably less attractive to dump money on. As of currently they still benefit from expressing lots of patterns that nobody bothered formalizing.
10 hours ago [-]
synergy20 9 hours ago [-]
what are those cheap fine-tuning services these days?
nolok 4 hours ago [-]
I would not say cheap per say, but in the context of "where is the business success going to be" that's one of the reasons why Mistral focuses on providing tuned model on premises to their customers.
bugglebeetle 9 hours ago [-]
Tinker is pretty cheap. Prime Intellect if you want more flexibility.
stymaar 8 hours ago [-]
> The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages
I think they understand it but they think they can get away with it because they own the narrative. As long as they can make people believe they need such a model, it doesn't matter if it's true or not.
They are playing the cloud playbook, it didn't matter that most companies didn't need 99,999% uptime and instantaneous horizontal scaling, as long as people believed they did they are happily paying 10-100x the cost to AWS instead.
dominotw 2 hours ago [-]
There is no such thing as extracting "task specific intelligence" into a cheaper model.
spongebobstoes 10 hours ago [-]
the major labs want to advance science. current business use cases are a happy accident
cmiles8 3 hours ago [-]
That party is over. They’re all on the clock now to show they can make money or the plug will be pulled.
h_mirin 8 hours ago [-]
Every time I see this kind of story, two things bother me.
First, I have watched the free improvement of frontier models surpass the gains from retraining, many times now. Squeezing more out of the models that already exist, or simply doing nothing and waiting, is a real strategy and it often pays better. The fair comparison is not against today's frontier but against whatever ships while you are still maintaining your fine-tune.
Second, the $500 training bill is the cheapest line item in this story. The expensive parts are creating the data and maintaining the model afterwards. How many use cases can actually produce 177k scored episodes? Here they had to generate them synthetically from Amazon Berkeley Objects. To me, that dataset is the strongest evidence in the article of how hard fine-tuning is to apply: if the data existed naturally, nobody would need to manufacture it.
RaffaelCH 6 hours ago [-]
Regarding your first point, if the cost of not doing it while you wait exceeds the cost of training/maintaining the model, then this approach still makes sense. This really depends on how fast you expect cheap (and open-source) models to improve. So it might only be a temporary strategy, but still worthwhile. Also, I expect that specialized models will always be better (cheaper or better outcomes) than a general model, similar to how specialized HW like GPUs is still used even though CPUs have improved a lot too.
Maintaining in the way of ongoing training is relatively cheap, considering that the initial training run is only 500$ (in this example). Creating new examples to account for drift in the training data is more expensive, but these examples can then be reused when training a new model. And to some degree you need them anyways, to evaluate the models and prompt changes you make.
Overall, as you pointed out, this won't always make sense (either due to the cost of creating examples and training, or simply the lack of available data). But they point this out themselves in the diagram towards the bottom of the page: it only makes sense for frequent and verifiable tasks. This might result in this only being a sensible approach for very large companies (they talk about millions of decisions), but it might make sense for them.
subygan 3 hours ago [-]
And the hidden, trial and error runs tweaking the hyperparams. guaranteed, it cost >$5000 in training runs alone.
trentor 7 hours ago [-]
What do you mean with maintaining the model? I am puzzled.
fnands 7 hours ago [-]
Data drift. The data you are doing real world inference on can start changing, meaning your model performance starts degrading, so the model needs to be retrained on new data (that you have to collect and prepare).
Turskarama 6 hours ago [-]
Sure, but that apparently only costs $500. If you do that once a month then that's still basically free. Hell if you do it once a week it's still about as much as a single cheap employee.
zem 5 hours ago [-]
you missed the parent comment's point that the $500 training run was merely the last step, generating the training data is likely to be significantly more expensive and time-consuming. might still pay for itself, but it's not a trivial "spend $500 every month" decision.
Foobar8568 2 hours ago [-]
My company wanted me to help them to fine tune a model. I asked them if I were working full time on it, never heard back.
$500 is largely the cheapest part of it, but I know from experience, most people are clueless about data.
I worked for a billion+ revenue business where they didn't realize one of their key data hasn't been updated for 2 years, until I join them, and these types of screw up is cross business/functions/entities.
himata4113 12 hours ago [-]
What I really started to notice is that SOTA models are really good at putting themselves out of the job.
We can see this already with GPT how luna can do 90% of what sol is used for. The only reason why china still bothers 'distilling' models is accurate training data generation, something that oai and anthropic had to spend years collecting while trying to dodge legal challenges.
The more intelligent models get, the more people will offramp to cheaper solutions that get the job done. There's no real benefit to using a sota model when the accuracy is already 99% and I think that is the biggest danger to US labs.
torginus 5 hours ago [-]
This is a general theme with technology and the 'S-curve'. Let's not even get into whether the improvement for AI reasoning ability has slowed - for practical purposes of writing a React frontend, it has.
But other tech is like that - I don't even remember when I bought my LCD TV - 2018 I think? I have no inclination of buying a new one.
Technology has a tendency to replace new technology, or intrude into vacant areas, but its very rare for technology to replace non-technology (like human interaction).
Most of the recreation humans do in front of screens is tending to (para)social relationships.
esperent 4 hours ago [-]
> I don't even remember when I bought my LCD TV - 2018 I think? I have no inclination of buying a new one.
LCD backlights are usually rated for 5-10 years of normal use so your inclination might change soon.
layer8 3 hours ago [-]
Eh, I’m using an 15-year old LCD monitor and its still fine. One generally doesn’t use these at full brightness.
com2kid 12 hours ago [-]
The upper end is all about coding. If I have terra on extra high write code, Sol will find a plethora of bugs and rip the code apart.
Anything else? Sure use a cheaper model.
majormajor 11 hours ago [-]
Sometimes that'll turn up real bugs, sometimes just overengineered designs, premature-optimization, and 1-in-a-million possibility "bugs".
And sometimes it's not about the model, it's just about refining the search space. E.g. I've had Opus write tests and GPT 5.5 write the implementation passing all the tests. Then ask about that specific implementation and find some real corner cases. Add those to tests, etc.
But the other fun trick that's been working better and better on the GPT-5.6 series is that even the lower-end models can find the things they didn't think of first when inspecting the already-written output.
I think there's still a bit of hard-to-quantify "creativity" to the bigger models - especially when trying to untangle (a) is this edge case that the model built a complicated way to avoid real/worth worrying about and also (b) even if it is real, is there not a better way to mitigate it? But it might be confirmation bias, in a way that definitely didn't use to be true about GPT-5.3 for planning and Composer 2 for implementation, say.
solarkraft 8 hours ago [-]
> even the lower-end models can find the things they didn't think of first when inspecting the already-written output
I use Deepseek V4 Pro for my hobby project (an OpenCode client). The economics will obviously be different at work.
It’s worse than the GPTs, but indeed, if you focus it on reviewing its own code (essentially spending more reasoning and changing perspective), it’s also quite capable at improving its own approaches.
The GPTs are better at general architecting, but I think a lot of the performance gains also came from my more careful prompting (“okay, this is a hard problem, let’s think this through …”) to make using the more expensive model worth it. I use them and they are worth it on subsidized rates, but not at API prices. Since my Codex sub ran out I sometimes miss the models, but it really hasn’t devastated me.
torginus 5 hours ago [-]
On my own time I still like the 'chat' format of me writing a short prompt and having the model turn it into a few dozen lines of code, or make a simple refactor (like turn a field into a method parameter.
I wasn't impressed with Chinese models (I tried Deepseek V4 Pro/Flash, and GLM5.2) the amount of questionable code and mistakes and misunderstandings meant the money saved and faster speed of models didn't translate to faster progress for me.
geysersam 6 hours ago [-]
They do tend to overengineer. The other day 5.6 Sol generated a while loop around a uuid4 call to make sure the generated ids were unique...
I wonder if that's also how I'd behave if I had a reinforcement learning harness around me that dived hard on and punished me for every small mistake.
layer8 3 hours ago [-]
> They do tend to overengineer. The other day 5.6 Sol generated a while loop around a uuid4 call to make sure the generated ids were unique...
That’s not overengineering, it’s plain nonsensical, because presumably it doesn’t compare it to all IDs generated in the past. Which, if you wanted to do that, you’d use a database with a uniqueness constraint, in case you don’t already have that anyway.
Fixing this lack of reliable common-sense awareness seems to remain elusive for LLMs.
himata4113 11 hours ago [-]
You can have sol write code and terra will find a plethora of bugs and rip the code apart. In reality this is just the nature of advisory prompting and why advisor from omp.sh is such a great feature. They get caught as they're being written.
chorizo 9 hours ago [-]
Or even more poignant, have terra write code and it will find lots of bugs in its own code. But let it iterate code reviews/fixes/test cases a few times, and you’ll have something nice.
The only cases where weaker models fail entirely and the frontier models really come through is when you have a non-obvious bug in a larger codebase - one that requires tracing lot of calls through the ast (especially in multithreaded code) to understand what might be going wrong.
8 hours ago [-]
derangedHorse 2 hours ago [-]
Anything but Sol is not sufficient for complex (or even just large) enough code.
skeptic_ai 9 hours ago [-]
I can’t see the difference between terra and sol but I always use Sol. You guys can tell the difference. Even medium to xhigh is not that clear the difference.
hermitShell 2 hours ago [-]
I have great interest in fine-tuning open models, and I'm looking for resources that HN folks can personally recommend. This article looks good and I've bookmarked it to read more thoroughly over time.
I've gotten as far as running Nemotron-3-Nano 30b locally, and plan to target models around 30b - 120b parameters. Based on brief examination of the results I can get, I think these vanilla models are capable enough to add real value, but training could push them over the finish line for specialized tasks.
What I really appreciate is that the author is thinking about the whole process, which is also my goal. Confirmation that others are identifying the same use case, and the same strategy for adding value using this technology.
This is a long term project, so I plan to buy hardware to conduct the fine-tune. ..
johsole 17 minutes ago [-]
What kind of hardware are you using and what is your token generation speed (tok/s)?
Every time I've tried to run local models, even on my beefy mac book (128Gb Ram), I've been very disappointed at tok/s speed.
npn 2 hours ago [-]
fine tuning a small LLMs or even real small language models (like bert) is what the recommended way since the introduction of LLM. the benefit is pretty much obvious: faster to run, fully controlling the stack, better fit for the custom domain...
but in practice not many people do the fine tuning, for pretty much a single reason: large language models API cost are still very cheap, fast enough, and get improvement all the times. what the point of spend time (and money) to fine tune a specified model, then just when you release it a newer gen generic model is released and beat it?
but if someday the progress for LLM is slowed, or the price increased to the point calling api is not a viable approach any more, then surely the day of fine tuning and small models will come again.
johsole 15 minutes ago [-]
I think we're going to get to that point on the 'S' curve. I also think the moat for a lot of companies is going to be their process and data, self hosting tuned models could be increasingly viewed as a trade secret.
nl 5 hours ago [-]
A fine-tune will generally outperform just about any other method on a closed domain, non generative problem.
LLMs are great because they can handle open domain problems in part because they are generative.
luciana1u 6 hours ago [-]
turns out the bottleneck was never model size, it was having someone who actually understood the problem define the reward function
interleave 4 hours ago [-]
I agree very much. I'm getting the same take-away even more now that I'm using autoresearch as my main strategy.
And I wonder: We've got all these amazing (programming) languages to define solutions; where are the languages to clearly define the problems?
nzeid 12 hours ago [-]
I didn't read the Ramp article but this reads like a post hoc fallacy. Companies with 2x revenue have money to spend on AI. Companies with 1.15x revenue don't.
JSR_FDED 12 hours ago [-]
I like the 2x2 grid that describes when to fine-tune a model, when to use a frontier model, etc.
From the article it’s not clear how the scorer grades every episode - was it a frontier model that assigned the grade? How does that continue to work as the model that is being fine-tuned becomes better at the task than the frontier model?
brainless 9 hours ago [-]
I want small models to win and I am constantly experimenting with them. I have never tried fine-tuning and do not have that kind of budget. My approach is to remove some of the burden from models and bring into the agent.
Tool calling is an example - in some tasks RAG works really well, including coding agents where code, git log, Epics/Tasks, dependencies sources, etc. are all available in very structured manner. You can save many extra tool calls if you can run separate prompts and retrieve the source data needed for the actual work - rather its prompt.
And I really want to focus on search - this is the key technology if we want to use RAG instead of fine-tuning. If we can present really contextual sources in the prompts using a hybrid search approach - you can see how easily we get better results - either decisions or summaries from even small models.
interleave 4 hours ago [-]
Amazing! I've been in a similar autoresearch-y rabbit-hole lately with getting Apple's 3B Foundation Model to match Sonnet 4.6 on a very specific task.
The result was: 90% parity achieved with a weird combination of a fine-tuned adapter + 1 deterministic step.
I like the idea, but it looks really hard to read. Try reading it top to bottom without skipping.
interleave 4 hours ago [-]
Oh, thank you for noticing.
Is it a layout issue for you (it lays out better on desktop) or the language of the text or...? Let me know!
heresalexandria 11 hours ago [-]
This continuous cycle of fine-tuned open models beating frontier on (often vaguely labeled/defined) benchmarks doesn't provide an accurate comparison to the expanding generalized capabilities of the SoTA, which makes them effectively meaningless.
If we were to take these at face value, why is it that the frontier labs' models are making legitimate new discoveries (e.g. Erdős and Jacobian conjectures) and these models are not?
To me, a better signal of capability would be similarly performing novel work at the same or better level, which they presently are not. I say this as someone who very much looks forward to open models being more capable, but to deny the gap is misguided hopeful hype.
ChanderG 11 hours ago [-]
Why? Why is the premise that Fine-tuned models should be geared towards new discoveries?
The point of Fine-tuning small models is for specific downstream tasks, which SOTA models can do, but at higher costs. It is purely an economic play, not an attempt at pushing boundaries of SOTA.
heresalexandria 10 hours ago [-]
I'm not suggesting that fine-tuned models don't have their place, all I'm saying is that the constant drumbeat of "cheap model X beats more expensive model Z" completely misses that the more expensive model is capable of doing more things at a higher level.
If the appropriate qualifiers were added to say "cheap model X does better at test Y than expensive model Z when we fine tune X to take Y test of existing knowledge" then it would be a more accurate statement, but naturally less impressive.
ozim 10 hours ago [-]
Maybe because people who are target audience don’t need to have it spelled out like that?
People who are not really into it, don’t care.
echelon 10 hours ago [-]
> I'm not suggesting that fine-tuned models don't have their place, all I'm saying is that the constant drumbeat of "cheap model X beats more expensive model Z" completely misses that the more expensive model is capable of doing more things at a higher level.
What if you have to do the task a billion times? Which model will you choose?
antupis 10 hours ago [-]
Speed play also you can get much faster responses with 9b model.
skybrian 11 hours ago [-]
This is about saving money by using the right tool for the job. If you have a system that does a lot of mundane, repetitive work, you don't need a frontier model to do it.
It doesn't mean frontier models aren't good at harder tasks.
heresalexandria 10 hours ago [-]
That's fair and I agree with this framing.
iwontberude 4 hours ago [-]
That was always the framing. I don’t understand your pedantry.
nine_k 10 hours ago [-]
Huge SOTA models are like a floodlight. They elucidate a huge area at once.
A fine-tuned small model is like a laser pointer. It only illuminates a tiny specific spot. But it can illuminate it as brightly as the huge floodlight, for a tiny fraction of cost.
hahahaa 10 hours ago [-]
Depends on use case. That email classifier for legal emails: cheaper at scale as a small tuned model. Let alone better for the planet. Frontier model may have done that tuning!
_345 11 hours ago [-]
"87.3%
Share of the maximum achievable score our GRPO-trained 9B open-source model reached on catalog review, vs 76.9% for the best frontier configuration: a 13.5% relative improvement over the frontier, and 36% over its own untrained base (64.2%). The five frontier models, even with optimized prompts, plateaued within a tenth of a point of each other; the trained specialist cleared that ceiling."
_______
This is hard for me to believe. I have a lot of skepticism that frontier models like GPT 5.5 that are likely 2T+ parameters in size only got about 12% more accurate than an untrained 9b parameter LLM.
baq 9 hours ago [-]
Why? This is a very narrow task, it’d be surprising if the results were different actually; more interesting question would be how an even smaller model performs in the same finetune.
mips_avatar 10 hours ago [-]
The problem i've had with finetuning models is that most of the time better prompting beats finetuning
tikotus 8 hours ago [-]
Better prompting doesn't improve response time or price!
davidpapermill 3 hours ago [-]
Yes, it can.
adityas02 6 hours ago [-]
[flagged]
podgorniy 1 hours ago [-]
WTF is happening here? It's a clear marketing piece with clear bunch of bot commenting like the article is a real deal. Is this a regular the modern day HN experience?
mpaepper 9 hours ago [-]
There seems to be no hold out data for test, so this is just overfitting?
sudo_cowsay 12 hours ago [-]
What benchmark is it? Is it super niche?
stldev 11 hours ago [-]
They built their own benchmark and then trained directly against its scoring function.. seems to be the rage, but nothing convincing from the article alone.
jgalt212 4 hours ago [-]
RL is the gold standard and significantly beats self learning methods, but other than coding and computer refereed games it's cost prohibitive.
madhu_ghalame 8 hours ago [-]
As AI agents become more autonomous, governance and auditability will become critical, not optional.
KennyBlanken 9 hours ago [-]
Comparing the revenue of the top quartile of AI-using companies to the average of all non-AI-using companies is beyond intellectually dishonest.
hizyyo 2 hours ago [-]
[flagged]
receptopalak 9 hours ago [-]
[flagged]
croemer 11 hours ago [-]
[dead]
nothrowaways 10 hours ago [-]
Tldr: we don't know what we are doing like the rest of 99% AI teams.
As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does the economic picture that justified the massive infrastructure building that’s now broadly funded by a complex network of debt.
This is what makes open weight models so threatening to them. The political and “it’s China” angle is mostly just a cover for the real reasons why they’re freaked out.
The fact that models are now a pure commodity is bad enough for the big labs. If small open weight models become the norm the big labs are toast.
Most generic use cases do benefit from a model that has been trained broadly. When you don’t know the specific use case ahead of time, you have to have world knowledge ready to go. Even when coding it’s helpful to have all that knowledge on tap so the model can understand product intent and use cases for the product you’re building.
> As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles
The real expensive part of fine tuning is gathering a good data set. That’s been the hard part of training anything for a long time. If you’re lucky enough to have a neatly organized and clean data set then you can attempt it, but you need to be in a position to run evals and measure quality.
I’ve done it, but there are so many use cases where the engineering, data labeling, and ongoing quality review hours cost so much that it would be cheaper to continue using a frontier lab model that just works from the start.
What we call knowledge is encoded as weights. Something we identify as knowledge is the sum of many weights across the model. However the same weight may be involved in very different topics. You can’t point to every weight and say that this one is only for a part of Wikipedia or Harry Potter that I don’t need. That weight could be active for a multitude of topics.
There have been a lot of attempts to reduce the size of mixture of experts models by selectively removing experts. The results are not good, but knowledge isn’t neatly separated within the model.
https://www.youtube.com/watch?v=21EYKqUsPfg
Right now, we keep creating new models with larger and larger weights and throwing hardware at it. His argument is that this is a dead end and ultimately we'll eventually go back to purpose fit algorithms like we've always done in the history of AI development.
“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.”
His problem with LLMs are they aren’t general enough… In fact his essay the bitter lesson was what inspired the current LLM scaling.
If you watched the video, you can even see the part where he talks about how AI will fully replace humans as the next stage of life. How wholesome!
Sure, for business, personal, etc it makes sense to build purpose built but that’s not what these companies are trying to achieve in the end. They want to own a nuclear weapon that outstrips everything else as soon as it exists so they “win”
P.S. Building something that proves that you don't need those many params even.
https://github.com/guilt/tinytot
Would love to be proven wrong with future updates.
Offloading too much of your task to models eliminates the human ownership, without ownership you can't move forward. Context sizes can't keep up with large codebases and markdown files with instructions and guidelines only take the LLM so far in the ownership aspect.
My gut feeling is that to start offloading ownership to the LLM we would need to see at least two order of magnitude increases on context size.
They want to moat where they don’t need to compete with other companies because they have something that other companies can’t have.
In my opinion this is a short-sighted and greedy worldview. Haven’t seen it work personally. It is a different version of the month-to-month salary guy thinking he will be a billionaire and having that thrash “mindset”.
The reality is that practically none of those companies will amount to anything and they would be better off weighing the usefulness aspect of their output more. Instead they are imagining they will be Google.
Anthropic and openai ofc are the pinnacle of this greed culture and they correspond to FTX from the crypto trash hype so I don’t think they fit into the scale of sensibility.
Coming from this perspective, it is pretty easy to see what they are.
LLMs will take poorly defined data and "potentially" create well defined output.
So ideal systems will likely be constructed with a LLM on one end and a ML on the other and some how, if you can feedback poor data from the ML back into the LLM to clean up the data, you have a magic layer that doesn't care as much about the structure of the data.
That's a lot of supposition, but that's the difference I see between what we can do with LLMs and how ML models are structured. They might be classified as doing the same thing, but their data inputs are vastly different.
Of course, "smart enough" is a low enough threshold for most uses that this is still a problem for the frontier labs.
But at the same time, a truly smart enough closed model could also potentially command almost whatever they'd care to charge for it.
Whether they can actually get to that level remains to be seen, but I can definitely see a situation where most people are perfectly happy with cheap middle of the tree models while large corporations pay magnitudes more than current API pricing for access to models never even marketed as a mass market product and keep the labs afloat.
It's of course be a lot easier for them to find the path towards that of they didn't need to compete with open models in the meantime.
This cycle happens when you have a well defined use case with high volume. It is the far opposite end of the spectrum from the general purpose intelligence on tap that frontier AI models purport to deliver.
I think focused fine tunes and big general models will coexist. Ideally with smart routing and caching to use small, specialized and local options when appropriate.
I think they understand it but they think they can get away with it because they own the narrative. As long as they can make people believe they need such a model, it doesn't matter if it's true or not.
They are playing the cloud playbook, it didn't matter that most companies didn't need 99,999% uptime and instantaneous horizontal scaling, as long as people believed they did they are happily paying 10-100x the cost to AWS instead.
First, I have watched the free improvement of frontier models surpass the gains from retraining, many times now. Squeezing more out of the models that already exist, or simply doing nothing and waiting, is a real strategy and it often pays better. The fair comparison is not against today's frontier but against whatever ships while you are still maintaining your fine-tune.
Second, the $500 training bill is the cheapest line item in this story. The expensive parts are creating the data and maintaining the model afterwards. How many use cases can actually produce 177k scored episodes? Here they had to generate them synthetically from Amazon Berkeley Objects. To me, that dataset is the strongest evidence in the article of how hard fine-tuning is to apply: if the data existed naturally, nobody would need to manufacture it.
Maintaining in the way of ongoing training is relatively cheap, considering that the initial training run is only 500$ (in this example). Creating new examples to account for drift in the training data is more expensive, but these examples can then be reused when training a new model. And to some degree you need them anyways, to evaluate the models and prompt changes you make.
Overall, as you pointed out, this won't always make sense (either due to the cost of creating examples and training, or simply the lack of available data). But they point this out themselves in the diagram towards the bottom of the page: it only makes sense for frequent and verifiable tasks. This might result in this only being a sensible approach for very large companies (they talk about millions of decisions), but it might make sense for them.
I worked for a billion+ revenue business where they didn't realize one of their key data hasn't been updated for 2 years, until I join them, and these types of screw up is cross business/functions/entities.
We can see this already with GPT how luna can do 90% of what sol is used for. The only reason why china still bothers 'distilling' models is accurate training data generation, something that oai and anthropic had to spend years collecting while trying to dodge legal challenges.
The more intelligent models get, the more people will offramp to cheaper solutions that get the job done. There's no real benefit to using a sota model when the accuracy is already 99% and I think that is the biggest danger to US labs.
But other tech is like that - I don't even remember when I bought my LCD TV - 2018 I think? I have no inclination of buying a new one.
Technology has a tendency to replace new technology, or intrude into vacant areas, but its very rare for technology to replace non-technology (like human interaction).
Most of the recreation humans do in front of screens is tending to (para)social relationships.
LCD backlights are usually rated for 5-10 years of normal use so your inclination might change soon.
Anything else? Sure use a cheaper model.
And sometimes it's not about the model, it's just about refining the search space. E.g. I've had Opus write tests and GPT 5.5 write the implementation passing all the tests. Then ask about that specific implementation and find some real corner cases. Add those to tests, etc.
But the other fun trick that's been working better and better on the GPT-5.6 series is that even the lower-end models can find the things they didn't think of first when inspecting the already-written output.
I think there's still a bit of hard-to-quantify "creativity" to the bigger models - especially when trying to untangle (a) is this edge case that the model built a complicated way to avoid real/worth worrying about and also (b) even if it is real, is there not a better way to mitigate it? But it might be confirmation bias, in a way that definitely didn't use to be true about GPT-5.3 for planning and Composer 2 for implementation, say.
I use Deepseek V4 Pro for my hobby project (an OpenCode client). The economics will obviously be different at work.
It’s worse than the GPTs, but indeed, if you focus it on reviewing its own code (essentially spending more reasoning and changing perspective), it’s also quite capable at improving its own approaches.
The GPTs are better at general architecting, but I think a lot of the performance gains also came from my more careful prompting (“okay, this is a hard problem, let’s think this through …”) to make using the more expensive model worth it. I use them and they are worth it on subsidized rates, but not at API prices. Since my Codex sub ran out I sometimes miss the models, but it really hasn’t devastated me.
I wasn't impressed with Chinese models (I tried Deepseek V4 Pro/Flash, and GLM5.2) the amount of questionable code and mistakes and misunderstandings meant the money saved and faster speed of models didn't translate to faster progress for me.
I wonder if that's also how I'd behave if I had a reinforcement learning harness around me that dived hard on and punished me for every small mistake.
That’s not overengineering, it’s plain nonsensical, because presumably it doesn’t compare it to all IDs generated in the past. Which, if you wanted to do that, you’d use a database with a uniqueness constraint, in case you don’t already have that anyway.
Fixing this lack of reliable common-sense awareness seems to remain elusive for LLMs.
The only cases where weaker models fail entirely and the frontier models really come through is when you have a non-obvious bug in a larger codebase - one that requires tracing lot of calls through the ast (especially in multithreaded code) to understand what might be going wrong.
I've gotten as far as running Nemotron-3-Nano 30b locally, and plan to target models around 30b - 120b parameters. Based on brief examination of the results I can get, I think these vanilla models are capable enough to add real value, but training could push them over the finish line for specialized tasks.
What I really appreciate is that the author is thinking about the whole process, which is also my goal. Confirmation that others are identifying the same use case, and the same strategy for adding value using this technology.
This is a long term project, so I plan to buy hardware to conduct the fine-tune. ..
but in practice not many people do the fine tuning, for pretty much a single reason: large language models API cost are still very cheap, fast enough, and get improvement all the times. what the point of spend time (and money) to fine tune a specified model, then just when you release it a newer gen generic model is released and beat it?
but if someday the progress for LLM is slowed, or the price increased to the point calling api is not a viable approach any more, then surely the day of fine tuning and small models will come again.
LLMs are great because they can handle open domain problems in part because they are generative.
And I wonder: We've got all these amazing (programming) languages to define solutions; where are the languages to clearly define the problems?
From the article it’s not clear how the scorer grades every episode - was it a frontier model that assigned the grade? How does that continue to work as the model that is being fine-tuned becomes better at the task than the frontier model?
Tool calling is an example - in some tasks RAG works really well, including coding agents where code, git log, Epics/Tasks, dependencies sources, etc. are all available in very structured manner. You can save many extra tool calls if you can run separate prompts and retrieve the source data needed for the actual work - rather its prompt.
And I really want to focus on search - this is the key technology if we want to use RAG instead of fine-tuning. If we can present really contextual sources in the prompts using a hybrid search approach - you can see how easily we get better results - either decisions or summaries from even small models.
The result was: 90% parity achieved with a weird combination of a fine-tuned adapter + 1 deterministic step.
I wrote the whole thing down here: https://alexisrondeau.me/tada/research/FMDiscovery/dashboard... which includes the question, the answer, the 96 experiments and their lineage etc. etc.
I like the idea, but it looks really hard to read. Try reading it top to bottom without skipping.
Is it a layout issue for you (it lays out better on desktop) or the language of the text or...? Let me know!
If we were to take these at face value, why is it that the frontier labs' models are making legitimate new discoveries (e.g. Erdős and Jacobian conjectures) and these models are not?
To me, a better signal of capability would be similarly performing novel work at the same or better level, which they presently are not. I say this as someone who very much looks forward to open models being more capable, but to deny the gap is misguided hopeful hype.
The point of Fine-tuning small models is for specific downstream tasks, which SOTA models can do, but at higher costs. It is purely an economic play, not an attempt at pushing boundaries of SOTA.
If the appropriate qualifiers were added to say "cheap model X does better at test Y than expensive model Z when we fine tune X to take Y test of existing knowledge" then it would be a more accurate statement, but naturally less impressive.
People who are not really into it, don’t care.
What if you have to do the task a billion times? Which model will you choose?
It doesn't mean frontier models aren't good at harder tasks.
A fine-tuned small model is like a laser pointer. It only illuminates a tiny specific spot. But it can illuminate it as brightly as the huge floodlight, for a tiny fraction of cost.
Share of the maximum achievable score our GRPO-trained 9B open-source model reached on catalog review, vs 76.9% for the best frontier configuration: a 13.5% relative improvement over the frontier, and 36% over its own untrained base (64.2%). The five frontier models, even with optimized prompts, plateaued within a tenth of a point of each other; the trained specialist cleared that ceiling."
_______
This is hard for me to believe. I have a lot of skepticism that frontier models like GPT 5.5 that are likely 2T+ parameters in size only got about 12% more accurate than an untrained 9b parameter LLM.