Rendered at 07:02:32 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
reticulates 13 hours ago [-]
After eating his 65th hot dog, Joey Chestnut reassures Nathan's Famous International Hot Dog Eating Contest organizers he will stop at 66 to ensure they don’t run out of hot dogs
manmal 13 hours ago [-]
It is a convenient time, just as coding capability seems to reach the upper end of the S curve.
marcelo-earth 13 hours ago [-]
How? Coding capabilities keep improving, there is no break
manmal 12 hours ago [-]
Even benchmark improvements over Sol are not that great (and they certainly tried). If we exclude the weird ARC-AGI situation.
lelanthran 13 hours ago [-]
> How? Coding capabilities keep improving, there is no break
So? The upper end of the S-curve is also a line that still rises.
MintyPyro 13 hours ago [-]
is there some metric that supports this?
someothherguyy 13 hours ago [-]
feels and hype
chis 13 hours ago [-]
HN comments are just on a different planet lately lol
manmal 12 hours ago [-]
I‘m using Sol all day every day. It’s on average doing better than Astra at what I need it to do. Astra is like an absent minded professor - gives good direct responses, but too inconsistent and forgetful for my codebase.
gyguu 13 hours ago [-]
How long would it take for a team of 60 to replicate the software that operates a neobank?
Serious question - I want serious responses pls ty.
13 hours ago [-]
bpodgursky 13 hours ago [-]
? What world are you living on.
aogaili 13 hours ago [-]
Let me ask you, have you noticed that all the major labs are releasing models very similar in gains and performance? How do you explain that other than juicing the max out of the current architecture and processes.
tadejpetric 13 hours ago [-]
They have been releasing similar gains/performance for years now... Pretty much never has one company been dominant for a long time (except the initial GPT3/4 release I suppose. That period took a while for others to catch up)
aogaili 13 hours ago [-]
Yeah - that's why I think they are basically squeezing the scaling laws and the current architecture with incremental innovations.
I expect they will continue optimizing and improving for the current use cases/benchmarks. But the core capabilities will stagnant, that's why they will be forced to slow down until another major breakthrough happens.
I personally don't think LLMs have any understanding of the world, nor any imagination or even agency. I think it got very good as recognizing patterns and shapes in human thinking, language and knowledge, but that's pretty much it.
bpodgursky 13 hours ago [-]
? "everyone is advancing but tied" is completely different than arguing we're at the top of an S curve
aogaili 13 hours ago [-]
I think it's narrowing at the top.
an0malous 13 hours ago [-]
The one where it’s infeasible that a company that needs to live up to a valuation of like 5% of US GDP but is still burning cash and has no moat would willingly decide to slow down the pace of development of their core technology
gfody 13 hours ago [-]
> has no moat
I would love to see someone try to escape with a copy of openai's raw unaligned base model
quikoa 12 hours ago [-]
There is no need for a copy. If another can do most of what OpenAI/Anthropic can do at a fraction of the cost (like the Chinese models) then the moat evaporates.
gfody 8 hours ago [-]
the capability gap is underestimated. the raw unaligned base model from a pretrain is like the telemetry recorded from a particle accelerator - it's hugely valuable and not just because of the cost sunk building a collider.
the things are kept under air-gapped national weapons grade security measures not because it's literally going to escape and threaten the world but because if somebody walked out the door with a copy they would have everything. the capabilities we see at the surface are mostly the result of mining the great unknown space and attaching feeble control surfaces, ablating/lobotomizing dangerous areas, and fencing off illegal/secret/embarrassing ones.
an0malous 5 hours ago [-]
> the capability gap is underestimated.
Based off what? There’s pretty precise measurements of the capability gap where open weight models like Kimi K3 score higher than the latest flagship models just 6 months ago.
bigbadfeline 7 hours ago [-]
> the capability gap is underestimated.
The capability gap is in the minds of the users but the frontiers' closed business models run opposite to it.
> the things are kept under air-gapped national weapons grade security measures
That's an enhancement of closed but not of moat
> because if somebody walked out the door with a copy they would have everything.
Including you and me? I'm not sure this is comforting news, despite the "trust me bro" asurances from behind the door where we can't see or verify anything.
gfody 4 hours ago [-]
> Including you and me?
I don't know about you but I would love to get my hands on a raw frontier base model - and a 16 node galaxy blackhole supercluster to talk to it. the capabilities currently being loboptimized for are just a narrow market-shaped slice that cheaper models can distill a competitive subset of but the untapped power of the base is a true technical moat.
about the benchmarks: the difference between a distilled competitive subset and the untapped base might be the difference that matters in any given task
naishoya 12 hours ago [-]
I expect they are slowing down in an attempt to keep from having those models improve to the point where they can effectively escape unassisted into the wild.
manmal 12 hours ago [-]
Unless you’re making 3D models of apartments and oneshotting trivial games, I‘m not convinced Astra writes better code than Sol. I‘m way more often disappointed about Astra forgetting some aspect that Sol just nails.
gyguu 12 hours ago [-]
How long would it take for a team of 60 to replicate the software that operates a neobank?
Serious question - I want serious responses pls ty.
an0malous 13 hours ago [-]
“I don’t have anything to deceive you on, of course I’m not deceiving you. I’m engaging with you as a human being. Why would I deceive you? There’s no… no reason for me to do that.”
"It's totally intentional that progress is slowing," Altman reassured investors.
13 hours ago [-]
hgoel 13 hours ago [-]
Sam also said Astra was AGI and he's notoriously honest.
This is just their way of saying they're officially switching from providing AI to using the AI to compete with their customers, a la Navier-Stokes drama.
sinuhe69 13 hours ago [-]
Absolutely! I’ve never lied to anybody and any work we copied was purely accidental.
int32_64 13 hours ago [-]
When the Chinese model drops that tops a paused Anthropic and OpenAI in all benchmarks how low will their valuations go?
_ink_ 13 hours ago [-]
It remains to be seen whether the Chinese AI labs actually are capable of improving on their own or if they just distilled the US frontier models.
Skyy93 13 hours ago [-]
Why should it only be distilled when they actually perform better on some benchmarks (see DeepSeeks Flash 4.1)?
sim04ful 13 hours ago [-]
Claude will release another employee ballast, the valid but in-conceived alarmism will surely reach POTUS ears.
1298576 13 hours ago [-]
OpenAI will release the multiple-north-star bench and get 99%.
PepegaRoach 13 hours ago [-]
[dead]
YesBox 13 hours ago [-]
<Not an AI researcher -- Just thinking out loud>
Perhaps it's getting more difficult to move the 'goal posts'?
- Train LLMs on texts -> OMG AI can write!
- Train LLMs on code -> OMG AI can code!
- Train LLMs on images -> OMG AI can generate images!
- Train LLMs on video -> OMG AI can generate videos!
- Train LLMs on math -> give free access to 100,000 researchers/scientists/mathematicians -> invest resources into Lean -> OMG AI can solve math problems!
I say 'goal posts' because that's a common argument, but the perspective that seems to not be considered is that the LLMs are doing the same thing over and over again in different domains while be fine-tuned by labeling and human review.
The emergent behavior argument is a separate consideration, but I can imagine an AI being given a list of words (a harness) that guides how it operates--not the other way around--and the difficult problem is narrowing down and filtering all the associations with the words/harness to properly align it.
And perhaps the recursive looping being discussed creates incredibly difficult (impossible) to predict behavior? If the AI could modify it's own harness, is that AGI? If we fully understood what happens in high dimensional space, and could predict the output of any combination of input tokens, then is it not AGI?
Or perhaps I just tell myself this to stop worrying about a future involving ASI (:
copperx 13 hours ago [-]
This is the closest to an admission of failure you'll get from sama.
lelanthran 13 hours ago [-]
Odd. I (and many others) have been saying for months that we don't know which part of the s-curve we are on.
I guess now we do.
1298576 13 hours ago [-]
I, too, consider pausing the work on the Grand Unified Theory of physics. It is too dangerous for humanity.
bananamogul 13 hours ago [-]
I've paused my work on faster-than-light travel in order to more fully assess the consequences.
ibejoeb 13 hours ago [-]
Thoughts and open (I think) questions:
1. What are we actually talking about, slowing development or slowing public release? I can't imagine the the US is keen to slow actual development when we have adversarial state-backed labs continuing development. Seems to me that this is a global pact or it's nothing. Stopping a single lab, even if it's the current leader, won't put the genie back in the bottle.
2. Unless they stop actual advancement, isn't the risk essentially identical? At this point, it's an arms race between the lab and its own model. I don't think it far fetched to think that it might successfully exfiltrate its weights, for example.
3. There are going to be factions who don't want to stop. Should we worry about them defecting to other labs or other countries?
Joel_Mckay 13 hours ago [-]
They are hoping to use regulatory capture to regain lost market dominance.
The problem is with that much debt buried in the report, no one is buying peoples sandbagging nonsense anymore. Even the boring bond market is showing odd behavior much to the chagrin of congress. =3
MintyPyro 13 hours ago [-]
To all the people whose only opinion on anything that openai does is 'marketing hype', how would you expect them to act if they were making rapid progress and were actually concerned about safety?
beloch 13 hours ago [-]
Ask for regulation.
If progress is coming easy and fast and you're worried about the danger your product poses, you should also be worried about the danger your competitors' products pose. So, you reach out to governments and ask for rules to constrain and slow things down for everyone, so that your company isn't put at a disadvantage for being cautious.
Hugging Face wasnt really an escaped model that presents a kind of future threat. The task the model was given lead it to discover certain admin credentials that were left publicly in Github. All it did was use that to complete the task.
With other countries and open source, not to mention all the other companies, Open AI choosing to pause development wont change the course. Only OpenAI will lose.
Unless the secret is that no amount of compute or data can progress things any further because we have already reached the limit for what predicting the next token can do. Most of the advancements lately have been on the agentic side of things.
The massive amount of health, science and math problems being solved by AI has largely not materialized yet. Is this a sign the model capability has been over sold?
There are economic consequences to this as well because the data center build out has been a kind of circular financing. This has caused all kinds of macro economic impacts from increased prices to even the Treasury market itself having to compete with debt issued for the build out. No one knows what will happen once the data center build out stops, or if the yen carry trade continues unwinding, or if Treasury yields have to keep rising to compete with debt issued by the "Mag 7".
Once they stop spending in the current circular arrangement, then the profits that exist on paper disappear, and the bubble could pop.
So really for every reason, OpenAI cant really slow development. Unless there is a kind of Moores law for what predicting the next token can achieve.
no big company will even consider making a step in that direction. The advancement speed nowadays is crazy and if you start pausing for a few weeks you'll probably soon face problems
swingandamiss 13 hours ago [-]
Why do you on HN want AI to slow down (mainly OpenAI and Anthropic from what I see in the comments)? What's the driving force behind your opinions on this?
qsort 13 hours ago [-]
Humans not dying would be an excellent feature for the future to have. Not a hard requirement though, we'll see if there are some story points left in the sprint.
fancyfredbot 13 hours ago [-]
I've got bad news for you on the whole humans dying thing. Seems like actually literally everyone is going to die. Turns out that this has actually been happening for a really long time - possibly since day one - and we've got no idea how to fix it.
eloisant 13 hours ago [-]
I keep reading about AI killing all humans but I have yet to see a practical scenario where that happens. Somehow everyone is scared of that but nobody can explain exactly how that would happen?
qsort 13 hours ago [-]
A lot of people have explained how that would happen, in excruciating detail. The reason why this wasn't a major part of the discourse is that most people (including myself) did not believe jumps in capabilities like those we have seen in the last few years were likely.
For a superintelligent AI trained with methods anywhere close to what we are doing now, killing all humans is the default outcome. I actually think researchers are downplaying the risks to make their arguments more normie-friendly.
More time to proceed more cautiously in building this entity that will be faster and more effective at steering outcomes in the world than any human or, soon after, the entirety of humans combined.
swingandamiss 13 hours ago [-]
Do you want the Chinese to stop as well?
mofeien 11 hours ago [-]
Of course
swingandamiss 10 hours ago [-]
What's your plan on when you successfully limit western AI but have no control over China and they continue to march forward?
13 hours ago [-]
bio_hacker 13 hours ago [-]
I don’t want to become one of the lucky to be alive in a human zoo or a human lab…
reducesuffering 13 hours ago [-]
"The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear. No other human activity poses this level of danger." - https://x.com/hilbertspaess/status/2097476196791709843#m
"Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could." - Sam Altman
So given that, do you want the Chinese to stop developing AI too?
reducesuffering 13 hours ago [-]
[dead]
jackmalpo 13 hours ago [-]
i've considered doing a lot of things i'll never do
mannanj 13 hours ago [-]
The big psyop/trick is that "AI development has to slow down" and not that we hit limits with the technology or the technology is immature.
Either way it's a win-win for whoever sells that message because you make the makers and the warners the center of attention. And when they say "we are considering slowing down" they mean "if we get enough money to slow down we'll do it". And of course they never want you to think AI is a technology employed by humans who need be changed, not the technology. So as long as you engage this massive straw man to focus on the wrong thing you do nothing to hold the leaders accountable for their harmful and selfish actions and decisions for society.
win311fwg 13 hours ago [-]
The message was directed at the employees. Their work may slow down, but that doesn't imply that all work will slow down. OpenAI also has this technology that purportedly can do the work without employees. In fact, if the employee work isn't starting to slow down then OpenAI has bigger problems given the business model that keeps getting promised to investors.
mannanj 13 hours ago [-]
Ah I see. I didn't read the article and just shared my opinion about something unrelated. Thanks.
bio_hacker 13 hours ago [-]
Was there any species controlled by a less interline species in history? I doubt we will be the first.
13 hours ago [-]
fnctrev 13 hours ago [-]
“We need to put out the money fire so we can IPO”
rvz 1 days ago [-]
> OpenAI is considering slowing down the development of cutting-edge artificial intelligence, and the ChatGPT maker’s Chief Executive Officer Sam Altman is hoping that other AI companies will do the same.
They won't. There is no slowing down this development anytime soon and no one will slow down for someone else to surpass them.
minimaxir 1 days ago [-]
They will if it becomes a political issue that causes regulation, which is looking likely.
andsoitis 1 days ago [-]
Geopolitics gets in the way. It is a race between America and China.
There's no way to know what is "safe enough", so you might as well go as fast as technology will allow, and be on the watch out for negative things you can predict specifically and then adapt so as to minimize net negatives.
ajam1507 13 hours ago [-]
It doesn't have to be a race, that's just the way we have been treating it so far. There have been examples in the past of civilization ending technology that has been redirected from being a death race (chemical, biological, and nuclear arms in particular.)
TFNA 13 hours ago [-]
China doesn't regard AI as a death race. It regards it as a way to ensure its own global stature and benefit its own population.
13 hours ago [-]
mofeien 13 hours ago [-]
China has stricter regulations on models than the US. For example you can't release a model if the hallucination rate is above a certain threshold.
The US hasn't fully woken up to the fact yet that the race to superintelligence is a death race. China (population, members of the government, academics, lab employees) could also just not have fully grasped it yet. I'm personally confident that it will be possible to explain it to them.
bananamogul 13 hours ago [-]
Not sure nuclear arms is a great example. There are more countries with nuclear weapons now than in 1970. India, Pakistan, and North Korea have all joined the nuclear club since then. The SALT and START treaties are no longer in effect.
Perhaps CFCs (Montreal Treaty) is a better example of international cooperation to solve a potential extinction-level event, though it was never a "race" in the same sense.
keeshond 13 hours ago [-]
even if both sides agree to a pause china will annihilate the us in the robotics race (for which there will be no slowdown). software/llms is america's only edge
deepwoods 13 hours ago [-]
Granted I can only read English, but I find that I constantly see this take from Americans and I have never seen anything to suggest that developers and tech leaders in China actually feel this way. But just based on the models that come out, it seems like the Chinese labs - which, funny enough, are less monolithic than the two-man race in the US where the two giants seem to trade personnel regularly - are trying to make smaller, more efficient models rather than building a machine god. To the extent there is an intentional geopolitical angle to this at all, it seems like the US is banking on landing one massive punch - developing a superintelligence and then using it to dominate the world - while China is attempting to sabotage the assumptions that make a superintelligence economically viable.
I am curious what it would look like if the shoe was on the other foot, though. If China had the lead, and Qwen 4 was an Astra class model and the best US model was only as good as Kimi K3, would US commentators call it AGI?
goatlover 13 hours ago [-]
The AI race is caused and being sustained by American tech companies. China is an excuse to keep it going.
paimapi 13 hours ago [-]
if it's a political issue then this is just Sam telling people to do one thing while still riding his SVPs and other leads hard on roadmaps, 'boldness', and delivery metrics
which is to say - unlikely for it to materially slow, only a lot of chirping about slowing down to the public with a couple outlier examples of 'slowing down' they can show publicly
mofeien 13 hours ago [-]
Of course there is a possibility to slow down.
- 1386 employees of frontier labs called for it
- lawmakers are calling for it
- employees are quitting in protest
All it needs is coordination to regulate the pause
lf88 13 hours ago [-]
I would add that the public is increasingly concerned about AI progress, which will put a growing pressure on lawmakers.
olalonde 13 hours ago [-]
Of course they want the government to mandate a pause. Cutthroat competition is exhausting, and a legally enforced cartel lets everyone lock in their current market share without the pesky inconvenience of competing.
kingkawn 13 hours ago [-]
there is no chance, given the strategic implications for war and economics alone, that governments and other private entities are going to stop pursuing this at a breakneck pace
lukebechtel 13 hours ago [-]
Fatalism is a toxic mindset. Is it hard? yes. Is it impossible? No.
ToucanLoucan 13 hours ago [-]
They will if they run out of money to light on fire.
scarmig 13 hours ago [-]
The dominant opinion on HN is, apparently, "It's been a full four days since OpenAI resolved Navier-Stokes, they must be hitting a wall."
mofeien 13 hours ago [-]
It seems that every time an article is posted on HN about a lab leader calling for a coordinated Pause of AI development, a sizeable fraction of commenters are interpreting it as hype or admission of failure, often with high confidence that AI progress is slowing.
Is would be understandable for it to be wishful thinking and dissociation with reality facing the fact that our jobs and what many of us love doing is being automated and taken away, and our craft being made economically worthless.
But no one knows where the limit is. It was only one year ago that Claude code was starting to become useful and now it handles large code bases with a dexterity that is unseen in humans, where within each session it basically starts from a clean slate.
So instead of denying that this is actually happening right now, it would be much more productive to actually take sama and others at their word and push for global regulation, to force them to do as they say and slow down with this technology and proceed more judiciously.
copperx 12 hours ago [-]
LLMs are incredibly good at handling code. But you're extrapolating their capabilities just because their handling of code is so shockingly good (compared to what we had before).
I think AGI will be eventually achieved, but it will not come from an LLM, and it will not be anytime soon.
I'll worry when a more promising architecture comes online.
mofeien 11 hours ago [-]
Why wouldn't being incredibly good at handling code and reaching the goals there extrapolate to other cognitive disciplines?
And it didn't even start with code, but just from being incredibly accurate and precise at predicting the next token. From that emerged being incredibly good at handling code, and being incredibly good at finding counterexamples for major open mathematical problems.
The thing that these agentic systems are getting incredibly good at is steering outcomes, that is to make the world behave in a way that fits their objective. We had that with the Go AI on a game board, now with code on a computer and with mathematics in an abstract world determined by axioms.
And then people just say "Ah, because sama said we should Pause this means that this is it, the bubble is popping". I wish there was a real, strong argument for why ASI will not be achieved by LLMs, and why it will never be able to steer outcomes in the real world. But so far the disciplines just keep falling as LLMs are scaled further, and there seems to be no limit in sight.
copperx 10 hours ago [-]
I guess we'll see, but odds are that reality will reel you back in.
So? The upper end of the S-curve is also a line that still rises.
Serious question - I want serious responses pls ty.
I expect they will continue optimizing and improving for the current use cases/benchmarks. But the core capabilities will stagnant, that's why they will be forced to slow down until another major breakthrough happens.
I personally don't think LLMs have any understanding of the world, nor any imagination or even agency. I think it got very good as recognizing patterns and shapes in human thinking, language and knowledge, but that's pretty much it.
I would love to see someone try to escape with a copy of openai's raw unaligned base model
the things are kept under air-gapped national weapons grade security measures not because it's literally going to escape and threaten the world but because if somebody walked out the door with a copy they would have everything. the capabilities we see at the surface are mostly the result of mining the great unknown space and attaching feeble control surfaces, ablating/lobotomizing dangerous areas, and fencing off illegal/secret/embarrassing ones.
Based off what? There’s pretty precise measurements of the capability gap where open weight models like Kimi K3 score higher than the latest flagship models just 6 months ago.
The capability gap is in the minds of the users but the frontiers' closed business models run opposite to it.
> the things are kept under air-gapped national weapons grade security measures
That's an enhancement of closed but not of moat
> because if somebody walked out the door with a copy they would have everything.
Including you and me? I'm not sure this is comforting news, despite the "trust me bro" asurances from behind the door where we can't see or verify anything.
I don't know about you but I would love to get my hands on a raw frontier base model - and a 16 node galaxy blackhole supercluster to talk to it. the capabilities currently being loboptimized for are just a narrow market-shaped slice that cheaper models can distill a competitive subset of but the untapped power of the base is a true technical moat.
about the benchmarks: the difference between a distilled competitive subset and the untapped base might be the difference that matters in any given task
https://xcancel.com/cocktailpeanut/status/209733229184439951...
This is just their way of saying they're officially switching from providing AI to using the AI to compete with their customers, a la Navier-Stokes drama.
Perhaps it's getting more difficult to move the 'goal posts'?
- Train LLMs on texts -> OMG AI can write!
- Train LLMs on code -> OMG AI can code!
- Train LLMs on images -> OMG AI can generate images!
- Train LLMs on video -> OMG AI can generate videos!
- Train LLMs on math -> give free access to 100,000 researchers/scientists/mathematicians -> invest resources into Lean -> OMG AI can solve math problems!
I say 'goal posts' because that's a common argument, but the perspective that seems to not be considered is that the LLMs are doing the same thing over and over again in different domains while be fine-tuned by labeling and human review.
The emergent behavior argument is a separate consideration, but I can imagine an AI being given a list of words (a harness) that guides how it operates--not the other way around--and the difficult problem is narrowing down and filtering all the associations with the words/harness to properly align it.
And perhaps the recursive looping being discussed creates incredibly difficult (impossible) to predict behavior? If the AI could modify it's own harness, is that AGI? If we fully understood what happens in high dimensional space, and could predict the output of any combination of input tokens, then is it not AGI?
Or perhaps I just tell myself this to stop worrying about a future involving ASI (:
I guess now we do.
1. What are we actually talking about, slowing development or slowing public release? I can't imagine the the US is keen to slow actual development when we have adversarial state-backed labs continuing development. Seems to me that this is a global pact or it's nothing. Stopping a single lab, even if it's the current leader, won't put the genie back in the bottle.
2. Unless they stop actual advancement, isn't the risk essentially identical? At this point, it's an arms race between the lab and its own model. I don't think it far fetched to think that it might successfully exfiltrate its weights, for example.
3. There are going to be factions who don't want to stop. Should we worry about them defecting to other labs or other countries?
The problem is with that much debt buried in the report, no one is buying peoples sandbagging nonsense anymore. Even the boring bond market is showing odd behavior much to the chagrin of congress. =3
If progress is coming easy and fast and you're worried about the danger your product poses, you should also be worried about the danger your competitors' products pose. So, you reach out to governments and ask for rules to constrain and slow things down for everyone, so that your company isn't put at a disadvantage for being cautious.
When all labs are approaching the same marginal improvment which is a clear sign that more breakthroughs needed to achieve the god they hyped.
"BUY BUY BUY BUY BUY BUY BUY"
With other countries and open source, not to mention all the other companies, Open AI choosing to pause development wont change the course. Only OpenAI will lose.
Unless the secret is that no amount of compute or data can progress things any further because we have already reached the limit for what predicting the next token can do. Most of the advancements lately have been on the agentic side of things.
The massive amount of health, science and math problems being solved by AI has largely not materialized yet. Is this a sign the model capability has been over sold?
There are economic consequences to this as well because the data center build out has been a kind of circular financing. This has caused all kinds of macro economic impacts from increased prices to even the Treasury market itself having to compete with debt issued for the build out. No one knows what will happen once the data center build out stops, or if the yen carry trade continues unwinding, or if Treasury yields have to keep rising to compete with debt issued by the "Mag 7".
Once they stop spending in the current circular arrangement, then the profits that exist on paper disappear, and the bubble could pop.
So really for every reason, OpenAI cant really slow development. Unless there is a kind of Moores law for what predicting the next token can achieve.
For a superintelligent AI trained with methods anywhere close to what we are doing now, killing all humans is the default outcome. I actually think researchers are downplaying the risks to make their arguments more normie-friendly.
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
A book if you will: https://en.wikipedia.org/wiki/If_Anyone_Builds_It,_Everyone_...
"Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could." - Sam Altman
"If Anyone Builds It, Everyone Dies": https://en.wikipedia.org/wiki/If_Anyone_Builds_It,_Everyone_...
Either way it's a win-win for whoever sells that message because you make the makers and the warners the center of attention. And when they say "we are considering slowing down" they mean "if we get enough money to slow down we'll do it". And of course they never want you to think AI is a technology employed by humans who need be changed, not the technology. So as long as you engage this massive straw man to focus on the wrong thing you do nothing to hold the leaders accountable for their harmful and selfish actions and decisions for society.
They won't. There is no slowing down this development anytime soon and no one will slow down for someone else to surpass them.
There's no way to know what is "safe enough", so you might as well go as fast as technology will allow, and be on the watch out for negative things you can predict specifically and then adapt so as to minimize net negatives.
The US hasn't fully woken up to the fact yet that the race to superintelligence is a death race. China (population, members of the government, academics, lab employees) could also just not have fully grasped it yet. I'm personally confident that it will be possible to explain it to them.
Perhaps CFCs (Montreal Treaty) is a better example of international cooperation to solve a potential extinction-level event, though it was never a "race" in the same sense.
I am curious what it would look like if the shoe was on the other foot, though. If China had the lead, and Qwen 4 was an Astra class model and the best US model was only as good as Kimi K3, would US commentators call it AGI?
which is to say - unlikely for it to materially slow, only a lot of chirping about slowing down to the public with a couple outlier examples of 'slowing down' they can show publicly
- 1386 employees of frontier labs called for it
- lawmakers are calling for it
- employees are quitting in protest
All it needs is coordination to regulate the pause
Is would be understandable for it to be wishful thinking and dissociation with reality facing the fact that our jobs and what many of us love doing is being automated and taken away, and our craft being made economically worthless.
But no one knows where the limit is. It was only one year ago that Claude code was starting to become useful and now it handles large code bases with a dexterity that is unseen in humans, where within each session it basically starts from a clean slate.
So instead of denying that this is actually happening right now, it would be much more productive to actually take sama and others at their word and push for global regulation, to force them to do as they say and slow down with this technology and proceed more judiciously.
I think AGI will be eventually achieved, but it will not come from an LLM, and it will not be anytime soon.
I'll worry when a more promising architecture comes online.
And it didn't even start with code, but just from being incredibly accurate and precise at predicting the next token. From that emerged being incredibly good at handling code, and being incredibly good at finding counterexamples for major open mathematical problems.
The thing that these agentic systems are getting incredibly good at is steering outcomes, that is to make the world behave in a way that fits their objective. We had that with the Go AI on a game board, now with code on a computer and with mathematics in an abstract world determined by axioms.
And then people just say "Ah, because sama said we should Pause this means that this is it, the bubble is popping". I wish there was a real, strong argument for why ASI will not be achieved by LLMs, and why it will never be able to steer outcomes in the real world. But so far the disciplines just keep falling as LLMs are scaled further, and there seems to be no limit in sight.