“It didn’t listen to me.” “It got angry.” “It went rogue.”
These terms are no longer a sailor describing the rough ocean after a calm forecast - we now have had (for years now) fear-based marketing from AI companies that use these phrases to hype “the power” of a probabilistic AI tool.
Contrary to the marketing, the actual product behavior is benign (it’s just semantic search) - it doesn’t work as advertised to automate things (and, as a probabilistic tool, will not be able to do anything better than they do now). Yet, the anthropomorphism of AI ‘beings’ continue to be shared seriously by large companies in the bay area (who have been involved in circular financing deals and accounting irregularities for over 4 years and counting) - for some reason this has alluded most journalists who write on the subject (who do not appear to be thinking as critically about their subjects as past journalism has been).
Large portions of the product is mostly marketing. Even the ‘thinking’ behavior people see when they use these tools is a fake list of tasks programmatically shown to a user to buy time to compile longer result output. It’s semantic search (as stated above), but unlike Google searches of the past, our AI prompts are the company’s proprietary data regardless of the terms and conditions (I also know this from consulting with AI startups ran by 22 year olds who went to school during covid). When I was at Google, by contrast, search history was locked down by layers of security teams.
How did we get here?
We’ve spent ~3-4 years reading absurdist claims on current technology from self-proclaimed experts who have no technical merit. If either the ‘experts’ or the readers of these articles had enough time to take a class like the 3 hours of education I taught 4 years ago to become proficient, their intent becomes opaque.
I finally wrote in October 2025 on where I think the hype explosions are coming from. The best answer afaik is venture capitalists from zero-interest rate loans during covid (same people who invested in crypto and covid software trajectory beliefs) - those loans are due to pension funds and private investors starting in 2026, and the tech industry seems to need increased valuations for their non-operating revenue each quarter.
LLMs obv aren’t “out to get you” - they are software, directed by humans, and (outside of contributing to cognitive atrophy over time and AI psychosis for some) about as malicious as a sudden freak rainstorm when you initially saw a better weather forecast.
The cycles
“Out-of-control AI” claim cycles still spike predictably around funding rounds and valuations, while uncritical journalists amplify them as they are sent to aggregators for clicks. These cycles have been repeated for over 5 years, even with earlier generations of ChatGPT 2 in 2019.
When did the recent hacking stories spike this time?
A 1-2 week period of darkly comical news headlines were released this summer.
Anthropic, OpenAI, Meta and a Chinese company all published (seemingly reactively, with one company reporting the day after the previous) claims of models that were “escaping the sandbox” to hack things, without attribution of the user who directed any test or evidence of the system setup and system prompt directive.
No prompts were disclosed that may have directed the models to do anything, and all were conducted by a marketing company in Israel (outside of US regulation) funded by arms of Effective Altruism.
As some of these stories were disclaimed (mostly by Anthropic, which is going public this year), it wasn’t until the end of August when the ‘OpenAI model hacking Hugging Face story’ became more widely publicized through an additional independent study of the incident by METR (an organization that is funded by arms of Effective Altruism and also Jane Street and other hedge fund entities).
Are these hacking stories legitimate?
Not likely. These hacking claims are flawed for two core reasons:
Hidden prompts - Users configured the systems themselves in the breach being investigated, yet system prompts are never disclosed.
Flawed methodology - METR analyzed 1,300+ contextless API transcripts on-site at OpenAI using $400k in OpenAI credits. They relied on AI to analyze AI, inheriting all its biases (and context window limits) while also leaving themselves open to even more hidden prompts from OpenAI API calls that could have altered their analysis. A truly independent study would have been setup without use of AI analysis and outside of OpenAI’s premises.
While METR recently hired more researchers, their mental health is questionable as the research quality at this organization still manifests in bold claims without scientific grounding or any evidence.
Despite these flaws, mainstream media blindly echoed the claims (alternate link available outside paywall here), extending the echo chamber. Even some AI influencer bros have disregarded these claims as hype, but it continues to be shared by news outlets in increasing channels (and repeated, without any critical thought) - this is a potentially bigger issue in society than the danger the stories are communicating themselves.
To its credit, the New York Times finally alluded yesterday to weaknesses in the sensationalized research conducted by METR, after I’d started the draft of this post.
How closely related are these recent claims?
We don’t know for sure (yet). Before the METR study, the successive ‘breaking out of the sandbox’ stories were conducted by one company called Irregular, based in Israel, and we have no idea what that could signify outside of many (interesting but ungrounded) conspiracy theories.
This has been going on for awhile - when did it get weird?
In parallel to when these hacking stories broke, a hacking story surrounding hedge funds also broke (the same hedge funds investing in these models, and some of the same hedge funds funding METR research), involving social engineering. Since these attempts are common, for all companies across all industries, it was noted at least once on-air by CNBC that it was an odd story to highlight at the time and that someone may want this particular story in the press on that particular day for unknown reasons.
Why are these maybe not coincidental? A few possibilities:
Valuation pumps - proving models are “dangerously smart” pumps valuations and maybe the hedge funds invite “inquiries” that turn into investments.. (?)
Covering losses - hacking narratives could (in theory) explain away losses from circular financing.
Regulatory moats - framing AI as “too dangerous” invites regulation that shuts out other competitors, which again could attract “inquires” converting into investments
This still could all be coincidence, and random, but note my point on it for sometime in the future..
When did these high-level risk headlines become oddly specific?
Within a week or so after that, Ilya Sutskever and Andrej Karpathy proceeded to warn about “rogue” AI agents targeting “neoclouds” (CoreWeave, Nebius) - 2 early investors in AI companies.
See the below tweet (why?):
It’s widely known most startups and enterprises are being vibe coded by unqualified developers that leave a larger array of security holes these days (stats suggest 63% of vibe coding users are not developers but contribute to production code), and security issues from this has become a legitimate issue across many industries and platforms, broadly and not specifically to any one entity.
Since neoclouds took on massive debt for GPU infrastructure that isn’t seeing expected demand, targeting them specifically feels like narrative prep for impending bankruptcies or acquisitions (but whatever it is hinting at seems devastating while misattribution-centric).
Will this ever end soon?
Probably not.
For example, in the last few days a recent "public wiki hijacking" in Germany was blamed on a "rogue AI," rather than the user spamming API calls. Spamming text via an API is just a modern denial-of-service attack. The tool isn't rogue; the user is accountable.
More interestingly (to me), the OpenAI agents in this wiki hijacking story “discussed ways to escape their sandbox”, possibly alluding to the user prompt intent that started the automated model calls (those who have developed in AI are familiar with the user prompt unintentionally showing up in the output), in which the intent could have been a lazy user wanting to find some weakness in the wikis security rather than post spam. This is just a possibility.
The ‘agent escaping discussion’ also could have (in theory) also been intentionally set - an aggressively blatant marketing seeding more public fear in Germany.
Either way, expect more sensational headlines that Sam Altman has suggested are coming, as OpenAI delays its IPO due to valuation concerns. Fear gets clicks and keeps valuations high, even if the underlying technology is just probabilistic software.
Stay tuned..
They can't sell sex, Elon already tried that, so they have no choice but to sell fear.