When the People Building AI Start Asking for Brakes
We’ve been following the Anthropic story for weeks now. Longer than most AI news cycles stick with us, honestly. Skip the headline part, the trillion dollar valuation, the confidential IPO filing. That’s just a number. Numbers don’t tell you anything.
Here’s what actually grabbed us. Same company chasing that valuation published research this month on something called recursive self-improvement. Plain terms: an AI system gets good enough to make itself smarter, then uses that improvement to get even better at improving itself again. A loop. And loops like that don’t sit still once they start moving. Anthropic says Claude writes more than 80 percent of the code merged into its own production codebase now. Eighty percent. Their own conclusion from that research? The world might need the option to slow down. Maybe pause outright. That’s an odd thing to say about your own product, right when you’re raising money on the strength of that exact product.
Politics moved fast after that. A few days later, the White House issues an export control directive, cuts off foreign nationals, including some of Anthropic’s own non-citizen staff, from two of its newest models. Fable 5. Mythos 5. Anthropic didn’t look for a workaround. They just shut both models down. Completely.
Now here’s where we have to slow down, because the version going around overstates what happened. This wasn’t some model running a cyberattack start to finish on its own. What actually happened was narrower than that, messier too. A jailbreak, non-universal, basically a technique where you ask the model to read code and flag vulnerabilities that were already known elsewhere anyway. Anthropic disputes the government’s framing outright. Says the finding gave no real advantage specific to Mythos. Says the same trick works on other public models, GPT-5.5 included. And there’s reporting, worth sitting with this one, that the jailbreak surfaced through Amazon’s CEO. Amazon. Which is both an investor in Anthropic and a direct competitor in cloud AI. Suddenly this isn’t such a clean story about a government discovering its conscience.
So where do we actually land. The self-improvement research is real, Anthropic’s own numbers back it up. The export control thing is messier, less dramatic than the cyberattack version everyone’s repeating. Both can be true at once. Capability keeps climbing. Risk climbs right along with it. That’s just the curve this whole industry is riding, like it or not.
What surprises us isn’t the curve. It’s how long everyone in government took to even notice the curve was worth acting on. Other than the UK’s 2023 AI Safety Summit, there’s been basically no coordinated global response matching how fast this stuff actually moves. People running these labs have said it themselves, out loud, in public: they expect to build systems smarter than humans, across the board, and there’s a real chance this goes badly. For years the government response looked like enthusiasm dressed up as policy. Subsidies. Fast-tracked permits. Red carpet treatment for anyone dangling AI jobs and investment.
This month might be the first real crack in that pattern. Messy though. Executive orders issued, then quietly walked back. A directive built on a narrow technical finding, not some smoking gun moment. Still. The direction feels real to us, even if the trigger was sloppier than the headlines made it sound.
Most of what we do at Seafund has nothing to do with large language models. Semiconductors, space, defence, robotics, that’s our world. And yet the same pattern shows up there too, every time. Powerful systems built and shipped without real safety oversight eventually force a reckoning. Governments don’t get to choose if that happens. They only get to choose whether they’re ahead of it or cleaning up the mess after.
We’ve heard this privately from people inside frontier labs more than once, and it surfaces publicly sometimes too. The old assumption: real regulation doesn’t happen until a disaster is big enough that doing nothing becomes politically impossible. Maybe, just maybe, this month proves that wrong. Maybe a smaller scare, handled early, even handled clumsily like this one was, gets real rules moving before the bigger disaster shows up.
What we’d actually want here isn’t complicated. A nuclear plant clears a safety bar before it operates. Same for a new airplane design. An elevator. A building. Depending where you live, even the person cutting your hair. Every one of those accepts some baseline safety standard as the cost of being allowed to operate at all. Doesn’t seem unreasonable to hold companies building what they themselves call one of the most consequential technologies in human history to that same bar. At least that.
We’re watching all this from outside the labs. Just investors, looking in. And the takeaway feels pretty clear honestly. Frontier AI has reached a point where governments feel they have no real choice but to step in directly, and that’s starting to look like actual restriction, not the encouragement and funding support that defined AI policy for years. That’s a real marker of where this industry sits right now. We’ll keep weighing it carefully, especially wherever AI exposure touches infrastructure, security, or anything where getting it wrong has real consequences.
FAQs
1. What is recursive self-improvement in AI?
Recursive self-improvement is when an AI system improves its own capabilities, enabling faster and more advanced development with minimal human intervention.
2. Why is AI safety important?
AI safety ensures advanced AI systems are reliable, secure, transparent, and aligned with human goals, especially in critical industries like healthcare, finance, and defence.
3. Why are governments regulating AI?
Governments are introducing AI regulations to address risks related to security, privacy, ethics, and the responsible deployment of advanced AI technologies.
4. What is frontier AI?
Frontier AI refers to the most advanced AI models capable of complex reasoning, coding, research, and automation, developed by leading AI companies.
5. How does AI regulation impact startups?
AI regulations encourage startups to build secure, trustworthy, and compliant AI solutions, improving long-term adoption and market confidence.
6. Why is responsible AI important for deeptech?Responsible AI helps reduce risks, improve transparency, and ensure safe deployment of AI in sectors like finance, defence, healthcare, and critical infrastructure.
7. What role do venture capital firms play in AI innovation?
Deeptech venture capital firms support AI startups with funding, strategic guidance, industry connections, and long-term support to help commercialize innovative technologies.
Table of Content
- 1. What Founder-Market Fit Really Means
- 2. Why Founder-Market Fit Matters More in Deeptech Than SaaS
- 3. Examples of Strong vs. Weak Founder-Market Fit
- 4. How VCs Evaluate Founders for Market Fit
- 5. How Founders Can Signal Fit During Fundraising
- 6. The Seafund Lens: FMF in Practice
- 7. FAQs