Anthropic has told California Governor Gavin Newsom that courts should have the authority to block AI models posing serious risks to the public, proposing judicial oversight as a critical tool for AI safety.
- Anthropic recommends court orders to block harmful AI models
- Supports targeted controls instead of full AI system kill switches
- Proposals align with California’s expanding AI safety discussions
What happened
Anthropic submitted a letter to California Governor Gavin Newsom advocating for the government’s ability to seek court orders that could block AI models identified as posing a serious public risk. This recommendation reflects the company’s prior policy calls and underscores the need for legal oversight of AI deployments. The letter was sent following Newsom’s convening of an expert panel to draft AI safety proposals, with other leading AI firms like OpenAI also offering recommendations.
The plea comes in the context of Anthropic’s own experience with government restrictions. In June, the Trump administration mandated Anthropic to restrict foreign access to some of its advanced AI models due to national security concerns, though this order was later lifted by the US Commerce Department. Anthropic stresses the importance of judicial intervention as part of a comprehensive approach to managing AI safety and risks.
Why it matters
Anthropic’s stance highlights a growing recognition within the AI industry and government that existing regulatory frameworks may be insufficient to manage emerging AI risks. Allowing courts to block the release of dangerous AI models could introduce a robust check against models that might cause harm before deployment, reinforcing public trust and safety.
The company also cautions against relying on broad “kill switches” capable of shutting down entire AI systems, arguing these may create a deceptive sense of security. Instead, Anthropic advocates for more surgical interventions such as disabling specific problematic AI agents or cutting access for individual users, which it believes offer more precise and effective risk mitigation.
What to watch next
California’s policy discussions on AI safety, spurred by Governor Newsom’s panel and stakeholder input, will be critical to watch as they may shape future state regulations on AI use and governance. Other AI industry players’ responses and legislative activities at the federal level—such as bipartisan Senate proposals and the AI Kill Switch Act in the US House—will influence the national framework around AI risk management.
Anthropic’s impending initial public offering is another key development. As one of the few AI startups preparing a major market debut, its policy positions and regulatory interactions could have significant implications for investor sentiment, industry standards, and how emerging AI companies navigate evolving governance landscapes.