Moral Hazard and Risk Management: California’s AI Law Confuses Compliance with Safety

Let’s face it.  AI is kind of scary.  We don’t know where it’s going.  And we all have dystopian visions of Bladedrunner or a self-aware and controlling Hal seared in our minds from science fiction. While we don’t yet know how far we are from achieving general artificial intelligence, AI models are already transforming how we search the web, write, and work. So, it’s no wonder that it’s so tempting—for politicians and social thinkers alike—to regulate the unknown.

However, as is often the case, we don’t fully understand what we’re doing. It was tempting to regulate machines in the 19th century, because they might destroy jobs. The infamous Luddites went so far as to destroy them. In hindsight, the machines transformed jobs, rather than destroying them; there is no lump of labor. For example, at the time of the American Founding, approximately 95% of the population worked in agriculture; today, about 2% of the population feeds itself, while the other 98% benefit from modern machinery and agricultural practices. The Luddites were misguided in their worries and destruction.

California’s AI Law

According to the State of California, the new California Frontier AI Act will build public trust as AI rapidly evolves. It will provide security while maintaining California’s innovative leadership in frontier AI. “Frontier” AI is simply a technical term for the most advanced models. In the words of Governor Gavin Newsom:  “California has proven that we can establish regulations to protect our communities while also ensuring that the growing AI industry continues to thrive.” This framing is certainly reassuring. But it also reflects an ambition to control the trajectory of AI through regulation; it does not necessarily reflect an understanding of how AI systems actually evolve or where the real risks lie.

Like the European Union, with its precautionary principle embodied in the EU AI Act of 2025, the State of California is concerned about reducing AI risk ex ante, through regulation. One novel element of the AI Law is its definition of “catastrophic risk” as a foreseeable, material risk where an AI system could cause the death or serious injury of more than 50 people, or result in property damage exceeding $1 billion. This includes scenarios like helping to create weapons, committing crimes without human oversight, or escaping the developer’s control.

The AI Law raises important questions. How can companies anticipate all the scenarios where these thresholds might be met? And how should they interpret vague standards like “evading the control of its developer”? Even the numerical thresholds prompt questions:  why 50 lives and not 49? Why $1 billion and not $800 million in damage? 

Moreover, the bill defines a “frontier model” based on computing power: the AI must have been trained using more than 10²⁶ FLOPS (for non-technical readers, that’s basically a very fast supercomputer). But this raises an obvious question: what about a smaller model that is deliberately trained to cause harm? It would fall outside the law’s purview—not because it’s any safer, but simply because it’s less complex.

The AI Law defines a “large frontier developer” as any developer whose group of affiliates had annual gross revenue exceeding $500 million in the previous calendar year. These companies must publish a public safety framework explaining how they follow standards and assess risks, including catastrophic ones. They must also report any substantial changes and critical incidents that occur. Failure to comply may result in fines of up to $1 million. Here again, the threshold invites questions: why $500 million in gross revenue? Why not $400 or $600? Does a company become less dangerous just because it earns slightly less? 

Compliance Over Risk Management

Rather than regulating the actual harm that might come from AI, California has chosen to focus on the models and developers. And it is attempting to reduce harm through paperwork. But paperwork does not necessarily mean protection. Frameworks and filings prove that a company is capable of compliance and can pass an audit—they don’t prove that the system is actually safe or that the company is taking risk management seriously.

We need only look at the Great Financial Crisis of 2008, where “safe” banks and assets turned out not to be. Regulations like the Recourse Rule simply redefined risk; the banks complied, but weren’t any safer.

Regulation often distorts incentives. Over time, firms learn how to navigate the game of regulatory compliance: what to declare, what to sidestep or conceal (legally), and how to create systems that appear compliant on paper, even when real risks persist. Resources might be poured into compliance without clear evidence that there is a direct link between compliance and safety. What begins as a tool for accountability could end up reinforcing opacity.

The AI Law’s regulatory requirements create an opportunity cost; the time and resources spent on regulatory compliance could be allocated to growing the company and innovating. But it also creates an opportunity cost in attention to risk management. California lawmakers seem to assume that companies can’t be trusted to protect us from AI harm, yet they can somehow be trusted to identify, assess, and document risks accurately. Moreover, when the state determines what constitutes “risky,” the burden of judgment subtly shifts. In a classic case of moral hazard, AI firms—who are, after all, the experts on AI—will now face incentives to pass the burden of substantive risk management to the state bureaucracy, and focus instead on regulatory compliance. The definition of risk becomes a matter of satisfying bureaucrats, not deep reflection by subject experts. What begins as oversight may ultimately result in the outsourcing of responsibility. Over time, this shift towards a compliance approach is likely to weaken accountability, replacing active judgment with procedural obedience.

Returning Risk Management to Responsible Experts

This aspect of the AI Law is not a bug, but a structural feature of how regulation can replace adaptive compliance with static compliance. Tacit knowledge (which some readers will recognize from the work of economist F.A. Hayek) is context-specific and hard to articulate. It is the kind of insight that emerges through iteration, feedback, and trial and error. It is often what actually protects us. And yet, it resists easy codification. No compliance form, however detailed, can fully capture or replace the situational awareness, judgment, or informal signals that guide safe deployment in the real world.

When regulation equates safety with documentation, as does the California AI Law, it reduces complex realities to checking boxes and filing forms. It confuses visibility with understanding. And in doing so, it stifles the kind of learning that makes systems truly safer over time.

So, why regulate this way? One possibility is ignorance: lawmakers may simply not know how to regulate AI effectively. They may lack the information (or imagination, indeed!) to foresee where real harm could emerge. A second reason is strategic: by adopting a risk-based framework and shifting the burden onto companies, the state gains cover. If something goes wrong, someone else takes the fall. And third, this approach gives the state control over information flows. Once it defines categories and thresholds, it can expand them as needed, with little resistance.

Every time a government draws a regulatory line and attempts to regulate through political definition, it’s also deciding what to leave outside the regulation. The State of California is not just thwarting innovation. It’s also thwarting the kind of genuine learning and true risk management that only experts, market interactions, and accountability can generate, and replacing it with political and regulatory definitions.  

There is a discussion of using the California AI Act as a model for federal AI regulation. That would be a mistake. It would be far better to leave risk management to the AI firms and discipline them through the tort system rather than ex ante regulation.