> ## Content Index > Fetch the complete content index at: https://samdervis.com/llms.txt > Use this file to discover other available public pages before exploring further. # AI Guardrails Should Match the Blast Radius - URL: https://samdervis.com/ai-guardrails-should-match-the-blast-radius/ - Published: 2026-09-24T16:31:04.000Z - Updated: 2026-09-24T16:31:04.000Z - Author: Sam Dervis - Tags: AI in Engineering, Engineering Leadership, Engineering Management “Always verify AI-generated code.” Good advice. Also slightly incomplete. It sits somewhere near “write good tests” and “keep the documentation updated” in the collection of engineering instructions that everyone agrees with and then immediately needs another 14 paragraphs to define. Because verifying an AI-generated README change and verifying a change to authentication are not really the same job. One might need a quick read. The other might need additional tests, security review, code owners, a deployment plan, and possibly the one engineer who still remembers why that service has three databases. So when engineering teams talk about AI guardrails, I think the starting question should be much simpler: **What happens if this thing is wrong?** ## Not all AI-generated code carries the same risk A lot of AI guidance still sounds like this: - Don’t share sensitive data. - Review the output. - Keep a human in the loop. - Use AI responsibly. All sensible. Also not particularly helpful when an engineer is sitting in front of a pull request wondering whether the AI-generated change needs the usual review, a security review, or a small committee and some snacks. The problem is that AI-assisted work gets discussed as one category. It isn’t. Generating test fixtures is very different from modifying authentication. Writing a small script is very different from changing how payments are processed. Updating documentation is very different from changing production infrastructure. We already understand this outside AI. Nobody sees a typo in a README and says: > We should probably involve the architecture review board. At least I hope not. We apply stronger controls when the consequences of failure are higher. AI-generated changes should work the same way. ## Start with blast radius The bigger the potential impact, the stronger the guardrails should be. That means asking questions like: - What happens if this change is wrong? - How quickly would we notice? - How easily could we reverse it? - What systems does it affect? - What permissions does the AI have? - Can it only suggest changes, or can it execute them too? That last one matters more than it used to. AI tools are moving beyond autocomplete. They can create files, run commands, modify code, open pull requests, interact with systems, and increasingly act with less supervision. There is a fairly big difference between: > “Here is a Terraform change you might want to review.” and: > “I have applied the Terraform change.” One is assistance. The other may be the opening sentence of tomorrow morning’s incident review. ## I find three zones useful I would not try to make this overly complicated. For most engineering organisations, three broad categories are enough to start the discussion. ### 1\. Low-risk: let AI help freely These are tasks where mistakes are cheap, easy to detect, and easy to reverse. For example: - generating test data - creating boilerplate - improving documentation - explaining unfamiliar code - suggesting names - writing small local scripts - generating initial test cases - formatting or transforming data This does not mean blindly trusting the output. It just means we probably do not need to invent another approval workflow. Existing checks may already be enough, e.g., tests, CI, Normal code review. An engineer looking at the change and saying, “Yes, this does what I intended.” Done. No steering committee required. ### 2\. Medium-risk: AI writes, humans challenge This is probably where a large part of AI-assisted software development belongs. The AI can help produce the code, but meaningful human review still matters. Examples might include: - application business logic - API behaviour - database queries - infrastructure configuration - dependency changes - non-trivial refactoring - concurrency logic - performance-sensitive code The important word here is **meaningful**. A human approving the PR is not automatically the same thing as human oversight. We have all seen approvals that effectively mean: > “The tests are green and I trust you.” That may be fine sometimes. It is less reassuring when the code was mostly generated by a system that sounds very confident even when it has misunderstood the problem completely. For these changes, review should look beyond whether the code is syntactically correct. The reviewer should understand: - the intent - the failure modes - the tests - the operational impact - edge cases - rollback options - security implications AI can speed up implementation. Unfortunately, it has not yet volunteered to join the incident call at 02:17. Accountability still belongs to the team. ### 3\. High-risk: slow down a little Some areas deserve stronger controls. Depending on the system, that might include: - authentication - authorisation - payment flows - sensitive data handling - security policies - destructive database operations - production infrastructure - cryptography - irreversible changes This does not mean AI cannot be used there. It means the cost of being wrong is high enough that we should be more deliberate. Maybe that requires: - named code owners - additional tests - security review - architecture review - explicit approval before execution - stronger logging - limited permissions The exact controls will differ by company. A healthcare system and an internal lunch-menu app probably do not need the same level of governance. Although depending on the office, the lunch-menu app may be considered critical infrastructure. ## Permissions matter just as much as code One thing I think engineering leaders should pay much more attention to is **what the AI is actually allowed to do**. There is a big difference between an agent that can: - read code - modify code - execute commands - access credentials - deploy changes - modify infrastructure - query production systems Those should not all come bundled together just because the tool supports them. We have spent years teaching engineers about least privilege. AI does not magically make that principle obsolete. If an agent only needs to inspect logs, do not give it permission to restart production. If it only needs to propose infrastructure changes, maybe it does not need credentials to apply them. If an action is difficult to reverse, introduce explicit approval. This is less exciting than talking about autonomous engineering agents. But “the agent had more permissions than it needed” is also the kind of sentence that looks painfully obvious in a postmortem. ## “Keep a human in the loop” needs a definition This phrase appears everywhere in discussions about AI. It sounds reassuring. Until you ask: **Which human?** The engineer who owns the system? Someone approving PRs between two meetings? The person who joined last Tuesday and is currently discovering that `payment-service-final-v2` is not, in fact, the final payment service? Human review only works when the reviewer has enough context to challenge what they are seeing. Otherwise, we are just adding a human-shaped checkbox to the process. For higher-risk changes, engineering teams should probably define what review actually means. Something like: **Low risk:** normal pull request review. **Medium risk:** reviewer understands the affected component, validates tests, and considers failure behaviour. **High risk:** appropriate code owner, stronger automated checks, and explicit approval before execution. Now “human in the loop” means something. ## Don't create an AI bureaucracy There is another trap here. In trying to manage AI risk, teams can create a completely separate set of processes for anything AI touches. New forms. New approval steps. New committees. Possibly a spreadsheet. We are engineers. A spreadsheet will appear eventually. But most organisations already have useful controls: - code ownership - CI/CD - automated testing - security scanning - architecture reviews - production permissions - separation of duties - observability - incident response - change management AI guardrails should mostly strengthen those systems, not replace them. The question is not: > How do we create a new AI governance process? It is: > Where does AI change the risk enough that our existing engineering process needs to change? That is a much more practical conversation. ## A simple way to start If I were defining AI guardrails with an engineering team, I would start by listing the kinds of work the team does. Then, for each one, ask: **What happens if the AI is confidently wrong here?** Then ask: - Would we notice quickly? - Can we reverse it? - Does another engineer need to review it? - Does it touch sensitive data? - Does it require production access? - Can it execute the change itself? - What controls do we already have? That gives you something much more useful than a policy document telling everyone to “use AI responsibly.” Because responsible according to whom? And responsible for what? ## The useful question isn't whether AI was involved AI use in engineering is only going to increase. - The tools will become better. - They will also become more autonomous. - Some tasks that feel risky today may become routine. - Others will remain areas where teams should be cautious. So I do not think the useful distinction is: **AI-generated vs human-generated.** The better distinction is: **What is the blast radius if this change is wrong?** Then apply the controls that make sense. A documentation typo probably does not need a security review. An authentication change probably deserves more than: > “Looks good to me.” Good guardrails should not make engineers afraid to use AI. They should make it possible to use AI faster in the places where speed is cheap, and more carefully in the places where mistakes are expensive. Which, now that I think about it, is more or less how we should have been building software all along.