Governing AI agents: a practical checklist for leaders

An AI that answers questions can give a bad answer. An AI agent that takes actions can post an invoice, change a permission or email a customer. The controls that matter are the ones you'd apply to a new employee with system access: who it is, what it's allowed to do, who checks its work, and what record it leaves. Here's the checklist I use before an agent touches a production system.

The checklist

Identity and access

• Give every agent its own identity, never a shared or personal account.
• Grant least privilege: a short list of tools, each with the minimum permissions it needs.
• Limit what an agent can see to what the person it's acting for could see.

Human oversight

• Decide which actions need approval, using dollar, risk and reversibility thresholds.
• Show the evidence with every recommendation, so approvers can check rather than just click.
• Widen autonomy only when measured accuracy supports it.

Data protection

• Classify data before agents are allowed to search it.
• Mask personal and regulated data in prompts and outputs.
• Confirm in writing whether your AI provider stores your prompts or uses them for training.

Security

• Treat everything an agent reads, including emails, documents and web pages, as untrusted. Prompt injection tops OWASP's Top 10 risks for large language model applications.
• Route prompts and responses through a gateway that filters, logs and rate-limits them.
• Test agents adversarially before launch and after every major change.

Audit and monitoring

• Log every prompt, tool call, decision and approval to your security monitoring.
• Measure accuracy continuously against what people actually decided.
• Watch for drift whenever models, data or prompts change.

Cost and accountability

• Set usage and cost limits per agent, and track cost per transaction next to the business result.
• Name a business owner for every agent, not just a technical one.
• Add agents to incident response: how you pause one, and how you undo what it did.

Frameworks worth aligning to

You don't need certification on day one, but aligning early saves rework. NIST's AI Risk Management Framework gives a practical structure in four functions: govern, map, measure and manage. ISO/IEC 42001 is the management-system standard for AI, useful when customers expect certification. OWASP's Top 10 for large language model applications is the security team's starting list. If you operate in the EU, the EU AI Act adds obligations that depend on how risky the use case is.

Start small, on purpose

Pick one workflow with clear rules and measurable outcomes, such as invoice exceptions or access requests. Run the agent in observe-only mode first and compare it with your team's decisions. Expand what it may do only when the numbers support it. That's the pattern in our agentic AI reference design.