Insights / Governance
Agentic AI Governance: A Checklist for Going Live
A practical AI agent governance checklist covering identity, tool access, data, cost limits, audit records, human review and recovery.
15 min read
Loading insights
Loading insights
Loading insight article
Insights / Governance
A practical AI agent governance checklist covering identity, tool access, data, cost limits, audit records, human review and recovery.
An agent is ready for a particular task when the team can demonstrate its operating boundaries: who is responsible, which information and tools are available, what needs approval, how usage is limited, and what happens when a run fails. Those boundaries need to hold during execution, including when an input is misleading or an integration behaves unexpectedly.
Use the checklist below to connect each decision to evidence. A missing control may mean narrowing the first release to reading approved information or preparing work for review. It need not mean abandoning a useful workflow.
This guide is for founders, product owners, engineering and security leads, operations managers, and legal and compliance teams. The aim is practical: keep useful automation under an authority the organization understands and can withdraw.
Model evaluation asks whether an output is useful and accurate. An agent introduces another responsibility: controlling the operations performed with that output. A convincing draft and permission to send it are separate decisions.
That distinction matters when an agent retrieves information, chooses a tool and continues working from the result. A rule written in its instructions may influence its behavior, but the software executing a tool call still needs to enforce access, scope and any required approval.
For example, an assistant preparing a client update can be given access to approved status fields and a restricted draft queue. Sending the update can remain unavailable until the system validates the recipient, message version and reviewer decision. The model's request to send does not establish those conditions.
Record the answer, the responsible owner and the evidence for every row. Revisit the record when the agent's permissions, tools, model or workflow change.
Scroll horizontally to view all columns.
| # | Check before release | Evidence to keep |
|---|---|---|
| 1 | Is its purpose and ownership clear? | A defined task, a business owner and an operational owner |
| 2 | Can the system distinguish the agent from its requester? | Agent identity and the user or system on whose authority it operates |
| 3 | Is access limited and revocable? | Approved scopes, credential lifetime and a demonstrated revocation procedure |
| 4 | Are information boundaries enforced? | Permitted sources, matter or tenant restrictions, and access tests |
| 5 | Are tools separated by the effects they can produce? | A tool inventory with allowed operations and affected resources |
| 6 | Does execution enforce the current rules? | Tests showing that unauthorized or changed actions cannot bypass the gate |
| 7 | Are consequential actions reviewed meaningfully? | A review queue, specific approval records and coverage when reviewers are unavailable |
| 8 | Can someone establish the outcome of a run? | Connected request, action, approval and destination records, including uncertain outcomes |
Include agents embedded in vendor products, internal scripts and experiments using live information. The inventory should reflect actual access, not just projects officially described as AI agents.
A useful starting point is to classify tools by what an error could affect. The categories below are a planning aid; the data and business context still determine the risk.
Scroll horizontally to view all columns.
| Action type | Typical operation | Example | Starting control |
|---|---|---|---|
| Read | Retrieve permitted information | Find a contract date or order status | Access checks and an activity record |
| Draft | Prepare material for later use | Save a proposed reply in a restricted queue | Source checks, restricted storage and review before use |
| Reversible change | Update a record that can be restored | Add an internal ticket label | Narrow permissions, limits and a tested restore procedure |
| High-impact action | Create an external or difficult-to-reverse effect | Send an email, issue a refund or delete a record | Explicit authorization and appropriate human approval |
Read and draft operations are not automatically harmless. Reading another client's matter can disclose information, and a saved draft creates another copy to protect. Reversible changes can also become consequential at volume.
Expose only the operations the workflow needs. A tool that creates a draft should not quietly include the ability to send, delete or change permissions.
Before execution, check the proposed operation against its target, the user's and agent's permissions, data sensitivity, possible impact and recovery limits. Keep these checks in the execution service so that a generated instruction cannot rewrite its own authority.
The gate can respond in five practical ways:
Approval should cover a specific action. If a recipient, attachment or material parameter changes after review, the earlier decision should not silently cover the new request. OWASP's transaction authorization guidance supports showing significant transaction details, server-side enforcement and protection against approval reuse.
Specify what happens when a required check is unavailable. For an operation that depends on valid approval or authorization, an unavailable check should leave the action pending or blocked rather than convert uncertainty into permission.
The NIST AI Risk Management Framework provides four functions for organizing risk management. The project examples below are an illustrative application, not an official mapping or a certification scheme.
Scroll horizontally to view all columns.
| NIST AI RMF function | Practical use in an agent project |
|---|---|
| Govern | Assign responsibility, agree boundaries and control who may change them |
| Map | Understand the task, affected people, data sources and connected systems |
| Measure | Evaluate output quality, security behavior and operating performance |
| Manage | Address findings, contain incidents and reconsider the permitted scope |
The OWASP Top 10 for Agentic Applications adds an agent-focused security perspective. Five relevant risks and example responses are shown here; this is a selection, not the complete Top 10.
Scroll horizontally to view all columns.
| OWASP risk | Example engineering response |
|---|---|
| ASI01 Agent Goal Hijack | Prevent instructions in retrieved material from granting new authority |
| ASI02 Tool Misuse and Exploitation | Restrict tool operations and validate calls before execution |
| ASI03 Identity and Privilege Abuse | Maintain distinct identities and constrain delegated access |
| ASI09 Human-Agent Trust Exploitation | Show reviewers the evidence and the real consequences of approval |
| ASI10 Rogue Agents | Monitor boundary violations and maintain a tested containment route |
The useful question for a framework item is what someone can inspect or test in this workflow. A framework name on a slide is not evidence that the corresponding control operates.
Use an identifiable service identity rather than borrowing a staff member's login. Keep the requesting user's authority connected to each operation; broad integration access must not expand that person's rights.
NIST's discussion of identity foundations for agentic AI explains the accountability problems with shared credentials and the importance of scoped authorization. Prefer suitable short-lived credentials where the integration supports them, and demonstrate how access can be revoked.
Keep credentials outside model prompts. Restrict risky execution environments, separate tenants or matters, and examine new tools and connections before enabling them. An added connector can change the workflow's exposure even when the model and prompt stay the same.
Test misleading instructions in documents, emails and tool output. OWASP's prompt injection guidance describes indirect attacks through external content and recommends layered defenses. Filtering helps, but it should not be the only protection between a manipulated response and a live operation.
For each source, document why the agent needs it, who owns it and which users may access it. Then trace where that information travels after retrieval: the model provider, temporary storage, review queue, generated files, logs and retained conversation history.
Work through five questions:
Check permissions again when the workflow requires current authorization. A request that was valid when queued may no longer be valid after a user's access is removed.
Evaluate generated content too. A status update should identify the correct matter, use approved sources and flag gaps instead of inventing an answer. Access controls and factual-quality tests address different failure modes.
A useful record connects the original request to what actually happened in the destination system. Capture:
An explanation generated afterward is not proof of the model's private reasoning. Distinguish observed events and enforced rule decisions from a narrative the model produces about them.
Protect the record against unauthorized access and modification. OWASP's logging guidance recommends excluding or masking sensitive data and safeguarding retained logs. Do not copy whole confidential documents into general-purpose telemetry simply to make it look complete.
Bound model usage, tool calls, retries and elapsed time for each run. Add a shared limit for the relevant account, team or tenant, and consider concurrent work and usage that has not yet appeared in a billing dashboard. OWASP's AI agent security guidance identifies unbounded loops and spending as risks and recommends execution limits and adversarial tests.
Costs are only part of the problem. Repeated calls can also send duplicate messages or create duplicate records. A timeout does not establish whether the destination completed an operation.
Where supported, use a stable request identifier and the destination's duplicate-prevention mechanism. AWS describes this pattern in Making retries safe with idempotent APIs. If a result is unknown, investigate or route it for review before retrying a consequential action. A locally generated identifier alone cannot guarantee how another service behaves.
Track cost alongside completed work, corrections and failures. Cheap output that creates more review or repair work may not improve the workflow.
Show reviewers the proposed change, supporting information, affected resources and reason for escalation. For a message, that includes the recipient, content and attachments. Give the reviewer a clear way to edit, reject or pause the action.
Record the version approved and check it at execution. Coordinate approval consumption across workers so that two attempts cannot independently use the same decision to perform two consequential operations.
Agree who covers the queue, how old an item may become and what happens when no reviewer is available. A timed-out review should remain pending or expire under the agreed rules; it should not become approval by default.
Keep corrections and rejection reasons. They help identify whether the agent needs better source information, a narrower task or a different review process. Expand volume only when reviewers can still examine the work meaningfully.
Access reviews, security tests, approval records and incident rehearsals can support a wider assurance process. Connect each relevant requirement to its control, owner and evidence rather than collecting disconnected screenshots shortly before an audit.
The applicable requirements depend on the organization, information and workflow. Using a NIST or OWASP reference does not itself establish compliance with a law, contractual obligation or professional duty.
For work involving ISO 27001, SOC 2, HIPAA or GDPR, the legal, compliance and assurance owners determine the obligations and evidence they need. Engineering implements and tests the technical controls within that scope. This checklist provides technical preparation, not a legal assessment or an audit outcome.
Test the stop control against queued jobs, active workers and requests already sent to external systems. Disabling a scheduler alone may leave other work running. Check what will happen after a restart too.
Use a test environment to rehearse at least these cases:
Write down the expected behavior first, retain the result and repeat failures after a fix.
Separate undo from repair. A reversible record change may be restored after checking for newer edits. An email already delivered cannot be retracted by rolling back a database. For external effects or disclosures, define containment, investigation and the responsible incident owners. Describe those limits before release.
Use stages to make the permitted scope visible. The following is a practical planning model, not an official maturity standard or a promise that every agent should reach the last stage.
Scroll horizontally to view all columns.
| Stage | Permitted work | Evidence before expanding scope |
|---|---|---|
| Test | Work with sample information | Agreed quality, access and failure tests pass |
| Shadow | Observe approved live information without acting on the output | Useful results, protected data and traceable runs |
| Draft | Prepare restricted work for a person to approve | Understood corrections and manageable review effort |
| Supervised action | Make specifically authorized, reversible changes | Proven monitoring, approval enforcement and recovery drills |
| Bounded autonomy | Perform named low-risk tasks within agreed limits | Sustained quality, cost control and operational coverage |
The joint guidance on Careful Adoption of Agentic AI Services recommends a cautious approach focused on low-risk, non-sensitive tasks. Confidential or consequential workflows require their own assessment; progressing through this table does not remove that responsibility.
Drafting may remain the right long-term boundary, especially where a professional retains responsibility for the final work. A new model, connector, data source or serious incident can also justify reducing authority and repeating the affected tests.
Choose a useful task and identify the smallest release that helps the person doing it. List the available actions, agree who owns the workflow and establish its information boundary.
Build the relevant permission checks, review steps, activity record and operating limits. Evaluate output quality alongside security behavior. Rehearse failure and recovery, then decide which operations the evidence supports.
Keep the decision record tied to a version and a review trigger. Reuse successful controls across other agents, but reassess the permissions and consequences for each new workflow.
I have 6+ years of engineering experience, including 3+ years in legal technology. My work covers AI agent development, AI automation, AI product development, application security testing and AI and data governance.
The implementation work can include approved-source retrieval, tool permissions, review screens, audit records and tests around sensitive operations. The Legal AI Intake case study shows the role of human review in a working system. My governance, guardrails and auditability page explains the broader approach.
If an agent is close to live use, start with the task and the operations you want it to perform. That provides a concrete basis for deciding what to build, what to restrict and what to test.
Tell me about the workflow, and I can help you decide what to build, control and test first.
Discuss your AI agent(opens in new tab)It covers the authority given to an agent, the information and tools it may use, the decisions reserved for people, and the evidence and recovery procedures around its actions. The scope needs to be defined for the actual workflow rather than inferred from the model's capabilities.
The system needs to evaluate both the generated output and the operations performed with it. An accurate answer does not establish permission to change a record or contact a recipient. Those permissions and approval conditions need to be checked during execution.
Limit the sources and fields available to the task, preserve user and matter or tenant restrictions, and trace copies through providers, storage, review queues and logs. Apply access and retention rules to derived output as well as original documents.
Keep enough to connect the request, identities, relevant versions, sources, proposed operations, rule decisions, human approval and destination result. Protect that evidence and avoid collecting unnecessary confidential material. Include refused actions and unresolved outcomes.
Use per-run and shared usage limits, bounded retries and concurrency controls appropriate to the workflow. Test multiple runs together and account for work already in progress. Alerts help an operator respond, but should not replace enforced limits.
No universal rule fits every operation. The workflow owner must define what may run within approved limits and what requires review. Consequential, sensitive or difficult-to-reverse operations need appropriate authorization and oversight; evidence from a drafting pilot alone does not justify broader authority.
It can organize technical evidence, but it does not determine whether all applicable obligations are met. The relevant legal, compliance and assurance owners assess those obligations and the sufficiency of the evidence.
Research checked on 5 November 2026. The checklist, mappings and stages are practical illustrations, not an official standards crosswalk or legal advice.
Tell me about the system or workflow you are working on.
30-minute intro call on Calendly
| 9 | Are resource use and repeated attempts bounded? | Per-run and shared limits, retry rules and tested alerts |
| 10 | Can the team contain a failure and recover appropriately? | Stop drills, outstanding-work checks and action-specific recovery procedures |