Governing Generative AI

Why Generative AI Needs Specific Governance

Traditional AI governance was designed for predictive models: a credit scoring algorithm, a fraud detection system, a recommendation engine. These systems are trained on known data, produce structured outputs, and are deployed by specialist teams.

Generative AI is different in ways that matter for governance:

• Broad access. GenAI tools like ChatGPT, Copilot, and Gemini are used by non-technical employees across the organization. Governance cannot rely on a small team of ML engineers following best practices.
• Non-deterministic outputs. The same prompt can produce different results. Testing and assurance require different methods than traditional model validation.
• Data flows in both directions. Users send data to GenAI systems through prompts. That data may be logged, used for training, or processed by third-party sub-processors. The input is the risk, not just the output.
• Rapid vendor evolution. Models are updated frequently, sometimes without notice. A tool assessed in January may behave differently by March.
• Confident errors. GenAI produces fluent, authoritative-sounding text even when the content is fabricated. Users without domain expertise may not catch errors.

Your existing governance framework (intake, risk tiering, security review, human oversight) still applies. This page shows how to apply it to the specific challenges of generative AI.

Acceptable Use Policies

An acceptable use policy (AUP) for generative AI sets clear boundaries for employees before individual use cases reach the council. A good AUP answers the questions employees are already asking: "Can I use ChatGPT for this?"

What to cover

Approved tools. List the GenAI tools the organization has vetted and approved. Be explicit: "Use Microsoft Copilot for internal document drafting" is more useful than "use approved tools." Update this list as new tools are assessed.

Permitted uses. Define what employees can do with approved tools. Common permitted uses include drafting internal documents, summarizing meeting notes, generating code with human review, and brainstorming. Be specific enough that employees can self-assess without filing an intake request for routine tasks.

Prohibited uses. State what is not allowed. Common prohibitions include:

• Entering customer personal data, trade secrets, or confidential business data into public GenAI tools
• Using GenAI outputs in regulated decisions (hiring, credit, benefits) without human review and council-approved safeguards
• Publishing GenAI-generated content externally without human review and disclosure where required
• Using GenAI to impersonate individuals or generate deceptive content

Disclosure requirements. Define when employees must disclose that GenAI was used. This varies by context: internal drafts may not need disclosure, but customer-facing content or regulatory filings likely do.

Human review expectations. State clearly that GenAI outputs must be reviewed by a qualified person before use in any consequential context. Define what "qualified" means for different use cases.

Keeping the AUP current

GenAI capabilities change fast. Build a review trigger into your Policy Refresh cycle, and update the approved tools list whenever the council assesses a new GenAI tool.

Risk Tiering for Generative AI

The toolkit's four-tier risk model applies to GenAI, but some use cases land at a different tier than you might expect. The key factors are data sensitivity, audience, and degree of automation.

The main adjustment for GenAI: do not under-tier based on the tool's general-purpose nature. A copilot that drafts emails is Tier 1-2. The same copilot connected to your HR database is Tier 3. Tier based on what the system accesses and what decisions it informs, not on the brand name.

Security and Prompt Injection

GenAI introduces attack surfaces that traditional application security does not address. Your security team should be aware of these risks even for vendor-hosted tools where you do not control the model.

Key threats

Prompt injection. An attacker (or a user) crafts input that overrides the system's intended behavior. This can cause the system to ignore safety instructions, leak system prompts, or take unintended actions. For agentic systems with tool access, prompt injection can lead to unauthorized actions.

Data extraction. Carefully crafted prompts can coax a model into revealing training data, system prompts, or information from other users' sessions (in poorly isolated multi-tenant systems).

Insecure output handling. GenAI outputs may contain executable code, SQL, or markup. If downstream systems execute GenAI output without validation, this creates injection vulnerabilities.

Excessive agency. Agentic AI systems that can browse the web, execute code, or call APIs amplify the impact of any prompt injection or hallucination. Scope and permission limits are critical.

What to do

For detailed checklists, see the Security Review page (especially the LLM-Specific Risks and Generative AI Specific sections) and the Red-Teaming guide for adversarial testing approaches.

At a minimum for any GenAI deployment:

• Apply input validation and output filtering
• Treat all GenAI outputs as untrusted in downstream systems
• Limit tool access and permissions for agentic systems to the minimum required
• Log prompts and outputs for audit and incident response
• Test for prompt injection before deployment and after model updates

Data Leakage Risks

Data leakage is the primary GenAI concern for security leaders. Unlike traditional systems where data flows are well-defined, GenAI creates new pathways for sensitive information to leave the organization.

How data leaks through GenAI

Prompts as training data. Some GenAI providers use customer prompts to improve their models. An employee pasting a confidential contract into a public chat interface may be contributing that text to the provider's training set. Check vendor terms carefully and opt out where possible.

Context window exposure. In shared or multi-tenant environments, poor session isolation could allow one user's context to bleed into another's responses. This is primarily a vendor architecture concern, but your assessment should confirm isolation guarantees.

Third-party sub-processors. Your vendor may route requests through other model providers. Data that you send to Vendor A may be processed by Vendor B's infrastructure. The Governing Purchased AI assessment covers this.

Employee behavior. The most common leakage vector is employees pasting sensitive data into GenAI tools, often with good intentions (summarizing a document, debugging code with production data). The AUP and training are your primary controls here.

Controls

• Classify data before it touches GenAI: public, internal, confidential, restricted. Define which classifications can be used with which tools.
• Use enterprise versions of GenAI tools that contractually exclude customer data from training.
• Deploy DLP (data loss prevention) tools that monitor for sensitive data in GenAI prompts where technically feasible.
• Train employees on what data they can and cannot use with GenAI tools. Make this part of your Training and Literacy program.

Human Oversight for GenAI Outputs

GenAI produces text, code, images, and analysis that sounds confident regardless of accuracy. This makes human oversight both more important and harder to do well. See Human Oversight for the full framework.

Why GenAI is different

Hallucination. GenAI can fabricate facts, citations, statistics, and legal references with complete fluency. A human reviewer who is not a subject-matter expert may not catch these errors.

Automation bias. Well-formatted, articulate output creates a strong presumption of correctness. Studies consistently show that people over-trust AI outputs, especially when the output matches their expectations.

Volume. GenAI can produce content at a scale that makes line-by-line human review impractical. Governance needs to define where review is required and where sampling or spot-checking is acceptable.

Practical guidance

• Match reviewer expertise to the domain. A legal document drafted by GenAI should be reviewed by a lawyer, not by the person who prompted the model.
• Define review standards by risk tier. Tier 1 may accept self-review by the user. Tier 2-3 should require review by someone other than the prompter, with domain expertise.
• Do not assume review is happening. Build review into the workflow (e.g., approval gates, diff views) rather than relying on policy alone.
• For high-volume use cases, define sampling rates and escalation triggers rather than attempting 100% review.

Agentic AI

Agentic AI systems go beyond generating text or images. They take actions: writing and executing code, sending emails, querying databases, calling APIs, browsing the web, and orchestrating multi-step workflows. Coding agents, autonomous research assistants, workflow automation tools, and systems connected through protocols like MCP (Model Context Protocol) all fall into this category.

The governance difference is straightforward. When a GenAI chatbot hallucinates, you get a wrong answer. When an agentic system hallucinates, it may take a wrong action: deleting records, sending unauthorized communications, or modifying production systems. The blast radius of failure is larger, and the speed of failure is faster.

Risk tiering

Agentic systems should be tiered based on what they can do, not what they are intended to do. A coding agent with read-only access to a repository is a different risk profile than one with permission to merge pull requests and deploy to production.

Most agentic systems should tier at 2 or above. Systems with write access to production data, customer communications, or financial systems are Tier 3 by default.

Permission scoping

Treat tool access as you would IAM permissions: least privilege, explicitly granted, regularly reviewed.

• Enumerate allowed tools. Define exactly which tools and APIs the agent can call. "Access to everything" is not an acceptable scope.
• Separate read from write. An agent that can query a database does not automatically need permission to modify it.
• Gate sensitive actions. Require human approval for actions that are irreversible, affect external parties, or touch sensitive data. Build these gates into the system, not into policy alone.
• Scope credentials narrowly. If the agent calls external APIs, issue scoped tokens with minimum required permissions. Do not pass user credentials through to agentic systems.
• Review permissions on a schedule. As agents are updated or given new capabilities, their permission scope should be reassessed.

Human oversight for multi-step execution

Traditional human oversight assumes a single decision point: the AI recommends, the human approves. Agentic systems challenge this model because they chain multiple steps together, each building on the last. Reviewing only the final output misses the decisions made along the way.

Design oversight for agentic systems around checkpoints, not endpoints:

• Define checkpoint triggers. Require human review before the agent takes high-consequence actions (e.g., sending external communications, modifying production data, spending above a threshold).
• Make the chain of reasoning visible. The human reviewer needs to see not just the proposed action but the steps that led to it.
• Set boundaries, not just approvals. Define what the agent is allowed to do without asking, and what requires escalation. This is more sustainable than approving every step.
• Plan for intervention. The human must be able to pause, redirect, or stop the agent mid-execution. If stopping the agent mid-workflow creates an inconsistent state, that is a design problem to solve before deployment.

Logging and auditability

You cannot govern what you cannot reconstruct. Agentic systems must log enough detail to answer the question: "What did the agent do, in what order, and why?"

At minimum, log:

• Every tool call and API request the agent makes, with timestamps
• The inputs and outputs of each step in the execution chain
• Which decisions were made autonomously and which had human approval
• Any errors, retries, or fallback actions

These logs serve both incident response (what went wrong) and ongoing governance (is the agent behaving as intended over time). Feed anomalies into your Monitoring process.

Common Vendor Scenarios

Most organizations encounter GenAI through purchased tools, not in-house development. The Governing Purchased AI page covers the full vendor lifecycle. Here is how common GenAI scenarios map to governance actions.

Enterprise copilots (Microsoft Copilot, Google Gemini for Workspace)

These tools are typically rolled out organization-wide, which means the governance decision affects every employee.

• Conduct an organization-wide assessment before rollout, not after.
• Define an acceptable use policy specific to the copilot (what data sources it can access, what tasks it should and should not be used for).
• Tier at 2 for general use. Escalate to Tier 3 if the copilot is connected to sensitive data stores (SharePoint sites with HR data, financial systems).
• Assign a champion or champion team to handle ongoing questions and collect feedback.

Foundation model APIs (OpenAI, Anthropic, Google)

API access means your developers are building applications on top of foundation models. The risk is in the application, not just the API.

• Assess each application built on the API separately. API access approval does not mean every application built on it is approved.
• Tier 3 by default. Security review required for every application.
• Require developers to document prompt designs, data flows, and guardrails for each application.
• Re-assess when the underlying model is updated (e.g., GPT-4 to GPT-5).

Embedded AI features (Salesforce Einstein, ServiceNow AI)

AI features are increasingly enabled by default in enterprise platforms. The governance gap is that nobody filed an intake request because nobody "bought AI."

• Proactively scan your existing platform portfolio for AI features that may have been enabled without assessment.
• Assess when features are first enabled, not when the platform was originally purchased.
• Champions in each business unit are your front line for spotting these.

Free and consumer-tier tools

The hardest category to govern. Employees use free ChatGPT, Claude, and other tools without procurement involvement.

• The AUP is your primary control. Be clear about what is and is not permitted.
• Block access at the network level if your risk appetite requires it, but recognize that personal devices make this imperfect.
• Offer approved alternatives. Employees use shadow AI because it helps them do their jobs. Give them a sanctioned option.
• Champions should surface shadow usage patterns without creating a punitive culture.

Where to Go Next

This page provides the unified view. For detailed guidance on each topic:

• Risk Tiering for the full four-tier model and tiering worksheet
• Security Review for the complete security checklist including LLM-specific risks
• Red-Teaming for adversarial testing approaches
• Human Oversight for the full oversight framework
• IP and Copyright for decision frameworks on IP ownership, copyright risk, and disclosure obligations
• Governing Purchased AI for the vendor lifecycle and assessment checklists
• Training and Literacy for building GenAI awareness across the organization
• NIST AI RMF for the Generative AI Profile