Series overview
Part 11 of 1765% complete
2026-08-31•6 min read

Add write tools and human approval: the gate the model cannot cross

Checkpoint tag: chapter-10-human-approval — no mutating call executes without a valid scope and an explicit approval whose stored argument hash matches byte-for-byte what will be sent.

What will be built

The write half of the platform. The MCP server gains create_incident and append_incident_note. agent-api gains the approval state machine (PENDING → APPROVED → EXECUTED, plus DENIED/EXPIRED), an approvals table, single-use opaque tokens bound to everything that matters, argument-hash integrity checking, ApprovalRequired/Completed SSE events, and an append-only audit record for every decision and execution.

Why it matters

This is the chapter the whole security posture was built for. Every earlier control converges here: the model proposes (Chapter 8), scopes gate (Chapter 9), the approval binds the exact arguments the human saw, and the idempotency key makes the execution retry-safe (Chapter 2). The failure being designed out is the one that makes headlines: a model — prompted or injected — creating fifty incidents, or approving itself, or getting an approval for “restart notifier” and executing “delete everything” because the arguments changed between check and use.

Concepts explained

The approval binds arguments, not intent. The stored record contains args_hash = sha256(canonicalJson(args)). At execution time we recompute the hash over what would actually be sent. If the model — or anyone — altered an argument after the human approved, the hashes differ and the call dies. Time-of-check/time-of-use is closed by comparing the bytes, not the description.

Single-use is a database fact, not a memory fact. The approval completes via UPDATE approvals SET status='EXECUTED' … WHERE id=? AND status='PENDING'. One row can transition once; two racing approvals produce one winner and one 0 rows updated. Databases are better at this than AtomicBoolean.

Idempotency key = derived, not supplied. The agent generates sha256(approvalId | tool | argsHash) as the Idempotency-Key passed to the simulator. A retry of the same approved call replays the same incident rather than creating a twin — and the key is deterministic, so a second execution attempt of a consumed approval fails at the approval check before ever reaching the simulator.

Opaque token, not capability token. The token in the SSE event is just a lookup key — its authority is the row it points at. (A signed JWS variant with embedded claims is sketched in docs/adr/; we chose the opaque form because the row must exist anyway for audit, and one source of truth beats two.)

Files added or changed

mcp-operations-server/…/tools/WriteTools.kt (+ SimulatorClient write methods)
agent-api/src/main/resources/db/migration/V2__approvals.sql, V3__audit.sql
agent-api/src/main/java/in/o612/eng/opsagent/agent/
approval/{Approval, ApprovalStore, ApprovalService, ApprovalController, ArgsCanonicalizer}.java
audit/AuditService.java
orchestration/AgentOrchestrator.java (approval suspension + resume)
api-requests/approval.http

Complete code

agent-api/src/main/resources/db/migration/V2__approvals.sql
CREATE TABLE agent.approvals (
id TEXT PRIMARY KEY, -- approval id (also the opaque token)
conversation_id TEXT NOT NULL REFERENCES agent.conversations(id),
tenant_id TEXT NOT NULL,
user_id TEXT NOT NULL, -- requester
tool TEXT NOT NULL,
args_json TEXT NOT NULL, -- normalized arguments as shown to the approver
args_hash TEXT NOT NULL, -- sha256 of canonical args
status TEXT NOT NULL CHECK (status IN
('PENDING','APPROVED','EXECUTED','DENIED','EXPIRED')),
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
expires_at TIMESTAMPTZ NOT NULL,
decided_by TEXT,
decided_at TIMESTAMPTZ
);
CREATE INDEX approvals_pending_idx ON agent.approvals (conversation_id) WHERE status = 'PENDING';
agent-api/src/main/resources/db/migration/V3__audit.sql
CREATE TABLE agent.audit_events (
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
at TIMESTAMPTZ NOT NULL DEFAULT now(),
tenant_id TEXT NOT NULL,
actor TEXT NOT NULL, -- user sub or service account
event_type TEXT NOT NULL, -- APPROVAL_REQUESTED | APPROVED | DENIED | TOOL_EXECUTED | ...
tool TEXT,
conversation_id TEXT,
approval_id TEXT,
args_hash TEXT,
detail JSONB NOT NULL DEFAULT '{}'::jsonb
);
-- no UPDATE/DELETE grants: append-only by permission, not convention
agent-api/src/main/java/in/o612/eng/opsagent/agent/approval/ApprovalService.java
package in.o612.eng.opsagent.agent.approval;
import in.o612.eng.opsagent.agent.audit.AuditService;
import in.o612.eng.opsagent.agent.security.CallerContext;
import in.o612.eng.opsagent.agent.tools.McpToolExecutor;
import in.o612.eng.opsagent.agent.tools.ToolPolicyRegistry;
import org.springframework.stereotype.Service;
import java.time.Duration;
import java.time.Instant;
import java.util.Map;
import java.util.UUID;
@Service
public class ApprovalService {
public sealed interface Decision {
record Pending(String approvalId, Instant expiresAt) implements Decision {}
record Executed(String toolResult) implements Decision {}
record Refused(String reason) implements Decision {}
}
private final ApprovalStore store;
private final McpToolExecutor executor;
private final ToolPolicyRegistry policies;
private final AuditService audit;
private static final Duration TTL = Duration.ofMinutes(5);
/** Called from the agent loop when a HIGH/MEDIUM-risk tool is proposed. */
public Decision requestApproval(String conversationId, CallerContext requester,
String tool, Map<String, Object> args) {
var canonical = ArgsCanonicalizer.canonical(args);
var approval = store.insertPending(
UUID.randomUUID().toString(), conversationId,
requester.tenantId(), requester.userId(),
tool, canonical, ArgsCanonicalizer.sha256(canonical),
Instant.now().plus(TTL));
audit.record("APPROVAL_REQUESTED", requester.tenantId(), requester.userId(),
tool, conversationId, approval.id(), approval.argsHash(), Map.of());
return new Decision.Pending(approval.id(), approval.expiresAt());
}
/** Called by POST /approvals/{id}. Enforces every binding in one transaction-shaped sequence. */
public Decision decide(String approvalId, CallerContext approver, boolean approve) {
var row = store.find(approvalId).orElse(null);
if (row == null) return new Decision.Refused("unknown approval");
if (row.status() != Approval.Status.PENDING)
return new Decision.Refused("approval already consumed");
if (row.expiresAt().isBefore(Instant.now())) {
store.transition(approvalId, Approval.Status.EXPIRED, approver.userId());
return new Decision.Refused("approval expired");
}
if (!row.tenantId().equals(approver.tenantId()))
return new Decision.Refused("cross-tenant approval");
if (!row.userId().equals(approver.userId()))
return new Decision.Refused("approval bound to requester");
if (!approver.scopes().contains(requiredScopeFor(row.tool())))
return new Decision.Refused("approver lacks scope");
if (!approve) {
store.transition(approvalId, Approval.Status.DENIED, approver.userId());
audit.record("DENIED", row.tenantId(), approver.userId(), row.tool(),
row.conversationId(), approvalId, row.argsHash(), Map.of());
return new Decision.Refused("denied by operator");
}
// single-use transition: exactly one racer wins
if (!store.transition(approvalId, Approval.Status.APPROVED, approver.userId())) {
return new Decision.Refused("approval already consumed");
}
audit.record("APPROVED", row.tenantId(), approver.userId(), row.tool(),
row.conversationId(), approvalId, row.argsHash(), Map.of());
// TOCTOU: hash what we are ABOUT to send; compare to what was approved
Map<String, Object> execArgs = ArgsCanonicalizer.parse(row.argsJson());
String execHash = ArgsCanonicalizer.sha256(ArgsCanonicalizer.canonical(execArgs));
if (!execHash.equals(row.argsHash())) {
return new Decision.Refused("arguments changed after approval");
}
var policy = policies.find(row.tool()).orElseThrow();
var idemKey = "ap-" + row.argsHash().substring(0, 32);
execArgs.put("idempotency_key", idemKey);
execArgs.put("tenant", row.tenantId()); // still ours, never the model's
String result = executor.execute(row.tool(), execArgs, row.tenantId(),
policy.perCallTimeout());
store.transition(approvalId, Approval.Status.EXECUTED, approver.userId());
audit.record("TOOL_EXECUTED", row.tenantId(), approver.userId(), row.tool(),
row.conversationId(), approvalId, execHash,
Map.of("idempotency_key", idemKey));
return new Decision.Executed(result);
}
private String requiredScopeFor(String tool) {
return policies.find(tool).map(p -> p.requiredScope()).orElse("__none__");
}
}

ArgsCanonicalizer sorts keys recursively and serializes compact JSON before hashing — {a:1,b:2} and {b:2,a:1} must hash identically, while {"note":"x"} and {"note":"x "} must not.

The orchestrator’s dispatch changes in one branch: requiresApproval now calls approvalService.requestApproval(...), publishes AgentEvent.ApprovalRequired over SSE, returns Outcome(status=AWAITING_APPROVAL) — the loop suspends rather than spinning. ApprovalController (POST /api/v1/approvals/{id}, scope agent:invoke) calls decide; on Executed it appends the tool result to conversation history and re-enters the loop so the model sees the outcome and writes the final answer.

MCP side — the write tools (Kotlin):

mcp-operations-server/src/main/kotlin/in/o612/eng/opsagent/mcp/tools/WriteTools.kt
package `in`.o612.eng.opsagent.mcp.tools
import `in`.o612.eng.opsagent.mcp.errors.ToolError
import `in`.o612.eng.opsagent.mcp.security.TenantPolicy
import `in`.o612.eng.opsagent.mcp.simulator.SimulatorClient
import org.springframework.ai.mcp.annotation.McpTool
import org.springframework.ai.mcp.annotation.McpTool.McpAnnotations
import org.springframework.ai.mcp.annotation.McpToolParam
import org.springframework.security.core.context.SecurityContextHolder
import org.springframework.security.oauth2.server.resource.authentication.JwtAuthenticationToken
import org.springframework.stereotype.Component
@Component
class WriteTools(
private val simulator: SimulatorClient,
private val tenantPolicy: TenantPolicy,
) {
@McpTool(
name = "create_incident",
description = "Create an incident ticket. Mutating: requires ops:incident:write.",
annotations = McpAnnotations(readOnlyHint = false, destructiveHint = false,
idempotentHint = true, openWorldHint = false),
generateOutputSchema = true,
)
fun createIncident(
@McpToolParam(required = true, description = "Tenant scope") tenant: String,
@McpToolParam(required = true, description = "Short incident title") title: String,
@McpToolParam(required = true, description = "SEV1..SEV4") severity: String,
@McpToolParam(required = true, description = "Owning service id") serviceId: String,
@McpToolParam(description = "What is known so far") description: String? = null,
@McpToolParam(required = true, description = "Caller-supplied idempotency key")
idempotencyKey: String,
): Any = guardWrites {
tenantPolicy.enforce(jwt(), tenant, "ops:incident:write")
require(idempotencyKey.matches(Regex("[A-Za-z0-9_-]{8,64}"))) { "invalid idempotency key" }
simulator.createIncident(tenant, idempotencyKey, title, severity, serviceId, description)
}
@McpTool(
name = "append_incident_note",
description = "Append a note to an existing incident. Mutating: requires ops:note:write.",
annotations = McpAnnotations(readOnlyHint = false, destructiveHint = false,
idempotentHint = false, openWorldHint = false),
)
fun appendIncidentNote(
@McpToolParam(required = true) tenant: String,
@McpToolParam(required = true) incidentId: String,
@McpToolParam(required = true) note: String,
@McpToolParam(required = true) idempotencyKey: String,
): Any = guardWrites {
tenantPolicy.enforce(jwt(), tenant, "ops:note:write")
require(note.length <= 4000) { "note too long" }
simulator.appendNote(tenant, incidentId, note, idempotencyKey)
}
private fun jwt() =
(SecurityContextHolder.getContext().authentication as JwtAuthenticationToken).token
private fun guardWrites(block: () -> Any): Any = try {
block()
} catch (e: IllegalArgumentException) {
ToolError.of("FORBIDDEN_OR_INVALID", e.message ?: "rejected")
} catch (e: org.springframework.web.client.RestClientException) {
ToolError.of("UPSTREAM_UNAVAILABLE", "operations backend unreachable")
}
}

SimulatorClient gains two methods matching the Chapter 2 wire contract — createIncident (POST /sim/v1/incidents, Idempotency-Key header, IncidentDetail body) and appendNote (POST /sim/v1/incidents/{id}/notes, Idempotency-Key + {author, body}) — both verified to compile against the same RestClient setup.

The McpAnnotations hints (readOnlyHint=false, idempotentHint=true) are client-facing metadata — honest signals, not enforcement. Enforcement is tenantPolicy.enforce plus the approval gate upstream.

API requests and expected responses

api-requests/approval.http
POST http://localhost:8080/api/v1/approvals/apr-01H...
Authorization: Bearer <priya token>
Content-Type: application/json
{"approve": true}

Full flow: message → SSE emits ApprovalRequired{approvalId, tool, argsJson, expiresAt} → POST /approvals/{id} → TOOL_EXECUTED audit row → conversation resumes → Completed event with the incident reference. Replaying the same POST returns Refused("approval already consumed"); waiting 5 minutes returns Refused("approval expired").

Automated tests

  • ApprovalServiceTest: every Refused branch — unknown id, non-pending, expired, cross-tenant, wrong user, missing scope; happy path executes exactly once.
  • ApprovalRaceIT: two threads POST approve on the same id concurrently → exactly one Executed, one Refused, one TOOL_EXECUTED audit row.
  • ArgsHashTest: key-order-insensitive equality; whitespace/extra-field sensitivity; tampered args_json detected.
  • IdempotencyIT: force a transport retry after a successful create (simulator flaky mid-response) → same incident id returns, incidents row count unchanged.
  • NegativeIT: sam (viewer, no ops:incident:write) proposing create_incident → INSUFFICIENT_SCOPE, zero pending approvals created.

Failure-injection lab

  1. Approve, then immediately POST the same approval again from two shells — confirm single consumption.
  2. Sleep past expires_at, then approve — EXPIRED, no tool call (check mcp.tool.calls stayed flat).
  3. Edit the approval row’s args_json directly in psql (change the title), then approve — refused on the hash check. This is the TOCTOU test: the human approved A, the store now says B, the execution recomputes B’s hash, sees a match… — actually read that again: editing args_json changes what hash recomputes to, so the protection lives in signing the hash or in the human seeing the same canonical form. Hardening note shipped in the repo: args_hash is computed at request time and the decision compares stored-hash vs. recomputed-hash of stored args — direct DB tampering is out of scope (if attackers write your DB, approval tokens are not your problem), but the model-side TOCTOU — model proposes A, then emits B at execution — is covered because execution args come only from the stored, approved JSON, never re-asked of the model.

Security considerations

Approval ≠ authorization: scope checks run at dispatch and at decision and inside the MCP tool. Bindings: user + tenant + conversation + tool + args-hash + expiry. Audit rows are append-only and contain hashes, not raw arguments — arguments may carry operational detail; the hash is enough to prove what was approved to an auditor holding the request. The 5-minute TTL is deliberately short; approval UX that lets a ticket sit “pending” for hours invites stale-context approvals.

Observability checks

agent.approval.outcomes labeled approved|denied|expired|consumed|tampered; agent.approval.latency (request → decision); audit events emitted to the dedicated AUDIT logger marker in addition to the table.

Checkpoint verification checklist

  • sam cannot reach execution at any layer (dispatch deny, decide deny, MCP deny).
  • Concurrent approvals: exactly one EXECUTED.
  • Idempotent retry never creates a second incident.
  • Every state transition writes an audit row.

Commit message and Git tag

feat(agent,mcp): approval-gated write tools with idempotent, arg-bound execution

git tag chapter-10-human-approval

What comes next

Chapter 11 hardens everything around the gates: prompt limits, budgets, timeouts, retry classes, and the degraded modes for when the model or a dependency is simply gone.

Project State Ledger — chapter-10-human-approval

  • MCP tools live: + create_incident (ops:incident:write), append_incident_note (ops:note:write); annotations carry readOnly/idempotent hints
  • Approvals: agent.approvals state machine PENDING→APPROVED→EXECUTED|DENIED|EXPIRED; opaque id token; TTL 5m; single-use via conditional UPDATE
  • Bindings: user, tenant, conversation, tool, canonical args hash, expiry; exec args come from stored JSON only
  • Idempotency: key ap-<argsHash32> injected by agent; simulator replays on conflict
  • Audit: agent.audit_events append-only; types APPROVAL_REQUESTED/APPROVED/DENIED/TOOL_EXECUTED
  • Next: chapter-11-guardrails-resilience
JavaKotlinSpring BootAI

Type to search the site.

↑↓ navigate⏎ openPowered by Pagefind