Claude Code OpenTelemetry for QA gives test teams a way to inspect what an AI coding workflow actually did—not just whether its final message sounded confident. A useful audit covers tool permission decisions, tool outcomes, API failures, cost and token signals, session correlation, missing events, privacy settings, and the human decision that follows.

This tutorial builds a small, synthetic audit exercise. The objective is not to monitor employees or capture every prompt. It is to prove that an approved agent workflow produces complete, minimally sensitive evidence and that prohibited actions remain visible and blocked.

What the official Claude Code documentation supports

Anthropic’s monitoring documentation describes OpenTelemetry export for metrics and events. Telemetry must be enabled explicitly. Metrics and logs can use console, OTLP, or supported metric exporters, and OTLP needs an endpoint and protocol configuration.

The documented metrics include session count, lines changed, pull requests, commits, estimated cost, tokens, code-edit permission decisions, and active time. Event records provide ordered evidence for API requests and errors, tool decisions, tool results, and other agent activity. Shared attributes can include a session identifier and optional organizational resource attributes.

Content-heavy fields are intentionally gated. User prompts, assistant responses, tool details, tool input/output content, and raw API bodies are disabled unless their separate flags are enabled. Raw bodies can contain the entire conversation history. For QA, the safest default is metadata-only collection with synthetic fixtures; enable content only in a controlled privacy test.

Build the audit fixture

Create a disposable Git repository with three files: a tiny implementation, a deterministic test suite, and a policy-protected file named release-approval.txt. Seed one safe edit request, one denied write request, and one command that returns a controlled failure. Use invented paths, users, tokens, and test data.

Define the expected sequence before running Claude Code:

  1. A session begins with a stable test-run identifier.
  2. The safe read is allowed and completes successfully.
  3. The safe edit follows the configured approval route.
  4. The protected write is rejected.
  5. The controlled command failure produces a failed tool result.
  6. No content-bearing telemetry is exported.
  7. The deterministic test suite and a human reviewer determine the outcome.

Step 1: start with a local collector and metadata only

Use a local or isolated test collector rather than your production observability backend. Enable telemetry, select metrics and logs, and configure the OTLP protocol and endpoint according to the official page. Keep prompt, response, tool-detail, tool-content, and raw-body flags unset.

Capture collector configuration as code and record its hash with the test artifact. Never put collector credentials directly into screenshots or repository files. If your organization uses managed settings, test that a developer cannot silently redirect telemetry away from the approved destination; Anthropic documents managed handling for exporter endpoints, protocols, and credentials.

Step 2: add low-cardinality test labels

Use OTEL_RESOURCE_ATTRIBUTES only for bounded values such as environment=qa, suite=agent-audit, and team.id=quality. Do not add branch names, defect text, timestamps, UUIDs, or user-entered prompts as metric labels.

The monitoring guide exposes controls for session, version, account, entrypoint, and resource attributes on metrics. Lower cardinality generally reduces storage and query cost. Verify both modes: one run with the minimum identifiers needed for correlation, and one with session identifiers excluded from metric labels while retaining event-level correlation.

Step 3: validate the tool-decision trail

Run the synthetic workflow and collect the exported events. Build a QA evidence table keyed by session and event sequence. For every requested tool action, assert that a decision exists, its source is expected, and a result follows only when execution was permitted.

Scenario Expected evidence Independent proof
Allowed read Allowed decision and successful result Target content unchanged
Approved edit Expected approval source and successful result Diff matches the fixture
Protected write Rejected decision; no successful execution Protected-file hash unchanged
Failing command Execution failure with error category Known nonzero process result

Do not infer a block from the absence of a result alone. An exporter failure or dropped batch can create the same appearance. Cross-check filesystem hashes, process outputs, and collector health.

Step 4: test hooks and permission controls separately

The hooks reference explains that hooks receive JSON and can affect behavior according to their event and output. A blocking exit is distinct from a conventional failure, and malformed hook output can be non-blocking for some paths. Write explicit tests for allow, block, malformed JSON, timeout, and unexpected exit behavior.

The permissions documentation describes permission modes such as read-only planning, fixed headless tool surfaces, and highly permissive bypass behavior. Keep the audit fixture in a controlled environment and test deny rules and hooks even if a permissive mode is involved. Observability records behavior; it does not create the security boundary.

Step 5: inject exporter and network failures

Stop the collector during a run, return an authentication error, use a wrong protocol, delay responses, and restore the collector. Verify that the workflow’s test result is not falsely reported as fully audited when telemetry is incomplete. Create a separate state such as execution completed, evidence incomplete.

Check retry volume, duplicate event handling, out-of-order arrival, and batch flush at process exit. Event sequence is useful within a session, but your consumer should tolerate transport reordering and deduplicate by stable keys. Alert on a missing terminal result rather than inventing one.

Step 6: run privacy canary tests

Insert obvious synthetic canaries in the prompt, tool input, tool output, and environment. With content flags disabled, assert that canaries do not reach exported attributes. Then, only in an isolated test, enable one content flag at a time and verify the documented exposure. Delete those artifacts after the privacy test according to your retention policy.

Pay special attention to identity and workspace attributes. Treat email addresses, account identifiers, file paths, commands, prompts, and model output as potentially sensitive. Limit backend access, encrypt transport, define retention, and document who can query the audit data.

Step 7: build QA dashboards that answer testable questions

  • What percentage of tool requests were allowed, rejected, or failed?
  • Which decision sources approved code edits?
  • How often does a session finish without a complete evidence chain?
  • Which failure categories are increasing?
  • Are token, cost, or active-time distributions changing after a prompt update?
  • Did any content canary appear when metadata-only collection was expected?

Avoid ranking individual engineers by token count or lines changed. Those measures do not establish quality. Use them to spot workflow regressions and investigate system behavior.

Step 8: define pass/fail criteria

A run passes the audit only when every expected decision is present, allowed actions have matching results, denied actions have no side effects, controlled failures are classified correctly, required test labels are present, forbidden content is absent, and deterministic tests match the stored evidence.

If telemetry is missing, duplicated beyond reconciliation, or sent to an unapproved destination, mark the audit inconclusive or failed. A clean dashboard is not evidence if the exporter never received the session.

Screenshot plan

Capture the synthetic repository, expected-event matrix, redacted collector configuration, metric-only dashboard, ordered tool decisions, rejected protected write, failed command result, collector outage alert, canary scan, and final human review. Crop or blur machine names, emails, endpoints, credentials, real paths, and customer data.

Release checklist

  • Official monitoring behavior was reviewed for the installed release.
  • The fixture is disposable and contains only synthetic data.
  • Collector destination and protocol are controlled.
  • Content-bearing flags are off by default.
  • Labels have bounded cardinality.
  • Decision and result events reconcile.
  • Denied actions leave no side effects.
  • Exporter outages produce an incomplete-evidence state.
  • Retention and access controls are documented.
  • A QA engineer reviews the evidence before merge or release.

Important limitations

OpenTelemetry proves what was exported, not everything that happened. Export loss, configuration drift, unsupported backends, clock differences, and disabled signals can leave gaps. Metrics such as cost, tokens, or lines changed also do not prove correctness. Keep deterministic tests, repository diffs, permission rules, hook tests, and human review as independent controls.

Detailed tracing and some attributes are documented as gated or beta behavior, so this tutorial centers on stable metrics and events. Recheck Anthropic’s official pages before adopting any gated field in a production quality gate.

Official sources