Observability — Structured Logging & OpenTelemetry¶

tuvl ships enterprise-grade observability out of the box: structured JSON logging via structlog and distributed tracing via OpenTelemetry (OTel). Both are active in production and automatically disabled in dev mode.

Table of Contents¶

1. Architecture Overview
2. Structured Logging
3. Distributed Tracing
4. Span Hierarchy
5. LiteLLM GenAI Telemetry
6. Metrics
7. Configuration Reference
8. Collector Setup
9. Dev Mode Behaviour
10. Data Masking & PII

1. Architecture Overview¶

trigger (HTTP / API / MCP / schedule / child run)
    │  FastAPI HTTP span (W3C traceparent propagated)
    ▼
tuvl.run                      one span per execution of a run (each claim: start, resume)
├── tuvl.agent <id>           one per agent invocation
│   └── litellm.completion    gen_ai.* spans for model calls (llm, loop turns, decide models, judges)
└── tuvl.agent <id> …

Two complementary records:

• The journal (tuvl_system_run_events) is the authoritative, durable record of what a run did — every agent start/finish, signal, model turn, tool call, decision, judge verdict, approval and pause. It drives the Insight Runs page, GET /api/runs/{id}/events (SSE), tuvl runs tail, replay and test fixtures. See runtime.
• OpenTelemetry spans and structured logs feed your tracing and log backends, correlated by trace_id/span_id and run_id.

2. Structured Logging¶

tuvl logs with structlog: JSON lines in production, a readable console format in development.

Log format — production¶

{
  "event": "Workflow execution started",
  "level": "info",
  "logger": "tuvl.core.engine.runner",
  "timestamp": "2026-05-26T10:00:00.000000Z",
  "workflow": "recruitment-pipeline",
  "trace_id": "4a8f3e...",
  "span_id": "8b2c1d..."
}

Log format — development (TUVL_ENV=development)¶

2026-05-26T10:00:00Z [info     ] Workflow execution started   workflow=recruitment-pipeline trace_id=4a8f3e...

Key log events¶

Per-run detail (what each agent did) is in the journal, not the logs.

Logging from code agents¶

from tuvl import agent, Ctx


@agent("enrich")
async def enrich(inp, ctx: Ctx):
    ctx.log.info("enrich.lookup", candidate_id=str(inp.candidate.id))  # bound to run_id and agent
    ...

Pass data as keyword arguments, never f-strings, so values are structured fields.

3. Distributed Tracing¶

tuvl uses the OpenTelemetry SDK with an OTLP exporter; init_telemetry() installs the TracerProvider at startup. Inbound W3C traceparent headers make run spans children of the caller's trace.

Span attributes set by tuvl¶

Context values are never span attributes, so secure fields can't leak into traces.

4. Span Hierarchy¶

A run that pauses (a human form, a tool approval, a breakpoint) and resumes on another worker produces one tuvl.run span per execution segment, all carrying the same tuvl.run_id; group by it to see the whole run. Engine work — model calls, HTTP calls, DB queries — nests under the agent's span.

5. LiteLLM GenAI Telemetry¶

When telemetry is enabled, tuvl registers LiteLLM's OpenTelemetry callback (litellm.callbacks = ["opentelemetry"]). Model calls appear as spans with the GenAI semantic conventions — gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reason — nested under the tuvl.agent span that made them.

6. Metrics¶

Run-level numbers (runs by status and end, tokens per agent, decision sources, approvals) come from the journal and the runs table: GET /api/runs, tuvl runs list --json, or SQL over tuvl_system_runs / tuvl_system_run_events. The one OTel counter is tuvl.hook.events (meter tuvl.hooks), incremented when a type: hook artifact with action: metric fires (tuvl.hook.name, tuvl.hook.event, tuvl.workflow.name).

7. Configuration Reference¶

All settings are read from .env or environment variables. Settings also support a per-project YAML override file at .tuvl/telemetry.yaml (see below).

Environment variables¶

.tuvl/telemetry.yaml override¶

Values from this file are applied only when the corresponding environment variable is not set, making it suitable for checked-in defaults that operators can override at deployment time.

kind: TelemetryConfig
version: v1
spec:
  enabled: true
  otlp_endpoint: http://otel-collector:4317
  service_name: my-tuvl-app

Config resolution order¶

For each setting, the first non-empty source wins:

1. Environment variable (TUVL_TELEMETRY_ENABLED, OTEL_EXPORTER_OTLP_ENDPOINT, TUVL_OTLP_ENDPOINT, TUVL_SERVICE_NAME)
2. .tuvl/telemetry.yaml spec fields
3. Hardcoded defaults (enabled=true, endpoint=http://localhost:4317, service=tuvl)

8. Collector Setup¶

tuvl exports spans over gRPC to any OTLP-compatible collector. Below are quick-start configurations for common backends.

OpenTelemetry Collector (recommended)¶

# docker-compose.yml
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    ports:
      - "4317:4317"   # gRPC OTLP
    volumes:
      - ./otel-config.yaml:/etc/otelcol/config.yaml

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"  # Jaeger UI

# otel-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [jaeger]

# .env
TUVL_OTLP_ENDPOINT=http://localhost:4317
TUVL_SERVICE_NAME=my-app

Grafana Tempo (via Alloy)¶

TUVL_OTLP_ENDPOINT=http://alloy:4317
TUVL_SERVICE_NAME=my-app

Honeycomb¶

OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io
# Set OTEL_EXPORTER_OTLP_HEADERS with your API key via collector or proxy

Datadog¶

Use the Datadog Agent as an OTLP collector:

TUVL_OTLP_ENDPOINT=http://datadog-agent:4317

9. Dev Mode Behaviour¶

When TUVL_DEV_MODE=true (set automatically by tuvl dev):

• init_telemetry() is a no-op — no TracerProvider is installed
• The OTel SDK default NonRecordingSpan is returned for all start_as_current_span calls — all span operations become no-ops with negligible overhead
• The LiteLLM OTel callback is not registered
• Logs use the coloured ConsoleRenderer instead of JSON
• trace_id / span_id are not injected into log events (no valid span context)

Dev mode never runs with TUVL_ENV=production — that combination aborts at boot.

10. Data Masking & PII¶

Fields declared secure: true in a ModelDefinition:

# models/employee.yaml
spec:
  fields:
    - { name: national_id, type: string, secure: true }

• are redacted ("*****") in journal events, SSE streams, run exports, pinned fixtures and the Insight views; checkpoints keep real values because they are the resume state;
• are never span attributes (tuvl puts no context values on spans);
• are not sent to models (llm, loop, judges, decision models) unless policy.allow_secure_to_llm: true — a secure field in a model agent's inputs is a V020 error.

The secure-field registry is built at startup from every loaded ModelDefinition.