#Observability

Send the harness's logs, traces and metrics to a backend you run or subscribe to (Grafana Cloud, Datadog, Honeycomb, a local OpenTelemetry Collector, or any other OTLP receiver), over OpenTelemetry's OTLP/HTTP protocol. It is off until you configure it: with no saved and enabled observability module, and no COLONIZER_OBSERVABILITY=on, nothing is written and nothing leaves the machine. The header a backend wants is kept in a secret, never in a setting or a log line.

This is separate from the live map and usage data, which report to colonizer.dev. Observability goes only to the endpoint you name.

• Settings and privacy: every setting of the module.
• Export policy and OTLP encoding: how every exported string is allowlisted, gated, redacted again, capped and hashed, and how requests are batched and encoded.
• The tailer: how the exporter reads the ledgers and survives rotation.
• Traces: one trace per colony, its span names, attributes and ids.
• Metrics (GET /metrics): the Prometheus catalogue the mothership can serve, how to scrape it, and what a series is allowed to name.
• Environment configuration: the OTEL_* variables, their precedence, and what an upgrade changes.
• Architecture: the design decisions behind it (issue #839).

#How it runs

The exporter is a separate binary, colonizer-observability, so the mothership itself links none of OpenTelemetry, protobuf or gzip, and a fault in the exporter cannot touch a colony. When export is on, the mothership writes <data>/observability/exporter.json (the effective settings and the colony list, never a header), starts the add-on, and hands it the headers as one line on its stdin. The add-on tails the ledgers the mothership already writes, maps them, sends them, and commits its read position to <data>/observability/state.json only after the backend acknowledged the batch.

Nothing in a colony's path waits on it: the ledgers are the spool. Lines older than Max backlog (days) (7 by default) are dropped instead of sent late, oldest first, and counted. A restart resumes from the committed offsets, and every record carries a deterministic colonizer.record.id, so a replay can be deduplicated.

Start from settles where a new endpoint starts: now (the default) sends only what is written from the moment it is configured; backlog also sends what the ledgers already hold, back to the backlog limit. It is settled once per endpoint, so a restart never skips anything, and a colony that appears later is always read from its first line. Each tick shares the read budget (Max read rate) equally among the files with something new, so one busy colony cannot hold back the rest. A colony that is finished (merged, no changes, stopped or failed) and fully read is not looked at again until its status or its directory changes. The tailer has the details.

#When the backend says no

The exporter holds at most one tick's batch in memory and reads nothing new while that batch waits. Its read offsets are committed, in one atomic write with the metric series, only after the backend acknowledged the whole batch; until then a restart or crash replays it from the ledgers.

#Health

The add-on writes <data>/observability/status.json at most once a second, and GET /api/observability/status returns it as exporter. state is one of:

It also carries endpoint, last_error (redacted, header values cut out), last_success_unix, consecutive_failures, exported, bytes_sent (since the add-on started), backlog_bytes (ledger bytes behind the committed offsets), dropped by reason, export_failures, last_partial_success, and signals: per signal (logs, traces, metrics) its state (ok, backing_off, auth_failed, off), endpoint, consecutive_failures, last_error, last_success_unix, records_sent and bytes_sent.

The add-on is found, in order, at COLONIZER_OBSERVABILITY_BIN, <data>/addons/observability/<version>/colonizer-observability, or beside the colonizer binary. Its version must match the mothership's. Release tarballs do not ship it yet; build it with:

cargo build --release -p colonizer-observability
cp target/release/colonizer-observability "$(dirname "$(command -v colonizer)")/"

GET /api/observability/status says whether export is on, why not, where each setting came from, the header names (never values) and the exporter's counts. POST /api/observability/test sends one log record and one metric point and reports what the backend answered. Both are in the cockpit at Settings → Modules → Observability, with a headers field and a Send test button.

#Configuration

Configure it in the cockpit (the module) or with the standard OpenTelemetry variables. A variable overrides its one field of the saved module; it never turns export on by itself, so an OTEL_EXPORTER_OTLP_ENDPOINT meant for another program in the same environment sends nothing.

The endpoint rules of a save apply to the effective value too: http:// only to a loopback or private address unless Allow plain http to a public host is on, and no credentials or query string in the URL.

The headers are the one secret: save them under Settings → Secrets → Observability headers (or in the observability pane) as a k=v,k2=v2 list. The value goes to the add-on on its stdin, never into exporter.json, its arguments or its environment, and the add-on cuts it out of any error text before it reaches status.json or a log line.

#What is exported

Every record goes through the export policy: only allowlisted attributes, every string redacted again with the same detectors the harness uses on disk, capped, and repository names hashed under repo_names = hashed. Conversation content (prompts, completions, tool input and output, error text, file paths, titles) is never exported by this build, whatever the content switches say.

Resource: service.name, service.version (the mothership's version), service.instance.id (the install's host id), colonizer.fleet.id, plus OTEL_RESOURCE_ATTRIBUTES. Scope: colonizer.

#Logs

Each record has colonizer.record.id, colonizer.source, colonizer.stream (operational, activity or meta), and, when it belongs to one, colonizer.colony.id, colonizer.org and colonizer.repo. The Operational logs and Colony activity switches gate their streams; a stream that is off is never read.

Streaming text deltas are skipped. The decision and Jev ledgers name their colony in the row's session, which becomes colonizer.colony.id (with its org and repo). An activity line's detail is content for every kind but four whose text the harness writes itself: decision.shadow, decision.act and decision.fallback (the point and its pick) and outcome.suspended, which send it as summary. The meta stream is on whenever either log stream is.

An export_gap record marks a hole in what the exporter could read: reason (backlog, rotated_past, truncated, deleted, oversized_line), file, colonizer.colony.id, and, where known, bytes and lines skipped, and for a deleted colony whether archived holds a bundle of it.

#Metrics

Cumulative, pushed every 30 seconds while Metrics is on. The sums survive a restart: they are committed with the read offsets that produced them.

#Traces

One trace per colony while Traces is on: the root invoke_agent <repo> span from launch to outcome, a turn <n> span per turn, an execute_tool <tool> span per tool call, and a subagent <type> span under the Task call that started it, a question span from a question to its answer, a chat <model> span per gateway request and a host_step <step> span per host-chain verdict. Max trace size keeps each trace under Tempo's limit. Ids are derived from the install's host id, the colony id and the turn number or tool call id, so a replay sends the same spans. A span goes out when it ends; the root goes out once, at the colony's outcome. Structure only, never content. Traces has the span names, attributes, ids and sampling.

#Examples

These follow each vendor's public OTLP documentation; check it for your region or site.

A local OpenTelemetry Collector, which can then fan out to anything it supports:

COLONIZER_OBSERVABILITY=on
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318

Grafana Cloud. On your stack's OpenTelemetry connection page, copy the OTLP endpoint (of the form https://otlp-gateway-<zone>.grafana.net/otlp), the instance ID and an access token. The header is HTTP Basic auth of <instance id>:<token>, base64-encoded, with the space percent-encoded:

endpoint:  https://otlp-gateway-<zone>.grafana.net/otlp
headers:   Authorization=Basic%20<base64 of instance-id:token>

Honeycomb. The endpoint is https://api.honeycomb.io (US) or https://api.eu1.honeycomb.io (EU), and the header carries an ingest API key. Honeycomb files metrics under a dataset you name:

endpoint:  https://api.honeycomb.io
headers:   x-honeycomb-team=<ingest key>,x-honeycomb-dataset=colonizer-metrics

Datadog. The simplest path is the Datadog Agent's OTLP receiver, enabled as Datadog's OTLP Ingestion by the Datadog Agent documentation describes, on the same host:

COLONIZER_OBSERVABILITY=on
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318

To send without an Agent, take the per-signal intake URLs Datadog documents for your site, set OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT to them, and put dd-api-key=<API key> in the headers.

#Turning it off

Switch the module off in the cockpit, or set OTEL_SDK_DISABLED=true. The mothership closes the add-on's stdin; it commits its offsets and exits. Nothing more is sent.