Structured Logging¶

Hubuum writes newline-delimited JSON logs only. JSON is the stable operational interface for containers, collectors, and command-line tooling; there is no text formatter toggle. Configure verbosity with HUBUUM_LOG_LEVEL.

Configuration¶

Common Fields¶

Every log record includes:

Request-scoped records also include request_id and, when accepted from the client, correlation_id. Authenticated requests record principal_id on the request span after bearer token resolution. When an OpenTelemetry span is active, records also include fixed-width lowercase trace_id and span_id fields. See Distributed Tracing for export and propagation policy.

The server emits one server startup record at INFO after binding succeeds. It includes the package version, build Git SHA, bind address, TLS state, worker counts, storage and authorization backends, log format and level, and the number of enabled event sinks. Release and container builds populate git_sha; local builds report unknown unless HUBUUM_BUILD_GIT_SHA is set while compiling.

db_backend remains as a compatibility alias for storage_backend while the only selectable implementation is PostgreSQL.

Request Logs¶

Request completion is the canonical HTTP request log event. Actix's default text request logger is disabled to avoid duplicate unstructured logs.

Completion records include:

Severity is derived from the outcome:

Hubuum applies the status-to-severity mapping to downstream service errors as well as normal responses. Successful liveness and readiness probe completions use DEBUG to avoid flooding normal operational logs; failed probes retain their status-derived severity. Early middleware rejections, including client-allowlist denials, are returned as responses so they receive the same completion event and x-request-id header as handler responses.

Clients may send X-Correlation-ID. Accepted values are 1 to 128 visible ASCII bytes without whitespace. Hubuum echoes accepted values as x-correlation-id; invalid values are ignored without logging or echoing the supplied value. Hubuum always returns x-request-id.

Operation And Authorization Logs¶

Domain mutation logs are queued by the audit event writer and emitted at INFO only after the surrounding database transaction commits. Failed and rolled-back transactions discard their queued mutation logs. These logs use the audit catalog labels:

Service list/get paths log at DEBUG. The audit-event query path additionally uses the standardized operation=read helper with optional catalog entity, action, and entity ID filters.

Authorization decision logs use:

Authorization records include event_type=authorization, decision, principal_id, requested permissions as a JSON array, derived action and entity_type when the requested permissions share them, nullable collection_id and collection_count, and a short reason.

Storage Diagnostics¶

Logical storage calls run inside a storage_operation debug span with the bounded fields backend, capability, and operation. Successful calls emit storage operation complete at DEBUG. Expected domain outcomes such as not found, conflict, validation, and stale preconditions emit storage operation rejected at DEBUG. Database, unavailable, and internal failures emit storage operation failed at WARN.

PostgreSQL pool checkout and helper boundaries use the corresponding storage backend connection acquired, storage backend operation complete, and failure events. These include the bounded caller and operation fields, plus elapsed milliseconds. They do not include SQL text or arguments.

Use the lifecycle events to debug application use cases and the PostgreSQL events to debug connection or transaction behavior. A lifecycle call may contain more than one implementation-level operation.

Sensitive Data Rules¶

Do not log secrets or high-volume payloads. In particular, logs must not include bearer tokens or token fragments, token hashes, password hashes, sink secrets, raw Authorization headers, request bodies, response bodies, audit before or after snapshots, or remote target secret material.

Prefer stable identifiers, catalog labels, counts, and short reason strings over raw payload data.

jq Recipes¶

Pretty-print logs:

jq . hubuum.log

Show failed requests:

jq 'select(.message == "request complete" and (.status >= 500 or .severity == "ERROR"))' hubuum.log

Trace one request:

jq 'select(.request_id == "REPLACE-WITH-REQUEST-ID")' hubuum.log

Trace a client-supplied correlation ID:

jq 'select(.correlation_id == "REPLACE-WITH-CORRELATION-ID")' hubuum.log

Find all local records for an OpenTelemetry trace:

jq 'select(.trace_id == "REPLACE-WITH-32-HEX-TRACE-ID")' hubuum.log

List authorization denials:

jq 'select(.event_type == "authorization" and .decision == "deny")' hubuum.log

Operator monitoring¶

Use the shared operator package for Grafana dashboards, Prometheus recording and alerting rules, SLO definitions and response runbooks. The same assets work with the optional single-host stack, independently managed Prometheus/Grafana installations, and Prometheus Operator. Pin the package to your server release and scrape every process directly with deployment labels.