Event And Audit¶

Hubuum stores one canonical event stream in events. Audit reads query that stream directly, while external delivery fans matching events out through event_subscriptions and event_deliveries.

The event stream is append-only during normal application operation. Domain changes emit events in the same database transaction as the state change, so an event exists only if the change commits.

New events can also carry an internal validated OpenTelemetry trace link. It is not serialized in audit API responses. Fan-out and delivery workers use it as a span link so delayed work remains connected without persisting baggage or raw headers. See Distributed Tracing.

New event bodies are constructed as backend-neutral AuditDocument values. The type validates object-shaped snapshots and metadata and owns schema-version selection: version 1 has no numeric resource revision, while version 2 carries a positive revision in at least one snapshot. Storage adapters add entity and actor coordinates but do not define a separate document shape.

At the storage boundary, ordinary audited mutations require an explicit actor context and return a committed receipt identifying that durable event. A receipt is non-sensitive proof of persistence, not an audit projection; event snapshots and provenance remain permission-scoped reads. The normative adapter guarantees and their reusable certification are documented in the storage contract.

Audited mutation paths compare the requested state with the stored state before writing. Requests that would leave domain state unchanged are no-ops: they do not advance updated_at or append lifecycle events. This includes identical entity updates, moves to the current parent, repeated permission grants or revocations, repeated service-account disable requests, and membership changes that do not change effective membership.

Audit Log¶

Audit readers query the canonical stream with GET /api/v1/events. The endpoint is cursor-paginated and supports the normal pagination headers:

GET /api/v1/events?entity_type=collection&action=created&limit=50&sort=-occurred_at
Authorization: Bearer <token>

Supported audit filters are:

Supported sorts are id and occurred_at, with - for descending order. For example, sort=-occurred_at returns the newest visible events first.

Every audit response retains the compatibility fields actor_user_id and actor_kind and also includes shared mutation provenance:

{
  "actor_user_id": null,
  "actor_kind": "worker",
  "provenance": {
    "actor": {
      "kind": "worker",
      "principal": null
    },
    "initiator": {
      "principal_id": 1,
      "name": "admin"
    },
    "task_id": 12
  }
}

The actor is the principal or server component that performed the immediate mutation. The initiator is the principal that submitted the root task. Direct request mutations normally have a user actor and no separate initiator. Principal names are resolved when a page is read; durable IDs remain present when a principal has been deleted, while an unavailable name is null. Historical task events that predate stored initiator provenance derive it from the task's queued event with one bounded query per response page.

Audit visibility is collection-scoped:

• A caller sees collection events only for collections where the caller has ReadAudit.
• Events that reference related collections in event metadata are visible to a caller with ReadAudit on one of those related collections. Related-collection-only visibility returns the event identity, actor, summary, metadata, and schema version, but redacts before and after snapshots. A caller sees snapshots only when they also have direct ReadAudit on the event's own collection.
• Collection-less events are visible only to unscoped admins.
• Scoped tokens are constrained by both their token scope and the caller's underlying permissions.

Task lifecycle history is also stored in events. User-facing task reads should usually use GET /api/v1/tasks/{task_id}/events, which applies the task authorization model and returns task-focused history.

Convenience audit routes are available for common resources. These routes are thin wrappers around GET /api/v1/events, apply the same ReadAudit scoping, and accept the same pagination, actor, initiator, action, collection, and time filters:

GET /api/v1/collections/12/events
GET /api/v1/classes/34/events
GET /api/v1/classes/34/56/events
GET /api/v1/iam/users/78/events
GET /api/v1/iam/groups/90/events
GET /api/v1/export-templates/11/events
GET /api/v1/remote-targets/22/events

The object convenience route verifies that the object belongs to the class in the path. A mismatched class and object pair returns 404 Not Found rather than querying the object through an unrelated class URL.

Use the generic endpoint for relation, permission, token, sink, and subscription events where the useful identity is often in event metadata rather than a single resource path.

Sinks And Subscriptions¶

External delivery is configured in two layers:

• Event sinks are global transport definitions. Admins manage them through /api/v1/event-sinks.
• Event subscriptions are collection-scoped routing rules. Callers need ManageEventSubscription on the collection and manage them through /api/v1/collections/{collection_id}/event-subscriptions.
• Administrators manage collection-less events through /api/v1/system-event-subscriptions. System subscriptions match only events with neither a direct collection nor related collections.

A sink describes how to deliver. A subscription describes which events should be delivered to a sink. The primary subscription filters are entity_types and actions; Hubuum validates these against the event catalog and rejects impossible entity/action pairs at write time. Subscriptions may also include a filter object that narrows matching by stable event-envelope fields before delivery rows are created.

Supported filter fields are:

Each field is optional. Empty and omitted fields match all events for that dimension. Multiple populated fields are combined with AND; values inside one field are combined with OR. The filter can only narrow the subscription's collection-scoped visibility. It cannot deliver unrelated collection events to a subscription.

Example sink:

{
  "name": "inventory-webhook",
  "kind": "webhook",
  "config": {
    "headers": {
      "X-Integration": "inventory"
    }
  },
  "secret_ref": "inventory_webhook",
  "enabled": true
}

Example collection subscription:

{
  "sink_id": 1,
  "name": "collection-lifecycle-to-inventory",
  "description": "Send collection lifecycle events to inventory",
  "entity_types": ["collection"],
  "actions": ["created", "updated", "deleted"],
  "filter": {
    "actor_kinds": ["user"]
  },
  "routing": {
    "url": "https://inventory.example/hubuum/events"
  },
  "enabled": true
}

For email sinks, create narrow subscriptions rather than sending every audit event to human recipients. For example, this subscription sends only failed task lifecycle events to the configured mailbox. Create it through POST /api/v1/system-event-subscriptions:

{
  "sink_id": 2,
  "name": "task-failures-to-ops",
  "description": "Email ops when tasks fail",
  "entity_types": ["task"],
  "actions": ["failed"],
  "filter": {
    "actor_kinds": ["worker"]
  },
  "routing": {
    "recipients": ["Ops <ops@example.com>"]
  },
  "enabled": true
}

Both the sink and the subscription must be enabled for matching events to fan out to delivery rows. Disabling either one stops new matching deliveries without deleting historical events or existing delivery rows.

Webhook Sinks¶

Webhook delivery is the reference concrete sink. For a setup walkthrough, see chat and notification webhooks. By default, a webhook subscription posts the event envelope as JSON to the URL in its routing object:

{
  "routing": {
    "url": "https://example.com/hubuum/events"
  }
}

The request method is always POST. Hubuum sends the event UUID in both Idempotency-Key and X-Hubuum-Event-Id, and the JSON body includes the same event_id field. Consumers should deduplicate by event_id.

Webhook, AMQP, Valkey, and email deliveries all receive the same provenance object shown above. Actor and initiator names are resolved once for each claimed delivery batch. A deleted principal keeps its durable ID and has a null name when no current or tombstoned principal record can be resolved.

Webhook sink config may include static string headers and optional local delivery limits:

{
  "config": {
    "headers": {
      "X-Integration": "inventory-sync"
    },
    "timeout_ms": 5000,
    "max_response_bytes": 65536,
    "max_request_bytes": 1000000
  },
  "secret_ref": "inventory_webhook"
}

When secret_ref is set, Hubuum resolves the alias through the selected secret source and sends it as a bearer token. With the default environment source, the example maps to HUBUUM_EVENT_SINK_SECRET_INVENTORY_WEBHOOK.

Webhook HTTP execution uses the shared hardened outbound HTTP layer: HTTPS-only URLs, embedded credential rejection, DNS resolution and address screening, IP pinning, redirect refusal, timeout caps, response-size caps, and sensitive response-header redaction.

AMQP Sinks¶

AMQP delivery is available when Hubuum is built with the amqp feature. An AMQP sink publishes the event envelope as JSON to the exchange in sink config. AMQP event delivery requires TLS, so configure amqps:// URLs:

{
  "config": {
    "uri": "amqps://publisher:{secret}@rabbitmq.example/%2f",
    "exchange": "hubuum.events",
    "exchange_type": "topic",
    "declare_exchange": true,
    "durable": true,
    "mandatory": true,
    "max_payload_bytes": 1000000
  },
  "secret_ref": "rabbitmq_password"
}

When secret_ref is set, the AMQP URI must contain {secret}. Hubuum resolves the alias through the selected secret source, percent-encodes the value for URI userinfo use, and substitutes it into the URI. The default environment source maps this example to HUBUUM_EVENT_SINK_SECRET_RABBITMQ_PASSWORD. Literal credentials in sink URIs are rejected; use {secret} plus secret_ref instead.

The routing key is always {entity_type}.{action}, such as collection.created. Hubuum sets the AMQP message_id property to the event UUID and enables publisher confirms for each delivery attempt. Consumers should deduplicate by event_id or message_id.

Valkey Stream Sinks¶

Valkey stream delivery is available when Hubuum is built with the valkey feature. Hubuum uses the mature Redis protocol client for this transport, so the connection URL uses the standard Redis protocol URL form accepted by Redis and Valkey deployments. Event delivery requires TLS, so configure rediss:// URLs.

The subscription routing object selects the stream key:

{
  "routing": {
    "stream": "hubuum:events"
  }
}

The sink config holds the Valkey connection URL and optional stream trim settings:

{
  "config": {
    "uri": "rediss://default:{secret}@valkey.example/0",
    "max_len": 100000,
    "approximate_trim": true,
    "max_payload_bytes": 1000000,
    "io_timeout_ms": 25000
  },
  "secret_ref": "valkey_password"
}

When secret_ref is set, the URI must contain {secret}. Hubuum resolves the alias through the selected secret source, percent-encodes the value for URI userinfo use, and substitutes it into the URI. The default environment source maps this example to HUBUUM_EVENT_SINK_SECRET_VALKEY_PASSWORD. Literal credentials in sink URIs are rejected; use {secret} plus secret_ref instead. io_timeout_ms bounds the Redis protocol connection and socket I/O for the blocking driver call and defaults to 25,000 ms.

Each XADD entry includes discrete fields for event_id, entity_type, entity_name, action, and actor_kind, plus the full JSON envelope in the payload field. Consumers should deduplicate by event_id.

Email / SMTP Sinks¶

Email delivery is available when Hubuum is built with the email feature. This transport is intended for human-facing notifications, while the canonical event stream remains the source of truth for audit and retry state.

The subscription routing object holds the message recipients:

{
  "routing": {
    "recipients": ["Ops <ops@example.com>"],
    "cc": ["audit@example.com"],
    "bcc": ["archive@example.com"]
  }
}

to is accepted as an alias for recipients.

The sink config holds the SMTP connection URL, sender, optional reply-to address, and MiniJinja export templates for the subject and text body:

{
  "config": {
    "uri": "smtps://hubuum:{secret}@smtp.example.com",
    "from": "Hubuum <hubuum@example.com>",
    "reply_to": "noreply@example.com",
    "max_payload_bytes": 1000000,
    "subject_template": "Hubuum {{ entity_type }} {{ action }}: {{ entity_name | default_if_empty(summary) }}",
    "body_template": "{{ summary }}\n\nEvent: {{ event_id }}\nEntity: {{ entity_type }}\nAction: {{ action }}\n"
  },
  "secret_ref": "smtp_password"
}

SMTP URLs must use the TLS smtps:// scheme. Certificate verification uses the system trust store, including operator-installed private CAs (SSL_CERT_FILE and SSL_CERT_DIR on supported Unix platforms). When secret_ref is set, the URI must contain {secret}. Hubuum resolves the alias through the selected secret source, percent-encodes the value for URI userinfo use, and substitutes it into the URI. The default environment source maps this example to HUBUUM_EVENT_SINK_SECRET_SMTP_PASSWORD. Literal credentials in sink URIs are rejected; use {secret} plus secret_ref instead.

Template context exposes the event envelope fields at the top level, including event_id, entity_type, entity_name, action, summary, and occurred_at, and provenance. The full envelope is also available as event. Subjects must render to a single non-empty line, and bodies must render to non-empty text. Webhook max_request_bytes and the transport max_payload_bytes settings default to 1,000,000 bytes.

Delivery Semantics¶

Delivery is at least once. A successful transport-specific acknowledgement marks the delivery succeeded; transport errors or failed acknowledgements are retried with backoff until the configured attempt limit, then marked dead. By default, webhooks treat any 2xx response as successful and retry non-2xx responses. Optional response policies can require an acknowledgement, classify permanent errors, or defer HTTP 429.

Hubuum does not guarantee ordering across events. Consumers that need ordering should reconcile with occurred_at and the internal monotonic id, while still deduplicating by event_id.

Operators can inspect delivery rows through GET /api/v1/event-deliveries and GET /api/v1/event-deliveries/{delivery_id}. Admins can release a failed or dead delivery with POST /api/v1/event-deliveries/{delivery_id}/retry, or move a row to the dead-letter state with POST /api/v1/event-deliveries/{delivery_id}/dead.

Delivery workers are configurable and default-disabled:

HUBUUM_EVENT_FANOUT_WORKERS=1
HUBUUM_EVENT_FANOUT_BATCH_SIZE=100
HUBUUM_EVENT_FANOUT_POLL_INTERVAL_MS=5000
HUBUUM_EVENT_FANOUT_LOCK_TIMEOUT_MS=30000
HUBUUM_EVENT_DELIVERY_WORKERS=0
HUBUUM_EVENT_DELIVERY_BATCH_SIZE=100
HUBUUM_EVENT_DELIVERY_POLL_INTERVAL_MS=5000
HUBUUM_EVENT_DELIVERY_LOCK_TIMEOUT_MS=30000
HUBUUM_EVENT_DELIVERY_TRANSPORT_TIMEOUT_MS=25000
HUBUUM_EVENT_DELIVERY_RETRY_BACKOFF_BASE_MS=1000
HUBUUM_EVENT_DELIVERY_RETRY_BACKOFF_MAX_MS=300000
HUBUUM_EVENT_DELIVERY_MAX_ATTEMPTS=10

Keep delivery workers at 0 when the deployment uses the audit log only or when sink credentials are not ready. Set HUBUUM_EVENT_DELIVERY_WORKERS above zero once operators are ready for external transport delivery. HUBUUM_EVENT_DELIVERY_TRANSPORT_TIMEOUT_MS must be less than HUBUUM_EVENT_DELIVERY_LOCK_TIMEOUT_MS. Each worker claims at most min(HUBUUM_EVENT_DELIVERY_BATCH_SIZE, 8) rows, starts them concurrently, and waits for their acknowledgements before claiming more. Larger batch settings remain valid upper bounds; claimed deliveries no longer wait through local execution waves.

Immediately before transport, storage checks the claim token, in-flight status, and expiry. Expired or replaced claims do not start a send. The remaining lease is carried as a conservative monotonic deadline, including time spent waiting for the ownership check. Claim loading, dispatch delays, and the ownership check reduce the time available to transport. The difference between the lock and transport timeouts is reserved for recording success or failure (5 seconds with the defaults). Transport stops at its remaining deadline, and acknowledgment is bounded by lease expiry. Increase the lock timeout if database latency needs a larger acknowledgment allowance. No setting values or validation rules changed.

Claim-token fencing still rejects acknowledgments from replaced workers. A process interruption or lost acknowledgment after a sink accepts an event can still cause a retry; cancellation cannot undo an external side effect. Consumers must continue to deduplicate by event_id under the at-least-once contract.

Workers use PostgreSQL LISTEN/NOTIFY for low-latency wakeups across processes and fall back to the configured poll intervals for eventual progress. Event writes notify the fan-out channel only after commit, and fan-out notifies delivery workers when it creates delivery rows. An idle delivery worker also wakes at the earliest scheduled retry or expired-claim deadline, so retry backoffs shorter than the safety poll remain effective.

Operational Health¶

The admin endpoint GET /api/v1/event-deliveries/health returns a delivery pipeline snapshot for operators and dashboards. It includes:

• Fan-out backlog: undispatched events, in-flight fan-out claims, stale fan-out claims, oldest pending age, worker settings, and notification-versus-poll wakeup counters.
• Delivery backlog: status counts, retryable failed rows, stale delivery claims, oldest due age, worker settings, and notification-versus-poll wakeup counters.
• Per-sink and per-subscription delivery counts, stale claims, retryable rows, enabled flags, and oldest due age.

Use the fields together to distinguish common failure modes:

• fanout.pending_events > 0 with rising oldest_pending_age_seconds means events are being written but fan-out is not keeping up.
• Fan-out stale_claims > 0 means a worker claimed events but did not clear them before the lock expired.
• delivery.counts.pending or delivery.counts.retryable growing while fan-out is clear means delivery workers are not keeping up or are disabled.
• delivery.stale_claims > 0 means a delivery worker claimed rows and did not finish before the lock expired.
• Per-sink failed, dead, or retryable growth with other sinks healthy usually points to a sink configuration, credential, endpoint, or broker problem.
• Wakeup notifications_sent increasing while notification_wakeups stays flat and backlog only drains on poll_wakeups points to worker wakeup trouble or missing workers.

Alert thresholds are deployment-specific, but a practical baseline is to page when fan-out or delivery oldest due age exceeds the poll interval by several minutes, when stale claims remain non-zero across multiple lock-timeout windows, or when dead-letter counts grow for a production sink. First confirm worker counts and lock/poll settings in the health response, then inspect the affected delivery rows, fix the sink or worker condition, and use the retry admin action for rows that should be released from failed or dead.

Retention And Archival¶

Event retention purge is available as an operational worker, but is disabled by default because it deletes audit rows. Enable it only after choosing retention windows that match the deployment's audit requirements:

HUBUUM_EVENT_RETENTION_PURGE_ENABLED=true
HUBUUM_EVENT_RETENTION_DAYS=365
HUBUUM_EVENT_DELIVERY_RETENTION_DAYS=30
HUBUUM_EVENT_RETENTION_PURGE_INTERVAL_SECONDS=3600
HUBUUM_EVENT_RETENTION_PURGE_BATCH_SIZE=1000

The purge path is the only application path allowed to bypass the append-only events trigger. It sets the transaction-local events.allow_purge guard before deleting eligible event rows. Normal DELETE statements against events continue to fail.

Worker replicas coordinate claims with a PostgreSQL advisory lock. A claim transaction persists an immutable batch ID, the exact selected event IDs and documents, and the bounded terminal-delivery cutoff; it does not delete the events. Archive I/O then runs outside the database transaction. A completion transaction locks the same claim, deletes exactly its event IDs, performs the bounded terminal-delivery cleanup, and records the final counts. A partial index on terminal delivery age keeps that lookup bounded to its eligible queue.

If archival fails or the process exits, the pending claim and source events remain durable. The next worker receives the same batch ID and documents. Archives must therefore be idempotent by batch ID, and completion is also idempotent. A purge-count mismatch, malformed claimed document, maintenance transition, or busy coordinator is an error rather than a successful empty batch.

Events become purge-eligible after HUBUUM_EVENT_RETENTION_DAYS, but the purge will not delete an event while it has active pending, failed, or in_flight deliveries. Deleting an eligible event cascades to its remaining delivery rows. Terminal succeeded and dead delivery rows are also purged independently after HUBUUM_EVENT_DELIVERY_RETENTION_DAYS, using their updated_at timestamp so retention starts when the delivery reaches its terminal state.

Local file archival is opt-in. Most deployments should export or consume audit events through the database or event sinks instead of writing container-local files. Enable local JSON Lines archival only when the configured directory is durable and access-controlled:

HUBUUM_EVENT_RETENTION_FILE_ARCHIVE_ENABLED=true
HUBUUM_EVENT_RETENTION_ARCHIVE_PATH=/var/lib/hubuum/event-archive

Each claim is written atomically to <batch-uuid>.jsonl. Every line contains retention_batch_id, archived_at, and the full event row, including durable initiator_user_id and task_id values. Event schema_version is 2 for revision-aware mutation events; older stored events remain readable as version 1 with null revision fields. If the final file already exists, the worker accepts it only after validating its batch ID, line count, and exact event documents against the claim; any mismatch is fatal.

On Unix, the archive directory is a real non-symlink directory restricted to mode 0700. Temporary and final batch files are regular non-symlink files with mode 0600. The worker flushes and synchronizes file contents before atomic rename, then synchronizes the directory entry before acknowledging archival.

Schema lifecycle and object validation¶

class_schema events record staging, abandonment, activation, work requests, cancellation, and deletion. Activation metadata binds the old/new revisions, policy, immediate population effect, revalidation task, and dependent rebuild. object_validation uses succeeded, failed, and updated for committed valid, mismatched, and not-required findings. Impact failures are advisory and explicitly carry compliance_changed: false.

These events are also audit entries in the canonical stream. Their documents contain schema/object identities and safe categories, without object values or raw validator messages. They commit with evidence and checkpoint updates; rolled-back writes and rejected leases publish neither. The activation event records immediate pending/not-required effects; per-object findings follow in bounded batches. See class schema evolution.

Operator monitoring¶

Use the shared operator package for Grafana dashboards, Prometheus recording and alerting rules, SLO definitions and response runbooks. The same assets work with the optional single-host stack, independently managed Prometheus/Grafana installations, and Prometheus Operator. Pin the package to your server release and scrape every process directly with deployment labels.

Configurable Webhook Notifications¶

The webhook sink can send the original event envelope or render a custom JSON payload. Slack, Mattermost, and notification bridges use this same transport; there are no provider-specific sink kinds or bot API clients. Events remain raw facts: no threshold evaluator, aggregation, or alert-state machine is added. System subscriptions and task_kinds filters can select existing failed-backup events. Metrics alerting systems remain appropriate for queue delay and database pressure unless a component emits a corresponding raw event.

Payload Templates And Secrets¶

Set config.body_template to a MiniJinja template that renders valid JSON. Omitting it preserves the original event envelope. Templates expose the same bounded event context as email: envelope fields at the top level and the full original envelope as event. Use tojson for inserted values so quotes, newlines, and Unicode remain valid JSON. Templates share the isolated execution engine with fuel, recursion, context, and output limits. The rendered request also respects max_request_bytes and the global outbound request limit.

The additional context fields test and test_marker identify test deliveries. test_marker is [TEST] followed by a space for tests and an empty string for normal delivery. Templates choose where to display it; Hubuum does not inject fields into the receiver's payload. Test sends also carry X-Hubuum-Delivery-Purpose: test.

For a destination containing credentials, set config.url_secret_ref to an alias in the configured secret source. Its value must be the complete HTTPS URL. This is independent of the sink's existing secret_ref, which continues to supply a bearer token. Neither credential is available to templates or previews. A secret-backed destination requires omitting routing.url; subscriptions cannot override that destination. Without url_secret_ref, the subscription's routing.url works as before.

HTTPS validation, DNS screening, private-target policy, timeouts, response caps, and redirect refusal apply equally to literal and secret-backed URLs. Requests remain JSON POSTs; templates cannot change transport, headers, or credentials.

Slack And Mattermost Incoming Webhook Example¶

Follow Set Up Chat And Notification Webhooks for provider setup, secret storage, complete sink and subscription requests, and preview/test delivery. It includes separate Slack and Mattermost instructions, plus Discord and Apprise bridge recipes.

All examples use kind: "webhook". Keep provider credentials in the secret source, use tojson for dynamic payload values, and select an acknowledgement policy matching the destination. No provider-specific sink or bot client is required.

Response And Delivery Policies¶

Without config.response, webhooks retain their existing behavior: every 2xx is successful and every other HTTP status retries with backoff. Response rules are optional and declarative:

For body, use {"kind":"text_equals","value":"ok"} to compare trimmed response text, or {"kind":"json_equals","pointer":"/ok","value":true} to compare a JSON value using an RFC 6901 pointer. A missing value, invalid JSON, or failed comparison is a permanent failure. Response bodies that reach the configured capture limit cannot be safely checked for truncation and retry instead. Status errors and failed acknowledgements never include response contents or destination URLs. These rules do not interpret provider error names.

When rate_limit is enabled, valid Retry-After delta seconds or HTTP dates control the cooldown; invalid or absent values use 60 seconds. Transport failures remain retryable. Permanent failures dead-letter immediately. All sinks remain unlimited unless delivery_policy.min_interval_ms is configured. Values range from 1 to 86,400,000 milliseconds; null disables configured spacing.

The persisted schedule is shared across workers and subscriptions using a sink. It spaces sends without dropping or aggregating events. Deferred deliveries stay pending with deferred_reason (configured_rate or provider_rate) and next_attempt_at, without consuming failure attempts. Separate sinks do not share their schedules, even if they target the same external channel.

Delivery remains at least once: a received request followed by a lost acknowledgement can produce a duplicate. Custom payloads may include event_id for receiver deduplication; X-Hubuum-Event-Id always identifies the source event. Normal deliveries keep the event UUID as Idempotency-Key. Queued tests use {event_uuid}-test-{delivery_id}, stable across retries and distinct for each test, so tests cannot suppress normal sends at receivers using that header.

Preview And Test¶

Administrators can preview or enqueue a webhook test using an existing event UUID and saved subscription. Preview resolves no secrets and performs no HTTP request; it returns only the JSON payload and sink kind:

POST /api/v1/event-sinks/3/preview
Content-Type: application/json

{"subscription_id": 7, "event_id": "00000000-0000-4000-8000-000000000001"}

Use /api/v1/event-sinks/3/test with the same body to enqueue a delivery. Replace the example identifiers with real saved records. Both verify sink and event scope. Tests bypass filters and enabled flags for setup but respect throttling. Preview renders the same test context as delivery. Include test_marker in templates when a visible message label is wanted.

The test route returns 202 with an inspectable delivery and Location header. The request is audited as event_sink.invoked. Repeated tests create separate purpose: "test" records without suppressing normal event delivery. Workers render using the saved configuration at execution time.

Upgrade And Rollback¶

Stop older API and worker processes, apply 2026-10-01-000001_webhook_notifications, reconcile database role grants, and start matching binaries. Mixed old/new workers are unsupported because normal and test deliveries now use purpose-aware uniqueness.

Binary-only rollback to 0.0.16 is unsupported: its event worker's insert conflict constraint no longer exists after this migration. Before upgrading, stop all writers and take a PostgreSQL snapshot, retaining the old binaries and credentials. For recovery, stop the candidate, restore that snapshot, and start matching old APIs, workers, and restore executors. Writes after the snapshot are lost. The single-host updater selects downtime using the candidate's hubuum-admin --migration-mode preflight and leaves old binaries stopped if migration fails. The updater does not create the required snapshot for you.

Manual schema rollback is a separate, destructive operation. Its guard rejects enhanced webhook configuration, system subscriptions, rate policies, and test/deferred deliveries. Archive and remove retained event_sink.invoked audit records and their dependent deliveries with processes stopped before rollback. Format 7 backups preserve these settings and terminal delivery history, while transient scheduling resets on restore. Format 6 backups restore legacy defaults. Remove enhanced webhook settings before using older binaries, which cannot interpret their request or response rules.