Skip to content

Releasing Hubuum

This repository uses the CI workflow in .github/workflows/ci.yml for both validation and publishing.

What the workflows enforce

  • Cargo.toml package version must match the release tag.
  • CHANGELOG.md must contain a section for the release version.
  • docs/openapi.json must be regenerated for the release version.
  • docs/operational-contract.json must be regenerated for the release version.
  • A version bump in Cargo.toml must come with matching changelog, OpenAPI, and operational-contract updates.
  • The candidate OpenAPI contract must have no unaccepted breaks from the immediately preceding stable release.
  • The candidate must pass the adjacent stable release upgrade and declared recovery harness against an immutable release-image digest.
  • The candidate must restore current, adjacent-release, and history-free backup artifacts into isolated databases and pass restored API and worker smoke checks.
  • Every Rust package must retain an explicit support classification; internal packages cannot be published, and any supported package must pass rustdoc, clean packaging, and semantic compatibility checks.

Scripted release flow

Use the helper script in scripts/release.sh, replacing X.Y.Z with the new release version:

  1. Start from a clean local main.
  2. Run ./scripts/release.sh prepare X.Y.Z.
  3. Review the generated release branch release/vX.Y.Z, including the full Cargo.lock dependency refresh, and polish CHANGELOG.md if needed.
  4. Update README.md for the new release: version and date, pinned container example, release links and highlights, and changed installation or upgrade requirements. Keep these aligned with Cargo.toml, CHANGELOG.md, and the release artifacts. This review is required for every release; the helper script does not update the README automatically.
  5. Commit the changes, then open and merge that release branch.
  6. Wait for successful CI on the exact merged main commit, check out that commit on clean main, and run ./scripts/release.sh tag.
  7. Push the new tag with git push origin vX.Y.Z.

Do not tag a different commit while CI is still running: the tag workflow requires the exact tagged commit to have a successful main CI run.

The helper script:

  • creates the release/vX.Y.Z branch from main
  • updates Cargo.toml
  • updates all Cargo dependencies to the newest versions allowed by the workspace manifests
  • rolls the current Unreleased changelog notes into the new release section
  • regenerates docs/openapi.json and docs/operational-contract.json
  • runs the existing release validation scripts before you commit or tag

Once the tag is pushed, the CI workflow will:

  • verify the tagged commit already passed CI on main
  • regenerate OpenAPI and compare it with the immediately preceding stable tag
  • resolve the latest stable release, migrate its representative data under live API probes for rolling upgrades or an offline snapshot for incompatible migrations, exercise the declared recovery procedure, and prove the candidate can recover both adjacent-release and current backup artifacts before publishing
  • verify that the tag, Cargo.toml, changelog, and OpenAPI versions match
  • validate Rust API classifications and any supported crate compatibility
  • publish GitHub release archives and SHA-256 checksums for Linux x86_64, Linux ARM64, Windows x86_64, and macOS ARM64
  • use the matching changelog section as the GitHub Release notes
  • publish AMD64 and ARM64 GHCR images for the release tag

Rust package compatibility

The eight crates in the experimental storage adapter SDK are public and allow crates.io publication. They are released together under the separate process documented in Storage Adapter SDK Compatibility. All other workspace packages remain internal and set publish = false. The authoritative classification, package list, and promotion process are documented in Rust API Boundary.

CI rejects missing classifications and any internal package that enables Cargo publishing. An experimental-public or stable-public package must allow crates.io publishing with publish = true or an allowlist containing crates-io. It is automatically checked with the pinned cargo-semver-checks version, rustdoc warnings denied, all features, and Cargo's clean packaged-source build. Promote a package only in a dedicated change containing its API policy, release owner, versioning rules, and downstream migration or compatibility fixtures.

For an initial public release, CI records the absence of a crates.io baseline and skips only semantic comparison; rustdoc and clean packaging remain mandatory. Once the first crates.io release exists, the semantic compatibility check is mandatory, and registry lookup errors fail rather than bypass the check.

OpenAPI compatibility gate

The OpenAPI contract job treats two independent failures as release blockers. The generated document must exactly match docs/openapi.json, and the generated candidate must be compatible with the latest stable release. A tag build excludes its own tag while resolving the baseline, so it compares with the immediately preceding stable release. If the repository has no stable release yet, the structured report records an explicit skipped baseline rather than silently choosing another source.

The job installs checksum-pinned oasdiff, records its version and binary digest, and publishes one openapi-contract artifact containing:

  • the generated OpenAPI document and exact drift diff;
  • the baseline document, tag, source URL, and SHA-256 digest;
  • raw structural and classified changes;
  • compatibility.json; and
  • summary.md, grouped into additive, behavioral, and breaking changes.

Breaking findings fail the job unless each fingerprint is listed in .github/openapi-breaking-exceptions.json. An exception must name the exact baseline, have a unique stable identifier and future expiry date, explain the decision, provide client migration guidance, and point to matching text in the [Unreleased] changelog. Exceptions are never wildcards: every fingerprint must still be present, and unused fingerprints fail while their baseline is current. An exception for an older baseline becomes inactive automatically when a new stable release is published.

During normal development the policy validates [Unreleased]. After release.sh prepare moves those notes into the new version section, release-PR and tag checks also validate that exact candidate-version section. This keeps the documented approval attached to the release without weakening ordinary PR checks or searching unrelated historical notes.

For an intentional pre-1.0 break:

  1. Add a clearly marked breaking entry and migration steps to [Unreleased].
  2. Run the compatibility check to obtain the exact finding fingerprints.
  3. Add the smallest coherent exception group with those fingerprints, the current stable baseline, rationale, migration text, changelog marker, and a review expiry.
  4. Review the uploaded Markdown and JSON reports before merging.

Run the policy fixtures locally with:

oasdiff_bin="$(scripts/install-oasdiff.sh /tmp/hubuum-oasdiff)"
OASDIFF_BIN="$oasdiff_bin" scripts/test-openapi-compatibility.sh

Certified Upgrade And Rollback Window

CI certifies exactly one adjacent application transition: the latest stable release (N-1) to the candidate (N). It resolves N-1 through the GitHub release API, pulls the release image, records its immutable digest, and tests it with the candidate against one PostgreSQL database. The report records the candidate SHA, migration set and duration, maximum observed API latency and outage, the terminal test phase, and failure logs.

The candidate's hubuum-admin --migration-mode inspects applied migration history and reports rolling or offline. Unknown applied migrations fail preflight rather than approving a downgrade. The compatibility report records this mode and the recovery procedure actually tested.

For rolling-compatible transitions, CI drains old workers, probes the old API while migrating, and exercises cross-version reads and writes. It then restarts both the old API and worker against the migrated database, verifies event fanout and a completed export task, and resumes the candidate.

For offline transitions, CI stops the old API and worker before taking a PostgreSQL snapshot and applying migrations. It restores that snapshot before restarting and exercising both old processes. No zero-outage or binary-only rollback guarantee applies; latency/outage probe fields are null. In particular, 0.0.16 to 0.0.17 requires this offline sequence. The webhook migration removes a constraint used by old event workers, so restarting old binaries alone is unsafe. Recovery loses writes made after the snapshot. Operators must retain old binaries, credentials, and a verified database snapshot until accepting the upgrade. Hubuum does not automatically downgrade the schema, and releases older than the adjacent stable release are outside this certification.

The harness also creates an N-1 backup with history, an N-1 history-free backup, and an N backup. The candidate restores each into a separately created empty database, records bounded sanitized evidence, and starts its API and worker against the retained N-1 restore. Smoke checks cover authentication recovery, representative resource and history reads, excluded bearer tokens, and computed-field rebuilding. A scheduled workflow repeats the same drill so recoverability is checked between release events rather than inferred from backup creation alone.

Backup, restore, and import document formats may still change at a release boundary. Quiesce those operations while versions overlap and use the document format accepted by the application version that will process it. CI guarantees candidate restore compatibility only for the immediately adjacent stable release. A successful API rollback does not make a newer backup or import document readable by N-1; follow the existing deployment upgrade path when an older installation must cross a format boundary.

Native archives

Linux AMD64 and ARM64 archives are exported from the same Alpine builder used by the production container. All three executables are stripped, statically linked musl binaries. CI rejects the archive if any binary declares a dynamic runtime dependency, so users do not need system copies of glibc, libpq, or OpenSSL.

macOS and Windows use their native Rust targets. Their builds enable embedded migrations, bundled libpq, and vendored OpenSSL, so users do not need Homebrew, PostgreSQL client libraries, or OpenSSL packages. They retain the standard operating-system libraries expected by native executables.

Both tagged releases and main-latest use these platform contracts. Every archive includes hubuum-server, hubuum-admin, and hubuum-template-worker. Install all three together. The administrator exposes the embedded migration runner through hubuum-admin --migrate.

Container images

The CI workflow publishes one Alpine-based container image with both the rustls and OpenSSL TLS backends:

  • Versioned tags (ghcr.io/hubuum/hubuum-server:vX.Y.Z) and :main are the full image. It can also run plain HTTP when no TLS certificate and key are configured.

The full image also gets explicit aliases ending in -full.

The image never runs embedded Diesel migrations from a long-lived server entrypoint. It does not need the standalone Diesel CLI or psql; operators run migrations with hubuum-admin --migrate in a one-shot workload. The default single role mode uses HUBUUM_DATABASE_URL unless an optional migration URL override is configured; split mode requires the separate migrator credential in HUBUUM_MIGRATION_DATABASE_URL.

Publishing from main happens in the same workflow run and depends directly on the CI jobs passing. Documentation-only and repository-metadata pushes do not rebuild or replace main-latest archives or container images. The existing artifacts remain valid because their binary inputs are unchanged. Changes to Rust sources, embedded documentation, migrations, manifests, container inputs, or the publication workflow still run the complete validation and publishing path.

Operator monitoring

Use the shared operator package for Grafana dashboards, Prometheus recording and alerting rules, SLO definitions and response runbooks. The same assets work with the optional single-host stack, independently managed Prometheus/Grafana installations, and Prometheus Operator. Pin the package to your server release and scrape every process directly with deployment labels.