Development Guide¶

To edit or preview the documentation website, use the documentation workflow. The site builds from the Markdown in docs/ using the pinned Zensical container.

Python tooling¶

Repository Python scripts and tests require Python 3.11 or newer, available as python3 on PATH. They use only the standard library, including tomllib for TOML parsing. No Python packages, pip steps, or virtual environments are required. Older interpreters exit with an upgrade instruction before running the tool; the full test runner checks this before building or creating databases.

Install Python 3.11+ using your operating system's packages or your existing Python version manager, then select it in the shell used for repository commands. Changing a shell alias is insufficient for scripts that launch python3. Check the selection with:

python3 scripts/check-python-version.py
python3 -I -S tests/python/run.py unit

CI selects Python 3.11 explicitly from .python-version in jobs that execute Python tooling, including shell wrappers. A dedicated required job discovers all local Python regression tests on Python 3.11 and 3.12 without site packages. Live monitoring tests run in separate jobs. See the testing guide for the Rust/Python ownership boundary and unit and integration commands.

Git Hooks Setup¶

This project includes git hooks to maintain code quality standards. The hooks are stored in the hooks/ directory and can be shared across the team.

Setup¶

After cloning the repository, configure git to use the hooks directory:

git config core.hooksPath hooks

That's it! Git will automatically run hooks from the hooks/ directory from now on.

Pre-commit Hook¶

The hooks/pre-commit hook automatically runs cargo clippy and rebuilds docs/openapi.json before each commit. If clippy fails or the OpenAPI document cannot be regenerated, the commit is prevented.

Features¶

✅ Runs clippy with -D warnings to treat all warnings as errors
✅ Rebuilds and stages docs/openapi.json from the current code
✅ Prevents commits that fail clippy checks
✅ Prevents commits if OpenAPI generation fails
✅ Clear error messages guide developers on how to fix issues
✅ Stored in version control and shared with the team
✅ No installation script needed - git handles it automatically via core.hooksPath

Manual Checks¶

You can also manually run clippy at any time:

# Check for clippy issues
cargo clippy --all-targets

# Fix clippy issues with automatic suggestions
cargo clippy --all-targets --fix

# Rebuild the committed OpenAPI spec
cargo run --quiet --bin hubuum-openapi > docs/openapi.json

Production integration contract tests¶

Run the production LDAP, SMTP, AMQP, Valkey Streams, and webhook adapters against disposable TLS fixtures with Python 3.11+, Docker or its Podman compatibility command, and OpenSSL on PATH:

python3 tests/python/run.py integration event-transports

The runner uses digest-pinned OpenLDAP, Mailpit, RabbitMQ and Valkey images and a source-controlled HTTPS fixture. It generates a private CA, verifies certificates through the production clients, and removes its containers and temporary credentials after the run. Ports bind only to loopback. No Python packages are required.

The suite checks AMQP publisher acknowledgements, mandatory unroutable delivery, event identity and payload, Valkey exact trimming, and recovery of reused sink instances after service restart. HTTPS cases cover trusted and untrusted certificates, redirects, server errors, response limits, and timeouts.

Ordinary local cargo test leaves these fixture-dependent tests ignored. The runner explicitly enables them; missing fixture settings fail the tests. The Production integration contracts CI job runs for full validation and release tags, and contributes to the required CI gate. The shared Valkey login limiter also runs on release tags and is required before release publication.

LDAP cases exercise LDAPS and mandatory STARTTLS, invalid credentials and filter escaping, stable-subject refresh, untrusted CAs, and restart recovery. SMTP cases exercise authenticated implicit TLS, message delivery, temporary and permanent recipient rejection, invalid credentials, untrusted CAs, and cached-transport recovery. Mailpit rejects recipients deterministically at 100% for failure cases.

See integration coverage for the evidence boundary and remaining work in #248.

Architecture Overview¶

The codebase is incrementally split into application services, backend-neutral storage capabilities, model-facing APIs, and database-facing implementations. The root Rust library is an internal application composition crate rather than a supported embedding interface. See Rust API Boundary for package classifications, publishing policy, and promotion requirements.

src/services/*: Application-facing use cases. Migrated handlers call services instead of choosing persistence helpers directly.
crates/hubuum-domain and crates/hubuum-storage-core: Backend-neutral values, storage DTOs, errors, and extracted capability traits. These crates cannot depend on Actix, Diesel, global application configuration, or ApiError.
src/storage/*: Complete backend composition, exhaustive dispatch, common observation, and the opaque application handle. src/storage/factory.rs is the only process-composition path allowed to select a backend or inspect its connection settings. The test-only memory resource model exercises focused logical behavior. It is not a selectable backend and does not represent partial application support.
crates/hubuum-storage-postgres: The complete production PostgreSQL adapter: pool construction, TLS setup, endpoint parsing, private rows and queries, native transactions, schema, migrations, and operation implementations. It exposes private-field settings and backend-specific initialization errors only at the application composition edge.
crates/hubuum-storage-conformance: The experimental public six-part behavioral verifier for audited mutation receipts, no-ops, rollback, event delivery, telemetry, and exact revision conflicts. Retention retry safety has its own protocol verifier. It is a development dependency and is not linked into production binaries.
src/models/*: Application domain models and high-level operations. These should not contain Diesel query construction for non-trivial backend logic.
src/traits/*: Behavioral interfaces used by handlers and models inside the application. StorageContext is a sealed, opaque persistence capability. Consumers pass it to operations but cannot extract or select the database implementation. Normalization returns the context's existing StorageHandle; it never rebuilds a backend from a database pool. It deliberately has no authorization methods; policy-aware workflows accept the stronger AuthorizationContext.
crates/hubuum-storage-postgres/src/test_support.rs: Feature-gated, typed PostgreSQL fixtures for application integration tests. Root tests must not recreate adapter rows or SQL helpers.

Server, administration, and bootstrap entry points build validated StorageSettings and receive an opaque StorageHandle. They must not import PostgreSQL pools, Diesel, generated schemas, or adapter operations. Initial administrator creation, human-user lifecycle, local-password reset with token revocation, bearer-token lifecycle, complete template-audit reads, and export-template health aggregation are mandatory identity and operational contract methods, so a selectable backend cannot omit any process-lifecycle behavior.

Collection, class, object, class-relation, and object-relation services each depend on their exact backend-neutral storage trait. Metrics, readiness, maintenance state, event persistence health, atomic event fan-out, delivery, event retention, and token retention are also expressed as required storage traits with backend-neutral inputs and results. Delivery workers receive enriched work-item DTOs and acknowledge opaque claims through the contract. Audit reads, sink and subscription lifecycle, enabled-sink discovery, and claim-free delivery administration form a separate mandatory event-administration family. Retention archives receive storage-owned event DTOs rather than PostgreSQL row models. Runtime worker settings and counters are enriched above the event-health storage boundary. Validated event-worker and retention policies live in hubuum-domain; adapter-only query limits are derived through neutral accessors rather than leaking database terminology to consumers. A selectable backend must nevertheless satisfy the complete application storage contract. See Application and Storage Boundary for the required families, compatibility tests, performance gates, and opaque backend boundary.

Practical layering rule¶

When adding a feature:

Put new use-case orchestration in src/services when the surrounding slice has migrated.
Express persistence as an aggregate- or query-shaped capability in hubuum-storage-core; avoid generic table repositories.
Implement database details in crates/hubuum-storage-postgres/src/operations and delegate through its complete PostgreSQL backend implementation.
Keep model methods thin while they remain in unmigrated paths.
Add shared logical contract tests and retain PostgreSQL-specific query, transaction, migration, recovery, and concurrency tests.
Do not register the implementation as selectable until it satisfies every storage operation trait, all six family bounds, and the available-backend compatibility suite.

Module layout notes¶

To keep PostgreSQL adapter code navigable, production operations are split into focused modules under crates/hubuum-storage-postgres/src/operations, including:

lifecycle modules such as collection.rs, class.rs, object.rs, and relation.rs;
identity modules such as user.rs, service_account.rs, token.rs, group.rs, and authorization/*; and
workflow and operational modules for tasks, imports, restores, events, retention, metrics, and computed data.

Application code must use application services and mandatory storage contracts instead of importing these adapter-private modules directly.

Collection hierarchy implementation¶

Recursive collections are implemented in the PostgreSQL workspace adapter. The implementation is coupled to Diesel schema modules, PostgreSQL closure-table SQL, temporal history, and Hubuum's permission facts, while its public result remains StorageError and backend-neutral DTOs. Keep production hierarchy writes in crates/hubuum-storage-postgres/src/operations/collection.rs and permission queries under that crate's authorization operations.

Normal collection and class lifecycle handlers enter this implementation through their service and storage capabilities. Do not bypass those services from a handler. Application permission and metadata paths likewise use the mandatory authorization and operational contracts; only backend code and test-only fixtures may select PostgreSQL operations. Internal adapter location does not make behavior optional: a selectable backend must supply every capability before it can be registered.

When adding a collection creation path, use the shared collection insert helper from the collection backend so collections and collection_closure stay in sync. Do not insert directly into collections unless the closure rows are created in the same transaction. When changing permission checks, preserve the combined-permission rule: a single permission row on the target collection or an ancestor must satisfy all requested flags.

See Collection Hierarchy for user-facing behavior, move constraints, indexes, and the rationale for keeping its PostgreSQL mechanics adapter-local.

Pull request CI tiers¶

Ordinary prose and documentation-site changes run Markdown and documentation checks without the application matrix or benchmark cache builds. CHANGELOG.md retains the OpenAPI and operational-contract checks because release notes are inputs to compatibility exceptions. Embedded documentation, generated contracts, and architecture-test inputs retain their existing code checks; unknown files under docs/ also receive conservative validation.

Change selection compares pull requests against their merge base, includes both sides of renames, and fails if Git cannot produce a reliable diff. The existing CI gate remains required. Benchmark cache warming on main runs only when benchmark inputs change; manual benchmark dispatch still forces cache warming.

Pull request validation is selected from the complete base-to-head diff:

Draft documentation-only pull requests run Markdown lint when Markdown files changed.
Draft code pull requests run Rust formatting and the complete default-feature test suite on Linux with PostgreSQL. They do not run the other feature combinations, cross-platform tests, production container build, release build, or benchmarks.
Ready-for-review code pull requests run the complete CI suite. The ready_for_review event starts that suite immediately, and later pushes keep running it.
Ready-for-review documentation-only pull requests keep the smaller relevant checks. docs/openapi.json receives the OpenAPI contract check, while docs/export_template_guide.md is treated as code because it is embedded in a binary. docs/querying.md is treated as a test input because the API test suite compiles and validates its documented operator lists.

The OpenAPI contract check is blocking at every CI tier where API inputs change. It separately verifies exact generated-document drift and semantic compatibility with the latest stable release, then uploads generated, baseline, diff, JSON, and Markdown evidence. The policy scripts, severity configuration, and breaking-change exception file are themselves classified as OpenAPI inputs. See the release guide for the baseline and intentional-break rules.

The Rust API policy check classifies every Cargo package and prevents internal packages from becoming publishable accidentally. If a package is deliberately promoted to experimental or stable public status, the same job adds rustdoc, clean package, and semantic compatibility checks automatically. The change classifier discovers each declared policy document so deleting or moving one still selects this job in an otherwise documentation-only change. Run the local fixtures with python3 tests/python/run.py unit policies.test_rust_api and python3 tests/python/run.py unit policies.test_crates_io_baseline.

The ci:full pull request label forces the complete CI and benchmark suites, including on a draft or documentation-only pull request. The ci:benchmarks label forces all benchmark jobs without expanding the main CI tier. Adding or removing either label takes effect immediately; converting a pull request back to draft cancels superseded work unless a forcing label remains.

The lightweight change classifier is implemented by scripts/classify-ci-changes.sh. Unknown file types are classified conservatively as code, container, artifact, and benchmark inputs. Keep its tests in scripts/test-classify-ci-changes.sh synchronized with any new build inputs. Direct literal include_str! and include_bytes! inputs are discovered by that test and must be classified as code automatically. Renames are evaluated as a deletion plus an addition so both the old and new paths affect the selected tier. The stable CI gate job reports the combined result of every applicable PR or main validation job, is the check intended for branch protection, and must pass before main-latest artifacts or container images are published.

Benchmarks¶

Benchmarking runs in a separate GitHub workflow, .github/workflows/benchmarks.yml, via terjekv/rust-pr-bench v1.3.0, pinned to its release commit.

PR comparisons cache Cargo downloads, compiled outputs, exact-version benchmark runners, and verified benchmark executables under the hubuum-benchmarks namespace. Every push to main also runs a compile_only build to warm caches accessible to later PRs. To warm them manually, dispatch the Benchmarks workflow on main. Warming skips measurements, service lifecycle hooks, and PR comments; ordinary PR runs still measure both revisions and apply the configured regression limits.

Keep the comparison and warming jobs' action revision, namespace, benchmark selection, toolchain, features, and Cargo arguments aligned. Both use --locked and enable cache_binaries so exact source/build matches can skip compilation and linking. The action elects cache writers within each run; PRs can save their own entries and runner installations, while the main build supplies shared baseline entries. The existing Swatinem/rust-cache entries used by other CI jobs are separate and do not populate these benchmark caches. Platform and feature tests use 16 codegen units with LTO disabled, while benchmarks retain the production release profile; their compiled outputs are not interchangeable.

Executable reuse requires reproducible builds. If benchmarks gain external build inputs or custom compile-time environment variables, version them with matching binary_cache_key values in both jobs. Declare extra runtime assets through binary_cache_paths; Cargo build-script outputs are bundled automatically. Review the build/cache tables and performance.jsonl artifacts for hit/miss, transfer, compilation, and executable-reuse evidence. See the upstream cache setup and diagnostics.

Local execution¶

The benchmark targets are split one benchmark binary per file so CI can fan them out independently:

cargo bench -p hubuum-query --bench parse_query_parameter_callgrind
cargo bench -p hubuum-query --bench parse_integer_list_callgrind
cargo bench --bench json_sql_filters_callgrind
cargo bench -p hubuum-query --bench search_operator_parsing_callgrind
cargo bench --bench permissions_parsing_callgrind
cargo bench -p hubuum-query --bench jsonb_type_inference_callgrind
cargo bench -p hubuum-templates --bench size_limited_writer_callgrind
cargo bench --bench token_storage_hash_callgrind
cargo bench --bench request_hash_callgrind
cargo bench --bench unified_search_query_parsing_callgrind
cargo bench --bench unified_search_cursor_callgrind
cargo bench --bench object_validation_geo_callgrind
cargo bench --bench object_validation_nested_callgrind
cargo bench --bench database_url_parsing_criterion -- --noplot
cargo bench --bench password_hashing_criterion -- --noplot

The CI benchmark action auto-discovers every direct benches/*.rs target and reports both Criterion and Gungraun results. The container-build tests enforce that every Cargo benchmark remains directly discoverable.

Gungraun requires valgrind and the matching benchmark runner to be installed locally:

cargo install --locked --version 0.20.0 gungraun-runner

The PostgreSQL storage benchmark is opt-in and requires Docker. It provisions, migrates, and removes a disposable PostgreSQL container itself. Container startup, fixture creation, cleanup, and warmup happen outside the timed regions. It includes selective and non-selective structured related-object searches over 128 independent chains at the maximum supported depth of 10. The create scenario intentionally leaves its append-only audit events behind:

cargo bench --features postgres-bench \
  --bench storage_postgres_criterion -- --noplot

To run only the structured-search cases:

cargo bench --features postgres-bench \
  --bench storage_postgres_criterion -- structured_related_depth_10 --noplot

The runtime behavior validator is opt-in and requires a migrated, disposable database. It starts an all-role primary and an API-only standby, measures idle Prometheus counter deltas, sends fixed readiness traffic, and inserts one intentionally invalid export task to measure PostgreSQL notification-to-claim latency. This is an operational budget check rather than a Cargo benchmark. Build the server before running it; the development profile avoids irrelevant release-link overhead:

export HUBUUM_BENCH_DATABASE_URL=postgres://postgres:postgres@localhost/hubuum_runtime
cargo run --features embedded-migrations --bin hubuum-admin -- \
  --migrate --legacy-single-role-migration \
  --database-url "$HUBUUM_BENCH_DATABASE_URL"
cargo build --features runtime-behavior-check --bin hubuum-server
cargo run --profile dev --features runtime-behavior-check \
  --bin hubuum-runtime-behavior-check -- measure \
  --server-binary target/debug/hubuum-server \
  --database-url "$HUBUUM_BENCH_DATABASE_URL" \
  --sample-seconds 60 \
  --label local \
  --output target/runtime-behavior/local.json
cargo run --profile dev --features runtime-behavior-check \
  --bin hubuum-runtime-behavior-check -- assess \
  --head target/runtime-behavior/local.json

The JSON report separates connection-pool acquisitions by caller, records task/fan-out timer rates and per-iteration checkout ratios, verifies one readiness checkout per request, and records notification wake-up and task-claim latency. Connection acquisitions are pooled checkouts, not new PostgreSQL network connections.

The deterministic PostgreSQL query budgets use the normal isolated test database runner. The central storage suite covers point reads, hierarchy and permission traversal, paginated object and history reads, and event-producing writes:

source .env && ./run_tests.sh storage_performance

Import planning, export hydration, and event fan-out budgets live beside those private execution paths and run as part of the full test suite. Fixed-size operations pin exact domain, transaction-control, query-fingerprint, and connection-checkout counts. Cardinality tests compare small and large inputs to pin either constant query shapes or an explicit bounded-linear slope.

The capture excludes the pool's internal SELECT $1 checkout-validation probe from application query totals. Each checkout is counted separately, which keeps pool-use regressions visible without attributing a connection-health query nondeterministically to the next operation.

CI behavior¶

Draft pull requests skip benchmarks unless ci:full or ci:benchmarks is present. Ready pull requests run only the benchmark jobs affected by their base-to-head diff.
Change classification happens before PostgreSQL services or benchmark runners are allocated. Superseded benchmark workflow runs are cancelled.
The self-contained benchmark job runs both backends in one combined backend: all job, so PRs get a single consolidated benchmark export.
Gungraun's Callgrind measurements remain the practical gating signal with a low regression threshold.
Criterion still runs in the same combined job, but uses a very high regression threshold so it exports timing changes without acting as a meaningful gate.
The shared benchmark action runs and reports all direct Criterion and Gungraun targets, including the self-provisioning PostgreSQL storage target.
A two-process runtime behavior validation job records base/head Prometheus counter deltas and publishes both JSON reports plus a Markdown comparison. Absolute budgets guard idle polling, database checkout ratios, readiness behavior, and notification-driven task claims; a 25% base/head threshold catches larger behavioral regressions while tolerating timer-boundary jitter.
The PostgreSQL query-budget tests are the stricter gate: fixed operation totals, control/domain splits, query fingerprints, connection checkouts, and declared scaling slopes must remain stable.
On a target's first pull request there is no base result to compare, so CI records the initial baseline. Later pull requests compare base and head.

Adding or modifying benchmarks¶

Put new benchmark entrypoints in benches/.
Keep each benchmark target in its own file so the benchmark workflow can fan out per bench binary.
Add a matching [[bench]] stanza in Cargo.toml with harness = false.
Include callgrind in the benchmark filename when it should be auto-discovered by the CI workflow.
Include criterion in the benchmark filename when it should be Criterion-only in CI autodiscovery.
Prefer deterministic library-level code paths such as parsers, query builders, and serialization helpers over handlers that require network or database setup.
Put database-backed targets behind the postgres-bench feature and make their external fixture setup self-contained so the shared action can run them.
Seed, migrate, warm, and clean PostgreSQL fixtures outside measured regions. Mutation benchmarks run last against fresh isolated base/head databases; emitted audit events remain append-only, as they do in production.
Avoid code paths that read the global CONFIG (the clap-backed application configuration). Initialising it inside a benchmark binary panics on the harness's own CLI arguments (for example --iai-run). Where a function needs configuration values such as page limits, prefer a config-free entry point that takes them as parameters (see parse_unified_search_query_with_limits and validate_page_limit_with_max).

Schema evolution changes¶

Schema use cases run through required SchemaEvolutionStorage operations and the observed opaque storage handle. Keep document compilation proof, exact schema references, object resource revision fences, and authorized collection coordinates across boundaries. PostgreSQL locks and triggers live in the adapter crate; the memory adapter must preserve the same logical semantics.

Changes must retain atomic events/audit, bounded batches, lease recovery, impact epoch checks, import rollback, and logical backup integrity. Shared and native regressions live in src/tests/storage_contract/schema_evolution/. Run the config-free schema_validation_criterion benchmark for validation throughput. See class schema evolution for routes and invariants.