@veritio/storage

Host-injected storage helpers for Veritio audit trail evidence.

The package provides durable AuditStore factories for transaction-capable SQL and MongoDB boundaries, transactional evidence outbox helpers, plus a Redis tenant-tip cache helper. It does not read environment variables, open database connections, or bundle vendor clients.

Durable Stores

• createPostgresAuditStore: PostgreSQL-compatible stores.
• createNeonAuditStore: Neon-backed PostgreSQL-compatible stores.
• createMysqlAuditStore: MySQL-compatible stores.
• createMariaDbAuditStore: MariaDB-compatible stores.
• createMongoAuditStore: MongoDB stores with host-provided transaction boundaries.

Host applications must provide a transaction-capable client wrapper. The store uses tenant-scoped append ordering, idempotency-key hashes, expected previous hash checks, and persisted record integrity validation.

Transactional Outbox

The storage package exports the public outbox contract used by governed-change drafts:

• OutboxAdapter
• createOutboxDispatcher
• dispatchOutboxEntry
• createPostgresOutboxAdapter
• createNeonOutboxAdapter
• createMysqlOutboxAdapter
• createMariaDbOutboxAdapter
• createFileOutboxAdapter

SQL outbox adapters use the same host-injected transaction pattern as the SQL AuditStore factories. A host should enqueue the minimized governed-change payload inside the same database transaction as the application mutation when it wants to claim mutationBinding: "same_transaction".

await database.transaction(async (tx) => {
  const veritioOutbox = createPostgresOutboxAdapter({
    client: tx.veritioOutboxExecutor,
  });
  const before = await entries.get(tx, id);
  const after = await entries.update(tx, id, patch);
  const draft = createGovernedChangeDraft({
    scope,
    entity: projectEntryEntity,
    before,
    after,
    changedPaths: ["/amount"],
    change,
    activity,
    producer,
    occurredAt: new Date(),
    idempotencyKeyHash,
    mutationBinding: "same_transaction",
  });

  await veritioOutbox.transaction((outboxTx) => outboxTx.enqueue({
    id: idempotencyKeyHash,
    tenantId: scope.tenantId,
    payload: draft.outboxEntry,
  }));
});

The exact adapter wiring depends on the host database wrapper. The important invariant is that tx.veritioOutboxExecutor.transaction(...) participates in the already-active host transaction instead of opening an unrelated commit.

Dispatch is retry-safe. If a worker crashes after appending some events, the next run replays the same event and edge IDs and the evidence sink rejects duplicates or conflicts deterministically.

await createOutboxDispatcher({
  adapter: veritioOutbox,
  target: createFileEvidenceStore("./.veritio/evidence"),
}).dispatchBatch({ tenantId: scope.tenantId });

createFileOutboxAdapter is useful for local examples and self-hosted file workflows. It persists outbox rows atomically within its own file lock, but it does not prove a separate application database mutation committed in the same transaction. Use mutationBinding: "not_transaction_bound" or "best_effort" unless the host can prove the business mutation shares the same transaction boundary.

Redis

createRedisAuditTipCache stores and reads validated tenant chain tips. It is not an AuditStore and does not claim durable audit evidence on its own. Use it alongside a durable store when a cache is useful.

Object-Storage Archive (Cloudflare R2 / AWS S3)

createObjectAuditArchive is a derived cold tier for a tenant's evidence chains on any S3-compatible object store (Cloudflare R2, AWS S3, MinIO). It is NOT an AuditStore and must never own appends: object storage cannot couple gapless per-tenant sequencing to idempotency-conflict checks atomically, so sequencing stays in a conforming authoritative store and the archive seals already-sequenced records into immutable segments. Events and edges are the protocol's two independently sequenced chains, so they archive under separate key namespaces (sealSegment/sealEdgeSegment) and verifyTenant replays both, mirroring FileEvidenceStore.verify.

Segments store the exact canonicalJson bytes of each record as NDJSON lines, so hashes recompute byte-for-byte and a full tenant chain verifies from the archive alone. Sealing fails closed on gaps, overlaps, tampered records, and mixed tenants; an identical replay of an already-sealed range is idempotent. archiveAuditStoreTenant drains a tenant's un-archived event tail from any AuditStore incrementally, resuming from the archive's own tip (edge batches are sealed directly — AuditStore does not expose edge records).

The host injects an ObjectArchiveClient (put/get/list over bytes) built on its own S3 client — an R2 bucket binding, the AWS SDK, or Bun's built-in S3Client — and owns credentials, bucket provisioning, and retention/WORM policy. Serialize archiving per tenant; the archive assumes one sealer per tenant chain.

ClickHouse Read Model

createClickHouseAuditReadModel is a derived, eventually consistent analytical read model for scan-heavy groupings: activity-episode rollups, session reconstruction, and subject scans. It is NOT an AuditStore and must never be authoritative — ClickHouse has no synchronous unique constraints or transactional tip checks, so verify/export and idempotency stay on the authoritative store.

Projection is at-least-once: duplicates collapse via ReplacingMergeTree, and the read helpers query with FINAL so replays never double-count. The full canonical record travels as a raw String column (never ClickHouse's native JSON type), every projected and returned record is hash-revalidated, and all value filters bind through ClickHouse query parameters. The host injects a ClickHouseExecutor over the HTTP interface (see integration/clickhouse-read-model.test.ts for a fetch-based reference) and owns endpoint, credentials, and database selection.

AuditStore Conformance Suite

@veritio/storage/conformance exports createAuditStoreConformanceTests for adapter tests. Use it for in-memory fakes and for live database integration checks so all durable stores prove the same behavior: tenant-scoped ordering, idempotency conflicts, expected previous-hash checks, cloned returned records, and fail-closed integrity validation.

import { describe, test } from "bun:test";
import { createPostgresAuditStore } from "@veritio/storage";
import { createAuditStoreConformanceTests } from "@veritio/storage/conformance";

describe("postgres live AuditStore conformance", () => {
  for (const conformanceTest of createAuditStoreConformanceTests({
    name: "postgres live",
    async createTarget() {
      const host = await createHostInjectedPostgresHarness();
      await host.resetAuditTable();

      return {
        store: createPostgresAuditStore({ client: host.executor }),
        async mutateStoredRecord({ tenantId, sequence, mutate }) {
          const record = await host.readRecordJson(tenantId, sequence);
          await host.writeRecordJson(tenantId, sequence, mutate(record) ?? record);
        },
        close: () => host.close(),
      };
    },
  })) {
    test(conformanceTest.name, conformanceTest.run);
  }
});

The host harness owns database clients, credentials, connection strings, test containers, and cleanup. Keep environment-variable reads in the test bootstrap or CI setup, not in storage/src.

External DB Checks

External database checks run through the same package test command when matching connection strings are present. Without these environment variables, the live database suites are skipped and the in-memory conformance tests still run.

VERITIO_POSTGRES_TEST_URL=postgresql://... \
VERITIO_MYSQL_TEST_URL=mysql://... \
VERITIO_MONGODB_TEST_URL=mongodb://... \
bun run --cwd storage test

Supported live-test variables:

• VERITIO_POSTGRES_TEST_URL
• VERITIO_NEON_TEST_URL
• VERITIO_MYSQL_TEST_URL
• VERITIO_MARIADB_TEST_URL
• VERITIO_MONGODB_TEST_URL

The GitHub Actions verification job runs Postgres, MySQL, MariaDB, and MongoDB service containers and sets these variables for bun run verify. The VERITIO_NEON_TEST_URL job value points at the same Postgres-compatible service so the Neon factory stays covered without requiring a hosted Neon account. Projects that need hosted Neon proof can set VERITIO_NEON_TEST_URL to a disposable branch connection string in their own CI.

Derived-tier integration suites gate on their own variables the same way:

• VERITIO_S3_TEST_ENDPOINT (+ VERITIO_S3_TEST_ACCESS_KEY_ID, VERITIO_S3_TEST_SECRET_ACCESS_KEY, VERITIO_S3_TEST_BUCKET) for the object archive against any S3-compatible endpoint.
• VERITIO_CLICKHOUSE_TEST_ENDPOINT (+ VERITIO_CLICKHOUSE_TEST_USER, VERITIO_CLICKHOUSE_TEST_PASSWORD, VERITIO_CLICKHOUSE_TEST_DATABASE) for the ClickHouse read model.

For local runs, storage/docker-compose.yml provides disposable targets for every suite (Postgres, MySQL, MariaDB, MongoDB as a single-node replica set, MinIO with a pre-provisioned bucket, and ClickHouse) on ports that avoid common local services:

bun run --cwd storage db:up     # start containers and wait for health
bun run --cwd storage test:live # run every suite against them
bun run --cwd storage db:down   # discard containers and data

To point at databases you manage yourself instead:

1. Start a disposable Postgres, Neon branch, MySQL, MariaDB, or MongoDB test database outside this package.
2. For SQL stores, use the schema helpers or example schemas. For MongoDB, create indexes from MONGO_AUDIT_RECORD_INDEXES.
3. Run bun run --cwd storage test with the matching environment variables.

Redis is not a durable AuditStore target and should not run this conformance suite. Test Redis only as a validated tenant-tip cache beside a durable store.

SQL Schema Helpers

POSTGRES_AUDIT_RECORDS_SCHEMA_SQL and MYSQL_AUDIT_RECORDS_SCHEMA_SQL provide starting table definitions. Review them for your database policy, backup, retention, and migration requirements before applying them.

POSTGRES_OUTBOX_SCHEMA_SQL and MYSQL_OUTBOX_SCHEMA_SQL provide starting table definitions for transactional outbox rows. Apply them in the same database where the host mutation transaction runs when using the SQL outbox adapters.

Veritio supports evidence collection and verification workflows. It is not legal advice and does not make an application automatically compliant with any regulation or framework.