verify

Portfolio-wide post-deployment web QA/QC. One manifest. One report. Zero manual checks.

verify is the hard gate between deploy and done. Drop a .verify.json manifest into any project, run verify run, and get back a machine-readable report covering console errors, GA4/Ads firing, broken links, unrendered template tokens, SEO metadata, accessibility, Lighthouse budgets, visual regressions, form submissions, authenticated flows, and more. On failure, verify auto-raises a Vantage incident in the calling project with fingerprint-based deduplication. On the next PASS, it resolves itself.

5-Minute Onboarding

# In any project you want to QA:
cd ~/projects/your-project
npx @eleetai/verify init           # scaffolds .verify.json with sensible defaults
verify run --env prod              # run against production
# → writes runs/<project>/<timestamp>/report.json, exits 0 on PASS

That's it. If anything failed, the report tells you what, where, and how to fix it. A Vantage incident is opened in your project until you redeploy and rerun.

The coherence contract (exit code ↔ status)

Exit code and report.status are the same decision, reported two ways. They cannot disagree.

Consumers check exit code OR report.status, not both. They encode the same gate decision. The runner guarantees coherence: exitCode === f(status) always.

The contract is pinned at two test boundaries: (1) unit tests of deriveStatus / deriveExitCode exhaustively cover the 30-cell (tier × gate) matrix (tests/unit/exit-code-coherence.test.ts); (2) integration tests spawn the compiled CLI and assert the process exit code matches report.status for every contract row (tests/integration/cli-exit-code-coherence.test.ts). The process-boundary pin exists because consumers observe $? after the process exits, not function return values — those are different boundaries and can drift independently.

--fail-on-tier lets you tolerate noise without hiding it. Example: verify run --fail-on-tier P1 means P2/P3/P4 failures log as tolerated warnings but don't trip the gate. Surfaced in the report's summary.tolerated_warnings + summary.gate_tripped. A PASS_WITH_WARNINGS run does NOT raise a Vantage incident — the consumer explicitly opted in to tolerating those failures.

What it checks

Universal (every consumer gets these for free)

• Console errors, pageerror exceptions, and renderer crashes (v0.14.0) — captured via a unified structured event stream and classified by stage: mount (pre-first-interaction; default tier P0 — the white-screen class) → interaction (during scripted action; default P1) → warning (recoverable; default P2). Stack frames are parsed (V8 + SpiderMonkey + WebKit) and resolved against served .map files (best-effort; opt-out with --no-sourcemap-resolve). Allowlist + suppressions both supported with audit-grade reasons. The check that catches the React mount-time TypeError: Cannot read properties of undefined (reading 'slice') class — see "Authenticated routes" below for the architectural fix that lets this work on protected pages
• Network 4xx/5xx on main document and same-origin
• Resource 404s — images, CSS, JS, favicons
• Unrendered template tokens — {{var}}, ${...}, literal undefined leaking into visible text
• Mixed content — http:// on https://
• SEO metadata — title, description, canonical, OG tags, JSON-LD schema
• Broken links — same-origin crawl at declared rate limit
• Canonical host — canonical URL host matches manifest allowlist
• CSP violations — surfaced from console
• robots.txt + sitemap.xml — present and well-formed
• Security headers (HSTS, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, CSP) — absence FAILs at P1; version-bearing values in Server / X-Powered-By (e.g., nginx/1.18.0, PHP/8.1.2) FAIL as CVE-mappable fingerprint leaks. Bare vendor identifiers (CloudFront, cloudflare, Kestrel, etc.) surface as warnings — aligned with Mozilla Observatory / OWASP threat models
• CORS config (P1, auto-enabled when api.contracts has an origin different from envs.*.base_url) — issues two preflights (legitimate web origin + https://malicious.example) and an in-page fetch() to catch the class of bug where the API is reachable directly but cross-origin requests from the browser are blocked. Catches Origin reflection without allowlist (the bypass that passes a single legitimate-origin probe)
• Cache-Control on static assets (P2) — distinguishes hashed (content-addressed) assets from unhashed. Hashed assets without public, max-age=31536000, immutable FAIL or WARN; no-store on hashed assets defeats content hashing. Aligned with Lighthouse uses-long-cache-ttl and RFC 9111
• Supply-chain CVE (P0, auto-enabled with package-lock.json) — batches packages to OSV.dev, filters by max_severity (default "high" — critical+high FAIL, medium WARN, low ignored), 24h cache, network-unreachable → SKIP not FAIL. Used by GitHub, npm, Google's OSS-Fuzz
• Frontend JS library CVE (P0, v0.17.0) — scans the JavaScript libraries actually loaded in the browser via retire.js. Closes the gap that dependency-cve can't see — Stripe.js, GA, Cognito Hosted UI, GTM, chat widgets, CDN-pinned legacy jQuery — anything that bypasses package-lock.json. 24h cache for the signature DB; network-unreachable → SKIP not FAIL. ignore_libs[] declarative scope-narrowing for legacy embeds you can't upgrade
• Third-party-CDN resilience (P2, OPT-IN) — for each cross-origin script the site loads, opens a fresh BrowserContext, blocks just that URL via page.route(), re-navigates, asserts critical_selector still renders. Catches "site white-screens when X CDN goes down" before the user does
• Stripe checkout E2E (P0) — stripe-checkout-test auto-engages when js.stripe.com is detected; drives a real test-mode payment with documented test cards (incl. always-3DS) and asserts a manifest-declared success criterion. stripe-checkout-live (OPT-IN, requires --allow-live-payments) makes a real charge on a dedicated automation card + immediate refund + audit log of every transaction. Live 3DS challenges fail gracefully with Stripe Radar whitelist remediation

Expectation-driven (declared per-project in .verify.json)

• GA4 firing — intercepts /collect?v=2, compares tid to manifest
• Google Ads firing — AW- tag verification
• Forbidden text — regex scan for strings that must NOT appear (old brand names, unset template tokens)
• Required text — strings that MUST appear (USPs, compliance disclaimers)
• Form submissions — fill + submit + assert post-submit state
• Authenticated flows — Cognito or form login via Vantage-vault secrets
• Lighthouse budgets — LCP, CLS, TBT, a11y, SEO, performance scores

Orchestrator-level (automatic)

• Multi-viewport — desktop (1280×720) + mobile (390×844) for every route
• Visual regression — .verify/baselines/ comparison per route × viewport, diff PNG on failure
• Regression detection — tags the run regression: true when previous PASS becomes current FAIL
• JUnit XML output — --junit report.xml for CI consumption

Authenticated routes (v0.14.0+)

verify can drive authenticated portals (Cognito hosted UI, custom React/Vue login forms, anything that accepts email + password and sets a session cookie). The architectural pattern that makes this work — and the bug it closes — comes from the 2026-04-24 eleetai incident: a customer-facing /agreements page returned HTTP 200, passed Lighthouse + axe-core, but threw a TypeError during React mount. The page rendered as a white screen for the customer. Verify reported PASS because the check ran on the redirected /login page, not the post-auth /agreements.

The fix has three pieces. Set them in your env-level (or top-level) auth block:

{
  "envs": {
    "prod_portal": {
      "base_url": "https://app.eleet.ai",
      "routes": ["/login", "/guest-checkout"],
      "auth": {
        "type": "cognito",
        "login_url": "/login",
        "success_selector": "[data-testid=dashboard-heading]",
        "secret_labels": {
          "pools": {
            "customer": { "email": "eleetai-customer-portal-email", "password": "eleetai-customer-portal-password" },
            "internal": { "email": "eleetai-internal-admin-email",  "password": "eleetai-internal-admin-password" }
          }
        },
        "pool": "customer",                                    // [1] which pool's creds to use
        "persist_session": true,                               // [2] reuse the post-login session for every route
        "protected_routes": ["/dashboard", "/agreements", "/account", "/billing"]   // [3] auto-promoted into the route set
      }
    }
  }
}

What each piece does:

1. auth.secret_labels.pools + auth.pool — the new pooled shape lets you declare multiple credential pairs (e.g., a CUSTOMERS Cognito pool and an USERS pool) and select one per env. The flat {email, password} shape from v0.13.x and earlier still works unchanged.
2. auth.persist_session: true — when set, the orchestrator runs the auth flow ONCE before the per-route loop and seeds the captured storageState (cookies, localStorage, IndexedDB) into every subsequent route capture. This is the architectural fix: authenticated routes finally reach their post-login state, where console-errors can observe mount-time exceptions. Default false — current consumers (nc, geauxfile, marketing surfaces) see byte-identical behavior.
3. auth.protected_routes — already supported pre-v0.14, but now any path here that's NOT in routes[] is auto-promoted into the effective route set. The eleetai miss happened in part because /agreements wasn't in routes[]; the new behavior catches that class of forgotten coverage.

When persist_session: true:

• The auth-flow check SKIPs (with note session bootstrapped pre-route) — its job is already done by the orchestrator.
• If the bootstrap itself fails (login form unreachable, credentials wrong), auth-flow reports as ERROR at tier P0 and ALL protected_routes are skipped — no misleading PASSes against /login redirects.
• Credentials are fetched per-run via the Vantage vault using the existing X-Vantage-Project ACL — never cached to disk.

Related tuning: expectations.console_errors.tier_by_stage lets you customize the per-stage tier mapping (defaults: mount: "P0", interaction: "P1", warning: "P2"). See docs/PLAYBOOKS.md for the full Cognito-portal cookbook.

Invocation modes

• CLI — verify run [--env prod|staging] [--junit PATH] [--dry-run]
• MCP — drop templates/mcp/verify.mcp.json into any project's .mcp.json; Claude Code agents invoke verify_run, verify_get_latest_report, verify_list_runs, verify_update_baselines
• Scheduled — verify schedule --project NAME --cron "0 2 * * *" registers a Vantage cron-script agent for nightly runs

The contract

Both schemas are frozen at v1. Breaking changes are major-version events.

• Manifest: schemas/manifest.schema.json — the public API
• Report: schemas/report.schema.json — the wire protocol
• Examples: examples/nc.verify.json, examples/eleetai.verify.json

See docs/PLAYBOOKS.md for per-site-type cookbooks (static Next.js, Astro marketing, SPA with Cognito auth, landing pages optimized for ads).

Diagnosis cost attribution (v0.17.1+)

When diagnosis.enabled: true and a check FAILs, every fresh Claude call records its USD spend on the report:

• report.checks[i].diagnosis.cost_usd — number, USD. Sourced from the Claude Code CLI envelope (total_cost_usd). Absent on cache hits — the cache hit didn't make a new API call, so this deploy didn't spend on it.
• report.summary.diagnosis_total_cost_usd — number, USD. Sum of per-check cost_usd across the run. Absent (or 0) when no diagnosis fired or all hit cache.

Both fields are optional and additive (v1 wire preserved). Consumers attribute spend per deploy with a one-liner:

jq '.summary.diagnosis_total_cost_usd // 0' runs/<project>/latest/report.json

Run verify schema report to print the live shape.

Vantage integration

When .verify.json has vantage.enabled: true and project_id set:

No Vantage code changes required — uses shipped endpoints. See docs/SYSTEM_ARCHITECTURE.md for wire details.

Philosophy

Read ~/projects/zeos-apps/verify/VERIFY_SOUL.md for the full doctrine. In brief:

1. Schema is scripture. The manifest and report are the product.
2. Hard gate, not suggestion. No bypass flags. Suppressions are explicit, dated, auditable.
3. Checks are pure. One file, one function, one result.
4. Evidence or it didn't happen. Every check ships artifacts.
5. AI-native first. The report is designed for agents to parse and act on.
6. Platform, not product. Adding a check benefits every consumer immediately.

Status

Phase 0 — zeos scaffold complete. Schemas locking. Runner in progress.

See docs/MASTER_ROADMAP.md for current phase and open work.

verify is an eleet.ai service offering. Part of the zeos venture factory control plane.