JSPM

Git-native prompt operations for teams that want to treat prompts like software assets.

Apastra keeps prompt specs, datasets, evaluators, suites, and baselines as plain files in your repo. Your coding agent reads those files to scaffold changes, run evals, compare against baselines, and validate contracts without requiring a hosted platform.

What Is This?

Apastra is a file-based protocol and skill pack for prompt engineering workflows.

Documentation

• Getting started
• Architecture overview
• API reference
• System vision

Quick Start

1. Install the skill pack

Two install paths — pick whichever fits your project.

Option A — Git clone (language-agnostic, recommended):

git clone --single-branch --depth 1 https://github.com/BintzGavin/apastra.git .agent/skills/apastra
.agent/skills/apastra/setup

Option B — npm:

npm install apastra

Either path installs to the same layout:

• .agent/skills/apastra/ — SKILL.md instructions your agent loads
• .agent/scripts/apastra/ — deterministic Python runtime + shell validators

The setup script auto-installs pyyaml and jsonschema (falls back to clear manual-install guidance on PEP-668 environments). npm's postinstall.sh does the same.

2. Scaffold your first prompt workflow

Ask your agent:

"Use the apastra-scaffold skill to create a prompt spec, dataset, evaluator, and suite for summarizing text"

You will get a repo-native setup like:

promptops/
├── prompts/summarize-v1.yaml
├── datasets/summarize-smoke.jsonl
├── evaluators/contains-keywords.yaml
└── suites/summarize-smoke.yaml

3. Run an eval

Ask your agent:

"Use the apastra-eval skill to run the summarize-smoke suite"

The agent loads the suite, renders the prompt for each case, calls the model, scores the outputs, and reports a scorecard.

Suite: summarize-smoke
Status: PASS

Metrics:
  keyword_recall: 0.85 (threshold: 0.60)

4. Save a baseline

Ask your agent:

"Use the apastra-baseline skill to set the current results as the baseline"

Future evals can now detect regressions automatically when prompt quality drops below the accepted threshold.

That is enough to start using apastra locally. CI and release automation are available when you want them, but they are not required to get value from the repo.

Note for AI agents: This README is the quickstart. For the full architectural model and design principles, start with docs/vision.md.

Included Skills

All skills install together — there is no per-skill install path. Once installed under .agent/skills/apastra/, your agent discovers each sub-skill by its SKILL.md.

Core Concepts

Prompt Spec

A YAML file defining a prompt with a stable ID, input variables, a template, and an optional output contract.

id: summarize-v1
variables:
  text: { type: string }
template: "Summarize: {{text}}"

Dataset

A .jsonl file of test cases — one JSON object per line with a case_id and inputs.

{"case_id": "case-1", "inputs": {"text": "..."}, "expected_outputs": {"should_contain": ["key", "words"]}}

Evaluator

A scoring rule — deterministic checks, schema validation, or AI judge grading.

id: keyword-check
type: deterministic
metrics: [keyword_recall]

Inline Assertions (Quick Mode)

For simple checks, skip the evaluator file entirely — put assertions directly on your test cases:

{"case_id": "case-1", "inputs": {"text": "..."}, "assert": [{"type": "contains", "value": "summary"}, {"type": "is-json"}]}

Built-in assertion types: equals, contains, icontains, contains-any, contains-all, regex, starts-with, is-json, contains-json, similar, llm-rubric, factuality, latency, cost. Negate any with not- prefix (e.g. not-contains).

Quick Eval (Single File)

For rapid iteration, combine prompt + cases + assertions into one file (promptops/evals/my-eval.yaml):

id: summarize-quick
prompt: "Summarize in {{max_length}} words: {{text}}"
cases:
  - id: short
    inputs: { text: "The fox jumps over the dog.", max_length: "10" }
    assert:
      - type: icontains
        value: "fox"
thresholds:
  pass_rate: 1.0

Graduate to the full spec/dataset/evaluator/suite structure as complexity grows.

Suite

A test configuration that ties everything together: which datasets, which evaluators, which models.

id: smoke
name: Smoke Suite
datasets: [summarize-smoke]
evaluators: [keyword-check]
model_matrix: [default]
thresholds: { keyword_recall: 0.6 }

Baseline & Regression

A baseline is a saved scorecard from a passing run. Future evals compare against it. If quality drops beyond allowed thresholds, it's a regression.

File Structure

In your project (after install)

.agent/
├── skills/apastra/       # Agent-facing SKILL.md files (eval, baseline, scaffold, …)
└── scripts/apastra/      # Deterministic runtime (Python + shell validators)
promptops/                # Created by the scaffold skill on first use
├── prompts/              # Prompt specs (YAML)
├── datasets/             # Test cases (JSONL)
├── evaluators/           # Scoring rules (YAML)
├── suites/               # Test configurations (YAML)
└── policies/             # Regression policies (allowed thresholds)
derived-index/
├── baselines/            # Known-good scorecards
└── regressions/          # Regression reports

In this repo (what gets shipped)

promptops/ here contains the runtime source that lands in your project's .agent/scripts/apastra/ at install time — schemas, validators, resolver, runs, harnesses. You do not copy this directory into your project directly; setup / postinstall.sh does that.

How the Agent Runs Evals

Your IDE agent is the harness. When you ask it to run an eval:

flowchart TD
  A[Read suite spec] --> B[Load dataset cases + evaluators]
  B --> C[For each case: render prompt template]
  C --> D[Call the model with rendered prompt]
  D --> E[Score output using evaluators]
  E --> F[Aggregate into scorecard]
  F --> G{Baseline exists?}
  G -->|Yes| H[Compare against baseline]
  G -->|No| I[Report scorecard only]
  H --> J[Regression report: PASS/FAIL]

Deterministic steps (prompt rendering, digest computation, scorecard normalization, baseline comparison, schema validation) are delegated to Python + shell scripts under .agent/scripts/apastra/. Your agent handles the LLM-dependent parts: calling the model and grading with judge evaluators. No hosted service, no SaaS dependency — just files, scripts, and your agent.

Scaling Up (Optional)

When you're ready for more structure, apastra supports:

GitHub Actions CI

Five pre-built workflows gate merges and automate promotions:

Git-First Consumption

Apps can pin prompts by commit SHA, tag, or semver — npm and pip both support Git dependencies natively:

# consumption.yaml
version: "1.0"
prompts:
  summarize-v1:
    pin: "abc123"  # commit SHA, tag, or semver

Resolution order: local override → workspace → git ref → packaged artifact.

Governed Releases

Principles

• Files in Git are the source of truth — not a database, not a platform
• Your agent is the harness — no framework lock-in
• Append-only artifacts — never mutate old results; create new records
• Reproducibility by default — content digests, environment metadata
• Local-first, CI-optional — start with zero infrastructure

Planned Expansions

Planned Refinements

• Simplified minimal mode — auto-detected when ≤3 prompt specs exist; only prompts/, evals/, and baselines/ directories
• Project-level config — promptops.config.yaml for default model, temperature, thresholds, and auto-baseline behavior
• MCP integration — support MCP tool definitions in prompt specs and provide an MCP server adapter for agent discovery
• First-class cost tracking — total cost in every run manifest, cost delta in regression reports, cost_budget field on suites
• Approachable terminology — "your agent" everywhere user-facing; "harness" reserved for technical specs

License

Apache-2.0