just-bash-wiki

A just-bash plugin that implements Karpathy's LLM Wiki pattern — a persistent, LLM-maintained knowledge base with semantic search.

Built on top of just-bash-data (db + vec).

Install

npm install just-bash-wiki

Peer dependency: just-bash >= 2.14.0

Quick Start

import { Bash, InMemoryFs } from "just-bash";
import { createWikiPlugin } from "just-bash-wiki";

const bash = new Bash({
  fs: new InMemoryFs({}),
  customCommands: createWikiPlugin({ rootDir: "/wiki" }),
});

// Initialize wiki (creates collections + vector indexes)
await bash.exec(`wiki init --dim=1536`);

// Add a source
await bash.exec(`wiki source add '{"title":"AI Overview","type":"article","content":"..."}'`);

// Create pages
await bash.exec(`wiki page create '{"slug":"ai","title":"Artificial Intelligence","type":"concept","content":"# AI\\n...","tags":["ai"],"links_to":["ml"]}'`);

// Store embeddings for semantic search
await bash.exec(`wiki embed page ai '[0.1, 0.2, ...]'`);

// Search by vector similarity
await bash.exec(`wiki search '[0.1, 0.2, ...]' --k=5`);

// Health check
await bash.exec(`wiki lint`);

How It Works

The LLM uses wiki as a bash tool to maintain a structured knowledge base. The pattern has three layers:

Three core operations:

• Ingest — add a source, extract information, create/update wiki pages, store embeddings
• Query — semantic search across pages, read relevant content, synthesize answers
• Lint — find orphan pages, broken links, missing embeddings, unreferenced sources

API

createWikiPlugin(opts?)

Returns an array of Command objects (includes db, vec, and wiki commands).

interface WikiOptions {
  rootDir?: string;        // Data directory (default: "/wiki")
  encryptionKey?: string;  // AES-256-GCM encryption key
  authSecret?: string;     // JWT signing secret
  salt?: string;           // PBKDF2 salt prefix
  embeddingDim?: number;   // Vector dimension (default: 1536)
  metric?: string;         // Similarity metric (default: "cosine")
  quantize?: string;       // Quantization format (default: "float32")
}

Command Reference

Initialization

wiki init [--dim=1536] [--metric=cosine] [--quantize=float32]

Creates all collections and indexes. Safe to call multiple times — skips existing collections.

Sources

# Add a source document
wiki source add '{"title":"...","type":"article","content":"...","url":"...","author":"..."}'

# List sources (optionally filter by type)
wiki source list [--type=article]

# Get a source by ID
wiki source get <id>

# Count sources
wiki source count

Source fields: title (required), type, content, url, author, date. Auto-added: ingested_at, status.

Pages

# Create a page
wiki page create '{"slug":"ai","title":"Artificial Intelligence","type":"concept","content":"# AI\n...","tags":["ai","tech"],"links_to":["ml","neural-nets"],"source_ids":["src-id-1"]}'

# Update a page (MongoDB-style update operators)
wiki page update <slug> '{"$set":{"content":"...","tags":["ai","updated"]}}'

# Get a page by slug
wiki page get <slug>

# List pages with optional filters
wiki page list [--type=concept] [--tag=ai]

# Delete a page (cleans up cross-references and embeddings)
wiki page delete <slug>

# Find pages with no inbound links
wiki page orphans

Page fields: slug (required, unique), title (required), type, content, tags, links_to, source_ids. Auto-managed: linked_from, created_at, updated_at.

Page types: entity, concept, source-summary, comparison, synthesis, overview, index (or any custom type).

Embeddings

# Store/update a page embedding
wiki embed page <slug> '[0.1, 0.2, ...]'

# Store/update a source embedding
wiki embed source <id> '[0.1, 0.2, ...]'

Embeddings are stored in vector collections (page_embeddings, source_embeddings) for semantic search. The embedding dimension must match what was set in wiki init --dim=N.

Search

# Search pages by vector similarity
wiki search '[0.1, 0.2, ...]' --k=10

# Search sources only
wiki search '[0.1, 0.2, ...]' --k=5 --type=sources

# Search across pages AND sources
wiki search '[0.1, 0.2, ...]' --k=10 --type=all

Returns results sorted by cosine similarity with scores and metadata.

Lint

wiki lint

Runs a health check and reports issues:

Log

# View recent log entries
wiki log [--last=20] [--type=ingest]

# Add a custom log entry
wiki log add '{"type":"note","summary":"Started research on topic X"}'

All wiki operations are automatically logged with timestamps.

Stats

wiki stats

Returns a comprehensive overview: page/source/log counts, pages grouped by type, vector collection stats, and recent activity.

Index

wiki index

Returns all pages grouped by type with their slugs, titles, tags, and last update timestamps.

Direct Access to db and vec

The wiki plugin includes all just-bash-data commands. You can use db and vec directly for advanced operations:

# Aggregation: find most-linked pages
db pages aggregate '[{"$unwind":"$linked_from"},{"$group":{"_id":"$slug","inbound":{"$sum":1}}},{"$sort":{"inbound":-1}}]'

# Complex queries
db pages find '{"$or":[{"tags":{"$contains":"ai"}},{"type":"overview"}]}'

# Cross-collection vector search
vec search-across "page_embeddings,source_embeddings" '[0.1,...]' --k=10

Example: LLM Agent Workflow

A typical LLM agent session using just-bash-wiki:

1. Agent receives a new article to process
2. wiki source add '{"title":"...","content":"..."}'
3. Agent reads the source content and extracts key entities/concepts
4. wiki page create '{"slug":"entity-name",...}'  (for each entity)
5. wiki page update existing-page '{"$set":{"content":"updated..."}}'  (update related pages)
6. wiki embed page entity-name '[...]'  (store embeddings)
7. wiki lint  (check for issues)
8. wiki stats  (verify state)

When answering questions:

1. Generate embedding for the question
2. wiki search '[...]' --k=5  (find relevant pages)
3. wiki page get relevant-slug  (read full content)
4. Synthesize answer from page content
5. Optionally: wiki page create '{"slug":"analysis-topic",...}'  (save the answer as a new page)

Architecture

┌─────────────────────────────────────────┐
│              LLM Agent                  │
│         (generates bash commands)       │
└─────────────┬───────────────────────────┘
              │ tool_use
┌─────────────▼───────────────────────────┐
│            just-bash                    │
│      (sandboxed bash interpreter)       │
├─────────────────────────────────────────┤
│          just-bash-wiki                 │
│    ┌──────────┬──────────┬────────┐     │
│    │  wiki    │   db     │  vec   │     │
│    │ command  │ command  │command │     │
│    └────┬─────┴────┬─────┴───┬────┘     │
│         │          │         │          │
│    ┌────▼──────────▼─────────▼────┐     │
│    │      just-bash-data          │     │
│    │  ┌──────────┐ ┌───────────┐  │     │
│    │  │ DocStore │ │VectorStore│  │     │
│    │  │(js-doc-  │ │(js-vector-│  │     │
│    │  │  store)  │ │  store)   │  │     │
│    │  └──────────┘ └───────────┘  │     │
│    └──────────────────────────────┘     │
├─────────────────────────────────────────┤
│         InMemoryFs / ReadWriteFs        │
│          (virtual filesystem)           │
└─────────────────────────────────────────┘

Collections Created by wiki init

License

MIT