Package Exports
- llmist
- llmist/testing
Readme
llmist
Tools execute while the LLM streams. Any model. Clean API.
⚠️ EARLY WORK IN PROGRESS - This library is under active development. APIs may change without notice. Use in production at your own risk.
Most LLM libraries buffer the entire response before parsing tool calls. llmist parses incrementally.
Your gadgets (tools) fire the instant they're complete in the stream—giving your users immediate feedback. llmist implements its own function calling mechanism via a simple text-based block format. No JSON mode required. No native tool support needed. Works with OpenAI, Anthropic, and Gemini out of the box—extensible to any provider.
A fluent, async-first API lets you plug into any part of the agent loop. Fully typed. Composable. Your code stays clean.
🎯 Why llmist?
⚡ Streaming Tool ExecutionGadgets execute the moment their block is parsed—not after the response completes. Real-time UX without buffering. // Tool fires mid-stream
for await (const event of agent.run()) {
if (event.type === 'gadget_result')
updateUI(event.result); // Immediate
} |
🧩 Built-in Function Callingllmist implements its own tool calling via a simple block format. No Markers are fully configurable. |
🔌 Composable Agent APIFluent builder, async iterators, full TypeScript inference. Hook into any lifecycle point. Your code stays readable. const answer = await LLMist.createAgent()
.withModel('sonnet')
.withGadgets(Calculator, Weather)
.withHooks(HookPresets.monitoring())
.askAndCollect('What is 15 + 23?'); |
🚀 Quick Start
Installation
npm install llmist
# or
bun add llmist🖥️ Command Line Interface
# Quick completion
bunx llmist complete "Explain TypeScript generics" --model haiku
# Agent with tools
bunx llmist agent "Calculate 15 * 23" --gadget ./calculator.ts --model sonnet
# Pipe input
cat document.txt | llmist complete "Summarize" --model gpt-5-nano📖 CLI Reference | CLI Gadgets Guide
Your First Agent
import { LLMist, Gadget, z } from 'llmist';
// Define a tool (called "gadget" in llmist)
class Calculator extends Gadget({
description: 'Performs arithmetic operations',
schema: z.object({
operation: z.enum(['add', 'multiply', 'subtract', 'divide']),
a: z.number(),
b: z.number(),
}),
}) {
execute(params: this['params']): string {
const { operation, a, b } = params; // Automatically typed!
switch (operation) {
case 'add': return `${a + b}`;
case 'multiply': return `${a * b}`;
case 'subtract': return `${a - b}`;
case 'divide': return `${a / b}`;
default: throw new Error('Unknown operation');
}
}
}
// Create and run agent with fluent API
const answer = await LLMist.createAgent()
.withModel('gpt-5-nano') // Model shortcuts: sonnet, haiku, etc.
.withSystem('You are a helpful math assistant')
.withGadgets(Calculator)
.askAndCollect('What is 15 times 23?');
console.log(answer); // "15 times 23 equals 345"That's it!
📖 Getting Started Guide - Learn more in 5 minutes
✨ Key Features
🌐 Multi-Provider Support
// Use model shortcuts
.withModel('gpt-5-nano') // OpenAI gpt-5-nano
.withModel('sonnet') // Claude Sonnet 4.5
.withModel('haiku') // Claude Haiku 4.5
.withModel('flash') // Gemini 2.0 Flash
// Or full names
.withModel('openai:gpt-5-nano')
.withModel('anthropic:claude-sonnet-4-5')
.withModel('gemini:gemini-2.0-flash')Automatic provider discovery - Just set API keys as env vars (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY)
📖 Providers Guide | Model Catalog
🛠️ Flexible Gadgets (Tools)
Two ways to create tools:
// 1. Class-based with type safety
class Weather extends Gadget({
description: 'Get weather for a city',
schema: z.object({ city: z.string() }),
}) {
async execute(params: this['params']) {
// params is auto-typed!
const data = await fetch(`https://api.weather.com/${params.city}`);
return `Weather: ${data.temp}°C`;
}
}
// 2. Functional for simplicity
const calculator = createGadget({
description: 'Arithmetic operations',
schema: z.object({ operation: z.enum(['add']), a: z.number(), b: z.number() }),
execute: ({ operation, a, b }) => `${a + b}`,
});🪝 Lifecycle Hooks
Monitor, transform, and control agent execution with ready-to-use presets or custom hooks:
Quick start with presets:
import { LLMist, HookPresets } from 'llmist';
// Full monitoring suite (recommended for development)
await LLMist.createAgent()
.withHooks(HookPresets.monitoring())
.ask('Your prompt');
// Output: Logs + timing + token tracking + error logging
// Combine specific presets for focused monitoring
await LLMist.createAgent()
.withHooks(HookPresets.merge(
HookPresets.timing(),
HookPresets.tokenTracking()
))
.ask('Your prompt');Available presets:
logging()/logging({ verbose: true })- Event logging with optional detailstiming()- Execution time measurementstokenTracking()- Cumulative token usage and cost trackingerrorLogging()- Detailed error informationsilent()- No output (for testing)monitoring()- All-in-one preset combining logging, timing, tokens, and errorsmerge()- Combine multiple presets or add custom hooks
Production vs Development patterns:
// Environment-based configuration
const isDev = process.env.NODE_ENV === 'development';
const hooks = isDev
? HookPresets.monitoring({ verbose: true }) // Full visibility in dev
: HookPresets.merge(
HookPresets.errorLogging(), // Only errors in prod
HookPresets.tokenTracking() // Track costs
);
await LLMist.createAgent()
.withHooks(hooks)
.ask('Your prompt');Custom hooks for advanced control:
// Observers: read-only monitoring
.withHooks({
observers: {
onLLMCallComplete: async (ctx) => {
console.log(`Used ${ctx.usage?.totalTokens} tokens`);
await sendMetricsToDataDog(ctx);
},
},
})
// Interceptors: transform data in flight
.withHooks({
interceptors: {
interceptTextChunk: (chunk) => chunk.toUpperCase(),
},
})
// Controllers: control execution flow
.withHooks({
controllers: {
beforeLLMCall: async (ctx) => {
if (shouldCache(ctx)) {
return { action: 'skip', syntheticResponse: cachedResponse };
}
return { action: 'proceed' };
},
},
})📖 Hooks Guide | Examples
💬 Human-in-the-Loop
class AskUser extends Gadget({
description: 'Ask the user a question',
schema: z.object({ question: z.string() }),
}) {
execute(params: this['params']) {
throw new HumanInputException(params.question);
}
}
await LLMist.createAgent()
.withGadgets(AskUser)
.onHumanInput(async (question) => {
return await promptUser(question);
})
.ask('Help me plan my vacation');📖 Human-in-the-Loop Guide | Examples
⚡ Streaming & Event Handling
// Collect all text
const answer = await LLMist.createAgent()
.withModel('haiku')
.askAndCollect('Tell me a joke');
// Handle specific events
await LLMist.createAgent()
.withModel('sonnet')
.withGadgets(Calculator)
.askWith('Calculate 2 + 2', {
onText: (text) => console.log('LLM:', text),
onGadgetCall: (call) => console.log('Calling:', call.gadgetName),
onGadgetResult: (result) => console.log('Result:', result.result),
});
// Manual control
const agent = LLMist.createAgent().withModel('gpt-5-nano').ask('Question');
for await (const event of agent.run()) {
if (event.type === 'text') console.log(event.content);
}🧪 Mock Testing
import { LLMist, mockLLM, createMockClient } from 'llmist';
mockLLM()
.forModel('gpt-5')
.whenMessageContains('calculate')
.returns('The answer is 42')
.register();
const mockClient = createMockClient();
const answer = await mockClient.createAgent()
.withModel('gpt-5')
.askAndCollect('Calculate 2 + 2');
console.log(answer); // "The answer is 42" - no API call made!📊 Model Catalog & Cost Estimation
const client = new LLMist();
// Get model specs
const gpt5 = client.modelRegistry.getModelSpec('gpt-5');
console.log(gpt5.contextWindow); // 272000
console.log(gpt5.pricing.input); // 1.25 per 1M tokens
// Estimate costs
const cost = client.modelRegistry.estimateCost('gpt-5', 10_000, 2_000);
console.log(`$${cost.totalCost.toFixed(4)}`);
// Find cheapest model
const cheapest = client.modelRegistry.getCheapestModel(10_000, 2_000);📖 Model Catalog Guide | Custom Models
🔢 Native Token Counting
const messages = [
{ role: 'system', content: 'You are helpful' },
{ role: 'user', content: 'Explain quantum computing' }
];
const tokens = await client.countTokens('openai:gpt-5', messages);
const cost = client.modelRegistry.estimateCost('gpt-5', tokens, 1000);Uses provider-specific methods (tiktoken for OpenAI, native APIs for Anthropic/Gemini).
📚 Documentation
Getting Started
- Getting Started - Your first agent in 5 minutes
- Configuration - All available options
- Quick Methods - Simple APIs for basic tasks
Core Concepts
- Gadgets (Tools) - Creating custom functions
- Block Format - Parameter syntax reference
- Hooks - Lifecycle monitoring and control
- Streaming - Real-time response handling
- Human-in-the-Loop - Interactive workflows
Advanced
- Providers - Multi-provider configuration
- Model Catalog - Querying models and costs
- Custom Models - Register fine-tuned models
- Error Handling - Recovery strategies
- Testing - Mocking and test strategies
Reference
- CLI Reference - Command-line interface
- Architecture - Technical deep-dive
- Debugging - Capture raw prompts/responses
- Troubleshooting - Common issues
🎓 Examples
Comprehensive examples are available in the examples/ directory:
| Example | Description |
|---|---|
| 01-basic-usage.ts | Simple agent with calculator gadget |
| 02-custom-gadgets.ts | Async gadgets, validation, loop termination |
| 03-hooks.ts | Lifecycle hooks for monitoring |
| 04-human-in-loop.ts | Interactive conversations |
| 05-streaming.ts | Real-time streaming |
| 06-model-catalog.ts | Model queries and cost estimation |
| 07-logging.ts | Logging and debugging |
| 13-syntactic-sugar.ts | Fluent API showcase |
Run any example:
bun install && bun run build
bunx tsx examples/01-basic-usage.tsSee examples/README.md for full list and details.
🏗️ Architecture
llmist follows SOLID principles with a composable architecture.
Key components:
- LLMist - Provider-agnostic streaming client
- Agent - Full agent loop with automatic orchestration
- StreamProcessor - Process LLM streams with custom event loops
- GadgetExecutor - Execute tools with timeout and error handling
- GadgetRegistry - Registry for available tools
📖 Architecture Guide for detailed design documentation
🧪 Development
bun install
# Run tests
bun test # All tests
bun run test:unit # Unit tests only
bun run test:e2e # E2E tests only
# Build and lint
bun run build
bun run lint
bun run format🤝 Contributing
Contributions welcome! See CONTRIBUTING.md for guidelines, commit conventions, and release process.
📄 License
MIT - see LICENSE for details.
🔗 Links
Made with 🤪 by the llmist team