JSPM

  • Created
  • Published
  • Downloads 933
  • Score
    100M100P100Q106000F
  • License MIT

TypeScript LLM client with streaming tool execution. Tools fire mid-stream. Built-in function calling works with any model—no structured outputs or native tool support required.

Package Exports

  • llmist
  • llmist/testing

Readme

llmist

CI codecov npm version License

Tools execute while the LLM streams. Any model. Clean API.

⚠️ EARLY WORK IN PROGRESS - This library is under active development. APIs may change without notice. Use in production at your own risk.

Most LLM libraries buffer the entire response before parsing tool calls. llmist parses incrementally.

Your gadgets (tools) fire the instant they're complete in the stream—giving your users immediate feedback. llmist implements its own function calling mechanism via a simple text-based block format. No JSON mode required. No native tool support needed. Works with OpenAI, Anthropic, and Gemini out of the box—extensible to any provider.

A fluent, async-first API lets you plug into any part of the agent loop. Fully typed. Composable. Your code stays clean.


🎯 Why llmist?

⚡ Streaming Tool Execution

Gadgets execute the moment their block is parsed—not after the response completes. Real-time UX without buffering.

// Tool fires mid-stream
for await (const event of agent.run()) {
  if (event.type === 'gadget_result')
    updateUI(event.result); // Immediate
}

🧩 Built-in Function Calling

llmist implements its own tool calling via a simple block format. No response_format: json. No native tool support needed. Works with any model from supported providers.

!!!GADGET_START[Calculator]
!!!ARG[operation] add
!!!ARG[a] 15
!!!ARG[b] 23
!!!GADGET_END

Markers are fully configurable.

🔌 Composable Agent API

Fluent builder, async iterators, full TypeScript inference. Hook into any lifecycle point. Your code stays readable.

const answer = await LLMist.createAgent()
  .withModel('sonnet')
  .withGadgets(Calculator, Weather)
  .withHooks(HookPresets.monitoring())
  .askAndCollect('What is 15 + 23?');

🚀 Quick Start

Installation

npm install llmist
# or
bun add llmist

🖥️ Command Line Interface

# Initialize config (creates ~/.llmist/cli.toml)
bunx llmist init

# Quick completion
bunx llmist complete "Explain TypeScript generics" --model haiku

# Agent with tools
bunx llmist agent "Calculate 15 * 23" --gadget ./calculator.ts --model sonnet

# External gadgets (npm/git - auto-installed)
bunx llmist agent "Browse apple.com" --gadget webasto --model sonnet
bunx llmist agent "Screenshot google.com" --gadget webasto:minimal
bunx llmist agent "Navigate to site" --gadget git+https://github.com/user/gadgets.git

# Pipe input
cat document.txt | llmist complete "Summarize" --model gpt-5-nano

📖 CLI Reference | CLI Gadgets Guide

Your First Agent

import { LLMist, Gadget, z } from 'llmist';

// Define a tool (called "gadget" in llmist)
class Calculator extends Gadget({
  description: 'Performs arithmetic operations',
  schema: z.object({
    operation: z.enum(['add', 'multiply', 'subtract', 'divide']),
    a: z.number(),
    b: z.number(),
  }),
}) {
  execute(params: this['params']): string {
    const { operation, a, b } = params; // Automatically typed!
    switch (operation) {
      case 'add': return `${a + b}`;
      case 'multiply': return `${a * b}`;
      case 'subtract': return `${a - b}`;
      case 'divide': return `${a / b}`;
      default: throw new Error('Unknown operation');
    }
  }
}

// Create and run agent with fluent API
const answer = await LLMist.createAgent()
  .withModel('gpt-5-nano')  // Model shortcuts: sonnet, haiku, etc.
  .withSystem('You are a helpful math assistant')
  .withGadgets(Calculator)
  .askAndCollect('What is 15 times 23?');

console.log(answer); // "15 times 23 equals 345"

That's it!

📖 Getting Started Guide - Learn more in 5 minutes


✨ Key Features

🌐 Multi-Provider Support

// Use model shortcuts
.withModel('gpt-5-nano')    // OpenAI gpt-5-nano
.withModel('sonnet')       // Claude Sonnet 4.5
.withModel('haiku')        // Claude Haiku 4.5
.withModel('flash')        // Gemini 2.0 Flash

// Or full names
.withModel('openai:gpt-5-nano')
.withModel('anthropic:claude-sonnet-4-5')
.withModel('gemini:gemini-2.0-flash')

Automatic provider discovery - Just set API keys as env vars (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY)

📖 Providers Guide | Model Catalog

🛠️ Flexible Gadgets (Tools)

Two ways to create tools:

// 1. Class-based with type safety
class Weather extends Gadget({
  description: 'Get weather for a city',
  schema: z.object({ city: z.string() }),
}) {
  async execute(params: this['params']) {
    // params is auto-typed!
    const data = await fetch(`https://api.weather.com/${params.city}`);
    return `Weather: ${data.temp}°C`;
  }
}

// 2. Functional for simplicity
const calculator = createGadget({
  description: 'Arithmetic operations',
  schema: z.object({ operation: z.enum(['add']), a: z.number(), b: z.number() }),
  execute: ({ operation, a, b }) => `${a + b}`,
});

📖 Gadgets Guide | Examples

🪝 Lifecycle Hooks

Monitor, transform, and control agent execution with ready-to-use presets or custom hooks:

Quick start with presets:

import { LLMist, HookPresets } from 'llmist';

// Full monitoring suite (recommended for development)
await LLMist.createAgent()
  .withHooks(HookPresets.monitoring())
  .ask('Your prompt');
// Output: Logs + timing + token tracking + error logging

// Combine specific presets for focused monitoring
await LLMist.createAgent()
  .withHooks(HookPresets.merge(
    HookPresets.timing(),
    HookPresets.tokenTracking()
  ))
  .ask('Your prompt');

Available presets:

  • logging() / logging({ verbose: true }) - Event logging with optional details
  • timing() - Execution time measurements
  • tokenTracking() - Cumulative token usage and cost tracking
  • errorLogging() - Detailed error information
  • silent() - No output (for testing)
  • monitoring() - All-in-one preset combining logging, timing, tokens, and errors
  • merge() - Combine multiple presets or add custom hooks

Production vs Development patterns:

// Environment-based configuration
const isDev = process.env.NODE_ENV === 'development';
const hooks = isDev
  ? HookPresets.monitoring({ verbose: true })  // Full visibility in dev
  : HookPresets.merge(
      HookPresets.errorLogging(),              // Only errors in prod
      HookPresets.tokenTracking()              // Track costs
    );

await LLMist.createAgent()
  .withHooks(hooks)
  .ask('Your prompt');

Custom hooks for advanced control:

// Observers: read-only monitoring
.withHooks({
  observers: {
    onLLMCallComplete: async (ctx) => {
      console.log(`Used ${ctx.usage?.totalTokens} tokens`);
      await sendMetricsToDataDog(ctx);
    },
  },
})

// Interceptors: transform data in flight
.withHooks({
  interceptors: {
    interceptTextChunk: (chunk) => chunk.toUpperCase(),
  },
})

// Controllers: control execution flow
.withHooks({
  controllers: {
    beforeLLMCall: async (ctx) => {
      if (shouldCache(ctx)) {
        return { action: 'skip', syntheticResponse: cachedResponse };
      }
      return { action: 'proceed' };
    },
  },
})

📖 Hooks Guide | Examples

💬 Human-in-the-Loop

class AskUser extends Gadget({
  description: 'Ask the user a question',
  schema: z.object({ question: z.string() }),
}) {
  execute(params: this['params']) {
    throw new HumanInputRequiredException(params.question);
  }
}

await LLMist.createAgent()
  .withGadgets(AskUser)
  .onHumanInput(async (question) => {
    return await promptUser(question);
  })
  .ask('Help me plan my vacation');

📖 Human-in-the-Loop Guide | Examples

⚡ Streaming & Event Handling

// Collect all text
const answer = await LLMist.createAgent()
  .withModel('haiku')
  .askAndCollect('Tell me a joke');

// Handle specific events
await LLMist.createAgent()
  .withModel('sonnet')
  .withGadgets(Calculator)
  .askWith('Calculate 2 + 2', {
    onText: (text) => console.log('LLM:', text),
    onGadgetCall: (call) => console.log('Calling:', call.gadgetName),
    onGadgetResult: (result) => console.log('Result:', result.result),
  });

// Manual control
const agent = LLMist.createAgent().withModel('gpt-5-nano').ask('Question');
for await (const event of agent.run()) {
  if (event.type === 'text') console.log(event.content);
}

📖 Streaming Guide | Examples

🔗 Gadget Dependencies (DAG Execution)

LLMs can specify execution order between gadgets. Independent gadgets run in parallel; dependent gadgets wait for their dependencies. Failed dependencies automatically skip downstream gadgets.

📖 Block Format | Example

🧪 Mock Testing

import { LLMist, mockLLM, createMockClient } from 'llmist';

mockLLM()
  .forModel('gpt-5')
  .whenMessageContains('calculate')
  .returns('The answer is 42')
  .register();

const mockClient = createMockClient();
const answer = await mockClient.createAgent()
  .withModel('gpt-5')
  .askAndCollect('Calculate 2 + 2');

console.log(answer); // "The answer is 42" - no API call made!

📖 Testing Guide | Examples

📊 Model Catalog & Cost Estimation

const client = new LLMist();

// Get model specs
const gpt5 = client.modelRegistry.getModelSpec('gpt-5');
console.log(gpt5.contextWindow);    // 272000
console.log(gpt5.pricing.input);    // 1.25 per 1M tokens

// Estimate costs
const cost = client.modelRegistry.estimateCost('gpt-5', 10_000, 2_000);
console.log(`$${cost.totalCost.toFixed(4)}`);

// Find cheapest model
const cheapest = client.modelRegistry.getCheapestModel(10_000, 2_000);

📖 Model Catalog Guide | Custom Models

🔢 Native Token Counting

const messages = [
  { role: 'system', content: 'You are helpful' },
  { role: 'user', content: 'Explain quantum computing' }
];

const tokens = await client.countTokens('openai:gpt-5', messages);
const cost = client.modelRegistry.estimateCost('gpt-5', tokens, 1000);

Uses provider-specific methods (tiktoken for OpenAI, native APIs for Anthropic/Gemini).


📚 Documentation

Getting Started

Core Concepts

Advanced

Reference


🎓 Examples

Comprehensive examples are available in the examples/ directory:

Example Description
01-basic-usage.ts Simple agent with calculator gadget
02-custom-gadgets.ts Async gadgets, validation, loop termination
03-hooks.ts Lifecycle hooks for monitoring
04-human-in-loop.ts Interactive conversations
05-streaming.ts Real-time streaming
06-model-catalog.ts Model queries and cost estimation
07-logging.ts Logging and debugging
13-syntactic-sugar.ts Fluent API showcase
11-gadget-dependencies.ts Gadget dependencies (DAG execution)
20-external-gadgets.ts npm/git gadget packages

Run any example:

bun install && bun run build
bunx tsx examples/01-basic-usage.ts

See examples/README.md for full list and details.


🏗️ Architecture

llmist follows SOLID principles with a composable architecture.

Key components:

  • LLMist - Provider-agnostic streaming client
  • Agent - Full agent loop with automatic orchestration
  • StreamProcessor - Process LLM streams with custom event loops
  • GadgetExecutor - Execute tools with timeout and error handling
  • GadgetRegistry - Registry for available tools

📖 Architecture Guide for detailed design documentation


🧪 Development

bun install

# Run tests
bun test              # All tests
bun run test:unit     # Unit tests only
bun run test:e2e      # E2E tests only

# Build and lint
bun run build
bun run lint
bun run format

🤝 Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines, commit conventions, and release process.


📄 License

MIT - see LICENSE for details.



Made with 🤪 by the llmist team