JSPM

  • Created
  • Published
  • Downloads 933
  • Score
    100M100P100Q105975F
  • License MIT

Universal TypeScript LLM client with streaming-first agent framework. Works with any model - no structured outputs or native tool calling required. Implements its own flexible grammar for function calling.

Package Exports

  • llmist
  • llmist/testing

Readme

llmist

CI codecov npm version License

Universal TypeScript LLM client with streaming-first tool execution and simple, extensible agent framework

โš ๏ธ EARLY WORK IN PROGRESS - This library is under active development. APIs may change without notice. Use in production at your own risk.

llmist is an asynchonous, streaming-first, provider-agnostic LLM client that makes it easy to build AI agents with any modelโ€”no structured outputs or native tool calling required. Switch between OpenAI, Anthropic, and Gemini without changing your code, plug into any part of the Agent workflow, have tools (Gadgets) triggered while still streaming.


๐ŸŽฏ Why llmist?

  • ๐ŸŒ Universal - Works with any LLM provider (OpenAI, Anthropic, Gemini, custom)
  • ๐Ÿ“ No Structured Outputs - Flexible TOML/YAML/JSON grammar works with any text model
  • โšก Streaming-First - Built for real-time responses and efficient error handling
  • ๐Ÿช Powerful Hooks - Monitor, customize, and control every step of execution
  • ๐ŸŽจ Beautiful API - Fluent builder pattern with model shortcuts and presets
  • ๐Ÿงช Testing-Friendly - Built-in mocking system for zero-cost testing

๐Ÿš€ Quick Start

Installation

npm install llmist
# or
bun add llmist

๐Ÿ–ฅ๏ธ Command Line Interface

# Quick completion
bunx llmist complete "Explain TypeScript generics" --model haiku

# Agent with tools
bunx llmist agent "Calculate 15 * 23" --gadget ./calculator.ts --model sonnet

# Pipe input
cat document.txt | llmist complete "Summarize" --model gpt-5-nano

Built-in gadgets are included by default in agent mode:

  • AskUser - Prompts for user input when clarification is needed
  • TellUser - Displays important messages (info/success/warning/error) and can end conversations
# Disable built-in gadgets
bunx llmist agent "Task" --no-builtins -g ./my-tools.ts

# Override parameter format (default: toml, options: toml, yaml, json, auto)
bunx llmist agent "Task" --parameter-format yaml -g ./my-tools.ts

๐Ÿ“– CLI Reference | CLI Gadgets Guide

Your First Agent

import { LLMist, Gadget, z } from 'llmist';

// Define a tool (called "gadget" in llmist)
class Calculator extends Gadget({
  description: 'Performs arithmetic operations',
  schema: z.object({
    operation: z.enum(['add', 'multiply', 'subtract', 'divide']),
    a: z.number(),
    b: z.number(),
  }),
}) {
  execute(params: this['params']): string {
    const { operation, a, b } = params; // Automatically typed!
    switch (operation) {
      case 'add': return `${a + b}`;
      case 'multiply': return `${a * b}`;
      case 'subtract': return `${a - b}`;
      case 'divide': return `${a / b}`;
      default: throw new Error('Unknown operation');
    }
  }
}

// Create and run agent with fluent API
const answer = await LLMist.createAgent()
  .withModel('gpt-5-nano')  // Model shortcuts: sonnet, haiku, etc.
  .withSystem('You are a helpful math assistant')
  .withGadgets(Calculator)
  .askAndCollect('What is 15 times 23?');

console.log(answer); // "15 times 23 equals 345"

That's it! N

๐Ÿ“– Getting Started Guide - Learn more in 5 minutes


โœจ Key Features

๐ŸŒ Multi-Provider Support

// Use model shortcuts
.withModel('gpt-5-nano')    // OpenAI gpt-5-nano
.withModel('sonnet')       // Claude Sonnet 4.5
.withModel('haiku')        // Claude Haiku 4.5
.withModel('flash')        // Gemini 2.0 Flash

// Or full names
.withModel('openai:gpt-5-nano')
.withModel('anthropic:claude-sonnet-4-5')
.withModel('gemini:gemini-2.0-flash')

Automatic provider discovery - Just set API keys as env vars (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY)

๐Ÿ“– Providers Guide | Model Catalog

๐Ÿ› ๏ธ Flexible Gadgets (Tools)

Two ways to create tools:

// 1. Class-based with type safety
class Weather extends Gadget({
  description: 'Get weather for a city',
  schema: z.object({ city: z.string() }),
}) {
  async execute(params: this['params']) {
    // params is auto-typed!
    const data = await fetch(`https://api.weather.com/${params.city}`);
    return `Weather: ${data.temp}ยฐC`;
  }
}

// 2. Functional for simplicity
const calculator = createGadget({
  description: 'Arithmetic operations',
  schema: z.object({ operation: z.enum(['add']), a: z.number(), b: z.number() }),
  execute: ({ operation, a, b }) => `${a + b}`,
});

๐Ÿ“– Gadgets Guide | Examples

๐Ÿช Lifecycle Hooks

Monitor, transform, and control agent execution with ready-to-use presets or custom hooks:

Quick start with presets:

import { LLMist, HookPresets } from 'llmist';

// Full monitoring suite (recommended for development)
await LLMist.createAgent()
  .withHooks(HookPresets.monitoring())
  .ask('Your prompt');
// Output: Logs + timing + token tracking + error logging

// Combine specific presets for focused monitoring
await LLMist.createAgent()
  .withHooks(HookPresets.merge(
    HookPresets.timing(),
    HookPresets.tokenTracking()
  ))
  .ask('Your prompt');

Available presets:

  • logging() / logging({ verbose: true }) - Event logging with optional details
  • timing() - Execution time measurements
  • tokenTracking() - Cumulative token usage and cost tracking
  • errorLogging() - Detailed error information
  • silent() - No output (for testing)
  • monitoring() - All-in-one preset combining logging, timing, tokens, and errors
  • merge() - Combine multiple presets or add custom hooks

Production vs Development patterns:

// Environment-based configuration
const isDev = process.env.NODE_ENV === 'development';
const hooks = isDev
  ? HookPresets.monitoring({ verbose: true })  // Full visibility in dev
  : HookPresets.merge(
      HookPresets.errorLogging(),              // Only errors in prod
      HookPresets.tokenTracking()              // Track costs
    );

await LLMist.createAgent()
  .withHooks(hooks)
  .ask('Your prompt');

Custom hooks for advanced control:

// Observers: read-only monitoring
.withHooks({
  observers: {
    onLLMCallComplete: async (ctx) => {
      console.log(`Used ${ctx.usage?.totalTokens} tokens`);
      await sendMetricsToDataDog(ctx);
    },
  },
})

// Interceptors: transform data in flight
.withHooks({
  interceptors: {
    interceptTextChunk: (chunk) => chunk.toUpperCase(),
  },
})

// Controllers: control execution flow
.withHooks({
  controllers: {
    beforeLLMCall: async (ctx) => {
      if (shouldCache(ctx)) {
        return { action: 'skip', syntheticResponse: cachedResponse };
      }
      return { action: 'proceed' };
    },
  },
})

๐Ÿ“– Hooks Guide | Examples

๐Ÿ’ฌ Human-in-the-Loop

class AskUser extends Gadget({
  description: 'Ask the user a question',
  schema: z.object({ question: z.string() }),
}) {
  execute(params: this['params']) {
    throw new HumanInputException(params.question);
  }
}

await LLMist.createAgent()
  .withGadgets(AskUser)
  .onHumanInput(async (question) => {
    return await promptUser(question);
  })
  .ask('Help me plan my vacation');

๐Ÿ“– Human-in-the-Loop Guide | Examples

โšก Streaming & Event Handling

// Collect all text
const answer = await LLMist.createAgent()
  .withModel('haiku')
  .askAndCollect('Tell me a joke');

// Handle specific events
await LLMist.createAgent()
  .withModel('sonnet')
  .withGadgets(Calculator)
  .askWith('Calculate 2 + 2', {
    onText: (text) => console.log('LLM:', text),
    onGadgetCall: (call) => console.log('Calling:', call.gadgetName),
    onGadgetResult: (result) => console.log('Result:', result.result),
  });

// Manual control
const agent = LLMist.createAgent().withModel('gpt-5-nano').ask('Question');
for await (const event of agent.run()) {
  if (event.type === 'text') console.log(event.content);
}

๐Ÿ“– Streaming Guide | Examples

๐Ÿงช Mock Testing

import { LLMist, mockLLM, createMockClient } from 'llmist';

mockLLM()
  .forModel('gpt-5')
  .whenMessageContains('calculate')
  .returns('The answer is 42')
  .register();

const mockClient = createMockClient();
const answer = await mockClient.createAgent()
  .withModel('gpt-5')
  .askAndCollect('Calculate 2 + 2');

console.log(answer); // "The answer is 42" - no API call made!

๐Ÿ“– Testing Guide | Examples

๐Ÿ“Š Model Catalog & Cost Estimation

const client = new LLMist();

// Get model specs
const gpt5 = client.modelRegistry.getModelSpec('gpt-5');
console.log(gpt5.contextWindow);    // 272000
console.log(gpt5.pricing.input);    // 1.25 per 1M tokens

// Estimate costs
const cost = client.modelRegistry.estimateCost('gpt-5', 10_000, 2_000);
console.log(`$${cost.totalCost.toFixed(4)}`);

// Find cheapest model
const cheapest = client.modelRegistry.getCheapestModel(10_000, 2_000);

๐Ÿ“– Model Catalog Guide | Custom Models

๐Ÿ”ข Native Token Counting

const messages = [
  { role: 'system', content: 'You are helpful' },
  { role: 'user', content: 'Explain quantum computing' }
];

const tokens = await client.countTokens('openai:gpt-5', messages);
const cost = client.modelRegistry.estimateCost('gpt-5', tokens, 1000);

Uses provider-specific methods (tiktoken for OpenAI, native APIs for Anthropic/Gemini).


๐Ÿ“š Documentation

Getting Started

Core Concepts

Advanced

Reference


๐ŸŽ“ Examples

Comprehensive examples are available in the examples/ directory:

Example Description
01-basic-usage.ts Simple agent with calculator gadget
02-custom-gadgets.ts Async gadgets, validation, loop termination
03-hooks.ts Lifecycle hooks for monitoring
04-human-in-loop.ts Interactive conversations
05-streaming.ts Real-time streaming
06-model-catalog.ts Model queries and cost estimation
07-logging.ts Logging and debugging
13-syntactic-sugar.ts Fluent API showcase

Run any example:

bun install && bun run build
bunx tsx examples/01-basic-usage.ts

See examples/README.md for full list and details.


๐Ÿ—๏ธ Architecture

llmist follows SOLID principles with a composable architecture.

Key components:

  • LLMist - Provider-agnostic streaming client
  • Agent - Full agent loop with automatic orchestration
  • StreamProcessor - Process LLM streams with custom event loops
  • GadgetExecutor - Execute tools with timeout and error handling
  • GadgetRegistry - Registry for available tools

๐Ÿ“– Architecture Guide for detailed design documentation


๐Ÿงช Development

bun install

# Run tests
bun test              # All tests
bun run test:unit     # Unit tests only
bun run test:e2e      # E2E tests only

# Build and lint
bun run build
bun run lint
bun run format

๐Ÿค Contributing

Contributions welcome! Please ensure:

  1. โœ… All tests pass: bun test
  2. โœ… Code is formatted: bun run format
  3. โœ… Linting passes: bun run lint
  4. โœ… Types are properly defined
  5. โœ… Examples/docs updated for API changes

Commit Message Convention

This project follows Conventional Commits specification. All commit messages must be formatted as:

<type>(<scope>): <subject>

Types:

  • feat: - New feature (triggers minor version bump)
  • fix: - Bug fix (triggers patch version bump)
  • docs: - Documentation only changes
  • style: - Code style changes (formatting, missing semi-colons, etc)
  • refactor: - Code refactoring without feature changes
  • perf: - Performance improvements
  • test: - Adding or updating tests
  • build: - Build system or dependency changes
  • ci: - CI configuration changes
  • chore: - Other changes that don't modify src or test files

Breaking Changes: Add BREAKING CHANGE: in the footer to trigger major version bump.

Examples:

feat(agent): add support for streaming tool calls
fix(cli): prevent crash on invalid gadget path
docs: update API documentation for v2

Note: Git hooks will validate your commit messages locally.

Release Process

Releases are fully automated using semantic-release:

  1. Merge PR to main branch
  2. CI workflow runs automatically
  3. If CI passes, release workflow:
    • Analyzes commits since last release
    • Determines version bump based on commit types
    • Updates package.json and CHANGELOG.md
    • Creates git tag and GitHub release
    • Publishes to npm
    • Syncs changes back to dev branch

No manual version bumps needed!


๐Ÿ“„ License

MIT - see LICENSE for details.



Made with ๐Ÿคช by the llmist team