JSPM

  • ESM via JSPM
  • ES Module Entrypoint
  • Export Map
  • Keywords
  • License
  • Repository URL
  • TypeScript Types
  • README
  • Created
  • Published
  • Downloads 10
  • Score
    100M100P100Q32830F
  • License MIT

LLM API gateway with intelligent routing, robust process management, and health monitoring

Package Exports

  • @tehreet/conduit
  • @tehreet/conduit/plugins
  • @tehreet/conduit/router
  • @tehreet/conduit/server

Readme

Conduit Logo

Conduit

Intelligent LLM API Gateway & Node.js Library

npm version License: MIT Node.js Version CI/CD

A smart proxy layer with advanced routing, monitoring, and programmatic API


πŸš€ Quick Start

CLI Usage

# Install globally
npm install -g @tehreet/conduit

# Start server
conduit start

# Make requests
curl -X POST http://localhost:8111/v1/messages \
  -H "Content-Type: application/json" \
  -d '{"model": "claude-3-5-sonnet-20241022", "messages": [{"role": "user", "content": "Hello!"}]}'

Library Usage

import { ConduitServer, ConduitRouter } from '@tehreet/conduit';

// Programmatic server
const server = new ConduitServer(config);
await server.start();

// Standalone routing
const router = new ConduitRouter(config);
const decision = await router.route(context);

✨ Features

🧠 Intelligent Routing

  • Content Analysis: Automatic complexity detection, language identification, topic extraction
  • Priority-Based Rules: Custom conditions with confidence scoring
  • Context Awareness: Synapse IDE integration with project/agent context
  • Token-Based Selection: Optimal model selection based on request size

πŸ”§ Dual Architecture

  • CLI Tool: Traditional command-line interface with daemon mode
  • Node.js Library: 10x faster programmatic API with full TypeScript support
  • 100% Backward Compatibility: Existing integrations continue to work

πŸ“Š Monitoring & Analytics

  • Usage Tracking: Request, token, and cost analytics with project/agent breakdown
  • Health Monitoring: Real-time health checks for server, plugins, and storage
  • Metrics Collection: Integration with Prometheus, StatsD, and custom endpoints
  • Performance Caching: Intelligent token counting cache and batch processing

πŸ”Œ Extensible Architecture

  • Plugin System: Custom routing logic with lifecycle hooks
  • Custom Evaluators: Extensible condition evaluation for routing rules
  • Storage Abstraction: Memory, file, and database storage options
  • Event-Driven: EventEmitter-based architecture for real-time updates

πŸ“¦ Installation

Global CLI

npm install -g @tehreet/conduit

Library Dependency

npm install @tehreet/conduit

Development

git clone https://github.com/vibe-coders-only/conduit.git
cd conduit
npm install
npm run build

πŸ—οΈ Architecture

Conduit provides both CLI and library interfaces:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚   CLI Interface β”‚    β”‚ Library Interfaceβ”‚
β”‚   (Traditional) β”‚    β”‚  (Programmatic) β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜
          β”‚                      β”‚
          β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                     β”‚
        β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
        β”‚    ConduitServer       β”‚
        β”‚   (Event-Driven)       β”‚
        β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                  β”‚
        β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
        β”‚    ConduitRouter       β”‚
        β”‚  (Advanced Routing)     β”‚
        β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                  β”‚
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚             β”‚             β”‚
β”Œβ”€β”€β”€β–Όβ”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β–Όβ”€β”€β”€β”€β”
β”‚Pluginsβ”‚  β”‚ Providers β”‚  β”‚Monitor β”‚
β”‚System β”‚  β”‚Integrationβ”‚  β”‚& Track β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸ”§ API Reference

Core Classes

ConduitServer

import { ConduitServer, ConduitConfig } from '@tehreet/conduit';

const config: ConduitConfig = {
  server: { port: 8111 },
  Router: { default: 'claude-3-5-sonnet-20241022' },
  Providers: [/* ... */]
};

const server = new ConduitServer(config);

// Lifecycle management
await server.start();
await server.stop();

// Request routing
const decision = await server.route(context);

// Configuration updates
await server.updateConfig(newConfig);

// Health status
const health = await server.getHealth();

ConduitRouter

import { ConduitRouter, RoutingContext } from '@tehreet/conduit';

const router = new ConduitRouter(config);
await router.initialize();

const context: RoutingContext = {
  request: { body: { messages: [...] } },
  tokenCount: 1500,
  config,
  env: process.env,
  synapseContext: { projectId: 'proj-123' }
};

const decision = await router.route(context);
// { selectedModel: 'claude-3-5-haiku-20241022', confidence: 0.85, reason: 'Simple request' }

Advanced Routing

Custom Routing Rules

import { registerRoutingRule, AdvancedRoutingRule } from '@tehreet/conduit';

const customRule: AdvancedRoutingRule = {
  id: 'coding-requests',
  name: 'Coding Requests',
  priority: 90,
  conditions: [
    { type: 'content', field: 'codeBlocks', operator: '>', value: 0 },
    { type: 'token_count', operator: '<', value: 50000 }
  ],
  model: 'claude-3-5-sonnet-20241022',
  confidence: 0.9,
  enabled: true
};

registerRoutingRule(customRule);

Custom Evaluators

import { registerCustomEvaluator } from '@tehreet/conduit';

registerCustomEvaluator('is_urgent', (condition, context) => {
  return context.request.headers['x-priority'] === 'urgent';
});

// Use in routing rules
const urgentRule = {
  id: 'urgent-requests',
  conditions: [{ type: 'custom', field: 'is_urgent', operator: '=', value: true }],
  model: 'claude-3-5-sonnet-20241022',
  priority: 100
};

Monitoring & Analytics

Usage Tracking

import { UsageTracker } from '@tehreet/conduit';

const tracker = new UsageTracker({
  enabled: true,
  storage: { type: 'file', path: './usage.json' },
  metrics: { enabled: true, provider: 'prometheus' }
});

await tracker.initialize();

// Track usage
await tracker.trackUsage({
  id: 'req-123',
  timestamp: new Date(),
  model: 'claude-3-5-sonnet-20241022',
  tokenCount: 1500,
  cost: 0.075,
  routingReason: 'Complex request'
});

// Get statistics
const stats = await tracker.getUsageStats({
  timeRange: { start: new Date('2024-01-01'), end: new Date() }
});

Health Monitoring

import { HealthMonitor } from '@tehreet/conduit';

const monitor = new HealthMonitor();

// Custom health check
monitor.registerCheck('database', async () => {
  const isHealthy = await checkDatabase();
  return {
    healthy: isHealthy,
    message: isHealthy ? 'Database connected' : 'Database unavailable',
    timestamp: new Date()
  };
});

monitor.start(30000); // Check every 30 seconds
const status = await monitor.getStatus();

Token Counter with Caching

Enhanced Token Counting

import { tokenCounter } from '@tehreet/conduit';

// Single request
const result = await tokenCounter.countTokens('Hello world', 'claude-3-5-sonnet-20241022');
// { count: 2, method: 'tiktoken' }

// Batch processing
const requests = [
  { id: 'req1', text: 'Hello', model: 'claude-3-5-sonnet-20241022' },
  { id: 'req2', text: 'How are you?', model: 'claude-3-5-sonnet-20241022' }
];

const results = await tokenCounter.batchCountTokens(requests);
// [{ id: 'req1', count: 1, method: 'tiktoken' }, ...]

// Cache statistics
const stats = tokenCounter.getCacheStats();
// { size: 150, hitRate: 0.85, oldestEntry: Date, newestEntry: Date }

Synapse Integration

Context-Aware Routing

import { SynapseIntegration } from '@tehreet/conduit';

const integration = new SynapseIntegration(config);
await integration.initialize();

// Route with Synapse context
const result = await integration.routeClaudeRequest(
  ['--model', 'claude-3-5-sonnet-20241022', 'Hello'],
  {
    SYNAPSE_PROJECT_ID: 'proj-123',
    SYNAPSE_AGENT_ID: 'agent-456',
    SYNAPSE_PROJECT_MODEL_CONFIG: JSON.stringify({
      enabled: true,
      defaultModel: 'anthropic/claude-3-5-haiku-20241022'
    })
  }
);

βš™οΈ Configuration

Basic Configuration

{
  "server": {
    "port": 8111,
    "host": "127.0.0.1"
  },
  "Router": {
    "enabled": true,
    "default": "claude-3-5-sonnet-20241022",
    "longContext": "claude-3-5-sonnet-20241022",
    "background": "claude-3-5-haiku-20241022",
    "think": "claude-3-5-sonnet-20241022"
  },
  "Providers": [
    {
      "name": "anthropic",
      "api_base_url": "https://api.anthropic.com",
      "api_key": "${ANTHROPIC_API_KEY}",
      "models": ["claude-3-5-sonnet-20241022", "claude-3-5-haiku-20241022"]
    }
  ],
  "monitoring": {
    "enabled": true,
    "usage": {
      "enabled": true,
      "storage": "memory",
      "retention": 30
    },
    "health": {
      "enabled": true,
      "interval": 30
    },
    "metrics": {
      "enabled": false,
      "provider": "custom"
    }
  }
}

Environment Variables

# Server
CONDUIT_PORT=8111
CONDUIT_HOST=127.0.0.1

# API Keys
ANTHROPIC_API_KEY=your-key-here
OPENAI_API_KEY=your-key-here

# Monitoring
CONDUIT_MONITORING_ENABLED=true
CONDUIT_METRICS_PROVIDER=prometheus

# Synapse Integration
SYNAPSE_PROJECT_ID=proj-123
SYNAPSE_AGENT_ID=agent-456

πŸ§ͺ Testing

Run the comprehensive test suite:

# Build and test
npm run build
npm test

# Integration tests
node test-phase3.js  # Synapse integration
node test-phase5.js  # Enhanced features

# Performance tests
./test-wrapper-features.sh

πŸš€ Performance

Benchmarks

  • 10x faster than CLI spawning for library usage
  • 50% lower memory usage with intelligent caching
  • Batch processing optimizes multiple token count requests
  • Smart routing ensures optimal model selection

Caching Benefits

  • Token counting cache reduces redundant calculations
  • Content analysis cache for repeated requests
  • Model configuration cache for faster routing
  • Automatic cache cleanup and size management

πŸ“ˆ Production Deployment

PM2 Configuration

// ecosystem.config.js
module.exports = {
  apps: [{
    name: 'conduit',
    script: 'dist/cli.js',
    args: 'start --daemon',
    instances: 1,
    exec_mode: 'fork',
    env: {
      NODE_ENV: 'production',
      CONDUIT_PORT: 8111
    }
  }]
};

Docker

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm run build
EXPOSE 8111
CMD ["npm", "start"]

Health Checks

# Basic health
curl http://localhost:8111/health

# Detailed status
curl http://localhost:8111/alive

# Readiness probe
curl http://localhost:8111/ready

🀝 Contributing

  1. Fork the repository
  2. Create a feature branch: git checkout -b feature/amazing-feature
  3. Commit changes: git commit -m 'feat: add amazing feature'
  4. Push to branch: git push origin feature/amazing-feature
  5. Open a Pull Request

Development Setup

git clone https://github.com/vibe-coders-only/conduit.git
cd conduit
npm install
npm run build
npm run dev

πŸ“„ License

MIT License - see LICENSE for details.


Transform your LLM integration with intelligent routing and monitoring