free-tier-router

A TypeScript library that routes LLM API requests across multiple free-tier providers with intelligent rate limit management and automatic failover.

Features

OpenAI-compatible API - Drop-in replacement for the OpenAI SDK
Automatic failover - Seamlessly switches providers when rate limits are hit
Multiple providers - Groq, Cerebras, OpenRouter, NVIDIA NIM
Smart routing strategies - Priority-based or least-used selection
Rate limit tracking - Token and request tracking per provider/model
Streaming support - Full support for streaming responses
Generic model aliases - Use best, fast, etc. for automatic model selection

Supported Providers

Provider	Free Tier Limits	Sign Up
Groq	30 req/min, 14,400 req/day	Get API Key
Cerebras	30 req/min, 14,400 req/day	Get API Key
OpenRouter	20 req/min, 50 req/day	Get API Key
NVIDIA NIM	40 req/min	Get API Key (requires phone verification)

Installation

npm install free-tier-router

Quick Start

import { createRouter } from "free-tier-router";

const router = createRouter({
  providers: [
    { type: "groq", apiKey: process.env.GROQ_API_KEY },
    { type: "cerebras", apiKey: process.env.CEREBRAS_API_KEY },
    { type: "openrouter", apiKey: process.env.OPENROUTER_API_KEY },
    { type: "nvidia-nim", apiKey: process.env.NVIDIA_NIM_API_KEY },
  ],
});

// OpenAI-compatible interface
const response = await router.chat.completions.create({
  model: "llama-3.3-70b",
  messages: [{ role: "user", content: "Hello!" }],
});

console.log(response.choices[0].message.content);

With Metadata

// Get metadata about which provider was used
const { response, metadata } = await router.createCompletion({
  model: "best", // Generic alias - picks best available model
  messages: [{ role: "user", content: "Hello!" }],
});

console.log(`Provider: ${metadata.provider}, Model: ${metadata.model}`);
console.log(`Latency: ${metadata.latencyMs}ms, Retries: ${metadata.retryCount}`);

Configuration

const router = createRouter({
  // Required: at least one provider
  providers: [
    {
      type: "groq",
      apiKey: "...",
      priority: 0, // Lower = higher priority (default: 0)
      enabled: true, // Default: true
    },
    {
      type: "cerebras",
      apiKey: "...",
      priority: 1,
    },
    {
      type: "openrouter",
      apiKey: "...",
      priority: 2,
    },
    {
      type: "nvidia-nim",
      apiKey: "...",
      priority: 3,
    },
  ],

  // Routing strategy: "priority" (default) or "least-used"
  strategy: "priority",

  // Request timeout in ms (default: 60000)
  timeoutMs: 60000,

  // Custom model aliases
  modelAliases: {
    "my-fast-model": "llama-3.1-8b",
  },

  // Retry configuration
  retry: {
    maxRetries: 3,
    initialBackoffMs: 1000,
    maxBackoffMs: 30000,
    backoffMultiplier: 2,
  },

  // State persistence: "memory" (default), "file", or "redis"
  stateStore: { type: "memory" },
});

Available Models

Model	Tier	Providers
`gpt-oss-120b`	4 (XL)	Groq, Cerebras, OpenRouter, NVIDIA NIM
`llama-3.3-70b`	3 (Large)	Groq, Cerebras, OpenRouter, NVIDIA NIM
`qwen-3-32b`	2 (Medium)	Groq, OpenRouter
`llama-3.1-8b`	1 (Small)	Groq, Cerebras, OpenRouter, NVIDIA NIM

Generic Aliases

Use these instead of specific model names for automatic routing:

Alias	Description
`best`	Best available model (any tier)
`best-xl`	Best XL model (tier 4, 100B+)
`best-large`	Best large model (tier 3, 36-100B)
`best-medium`	Best medium model (tier 2, 9-35B)
`best-small`	Best small model (tier 1, 1-8B)
`fast`	Alias for `best-small`

Streaming

const { stream, metadata } = await router.createCompletionStream({
  model: "llama-3.3-70b",
  messages: [{ role: "user", content: "Tell me a story" }],
});

console.log(`Using: ${metadata.provider}/${metadata.model}`);

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

API Reference

Router Methods

// OpenAI-compatible interface
router.chat.completions.create(params);

// With metadata
router.createCompletion(params); // Returns { response, metadata }
router.createCompletionStream(params); // Returns { stream, metadata }

// Utilities
router.listModels(); // List available models
router.isModelAvailable(model); // Check model availability
router.getQuotaStatus(); // Get rate limit status for all providers
router.clearAllCooldowns(); // Reset rate limit tracking
router.close(); // Clean up resources

Metadata Object

interface CompletionMetadata {
  provider: "groq" | "cerebras" | "openrouter" | "nvidia-nim";
  model: string;
  latencyMs: number;
  retryCount: number;
}

Routing Strategies

Priority Strategy (default)

Providers are tried in order of priority (lower number = higher priority). Use this when you have a preferred provider.

const router = createRouter({
  providers: [
    { type: "groq", apiKey: "...", priority: 0 }, // Tried first
    { type: "cerebras", apiKey: "...", priority: 1 }, // Fallback
  ],
  strategy: "priority",
});

Least-Used Strategy

Distributes requests across providers based on remaining quota. Use this to maximize throughput across all providers.

const router = createRouter({
  providers: [
    { type: "groq", apiKey: "..." },
    { type: "cerebras", apiKey: "..." },
    { type: "openrouter", apiKey: "..." },
  ],
  strategy: "least-used",
});

Development

# Install dependencies
npm install

# Run tests
npm test

# Build
npm run build

# Run playground chat
npm run playground:chat

License

MIT