LOON — LLM-Optimized Object Notation

Token-efficient serialization for data pipelines and LLM prompts.

LOON converts structured data (JSON / CSV / XML / YAML / trees) into a compact wire format that minimizes tokens while staying losslessly reversible. Where most formats pick one trade-off, LOON ships three modes — one tuned for machine-to-machine transmission, one for direct LLM consumption, one for small or irregular objects — so you never pay for compression a given job doesn't need.

Think of it as a translation layer: keep JSON in your code, encode to LOON at the boundary where tokens cost money.

[!TIP] On a real 1,000-row retrieval session (Gemini 3 Flash), LOON llm used 64% fewer tokens than JSON and 47% fewer than TOON, at 60% lower cost than JSON. See Benchmarks.

Table of Contents

• Why LOON?
• Key Features
• The Three Modes
• When Not to Use LOON
• Benchmarks
• Quick Start
• CLI
• Wire Format
• getSpec() — when full must talk to an LLM
• Sessions & Context Caching
• API
• Architecture
• License

Why LOON?

LLM context windows keep growing, but tokens still cost money — and JSON is verbose. Every brace, quote, and repeated key is billed on every request. A uniform array of records pays for its field names once per row:

[
  { "id": 1, "name": "Alice", "dept": "Engineering", "salary": 120000, "active": true },
  { "id": 2, "name": "Bob",   "dept": "Sales",       "salary": 98000,  "active": false }
]

LOON llm mode declares the schema once and streams plain rows — no braces, no repeated keys, no quoting of unambiguous values, while staying readable to a model with zero primer:

@T1[2]{id,name,dept,salary,active}
1,Alice,Engineering,120000,true
2,Bob,Sales,98000,false

For storage or system-to-system transfer, full mode goes further (Base36 integers, sequences, dictionaries) and is restored bit-for-bit by the decoder.

Key Features

• Token-Efficient & Verified: On a real billed retrieval session, LOON llm cut total tokens 64% vs JSON and 47% vs TOON at 100% accuracy (see Benchmarks).
• Three Modes, One Job Each: full for transmission/storage, llm for direct model reading, compact for small/irregular data — no compromise format.
• Lossless Round-Trips: A deterministic decoder restores the original JSON exactly; all modes recover the benchmark datasets 10/10.
• No Primer Needed (llm): Plain decimal numbers, literal true/false/null, schema declared once. The model reads it; it never has to decode it.
• Multi-Format Bridges: Encode from and decode to JSON, CSV, XML, YAML, and nested trees through one API.
• Prompt-Cache Friendly: LoonSession splits a cacheable schema/spec prefix from per-call rows, so repeated queries bill mostly cache reads.
• CLI Included: npx loon-core data.json — convert, round-trip, and see token savings with --stats.

The Three Modes

Plus:

• compat — JSON-hybrid ({S,T,R} arrays). A 100%-valid-JSON escape hatch for environments that must parse with JSON.parse.
• local — deprecated alias of llm (kept for back-compat).

Design rule. full is allowed every compression trick because its reader is a decoder. llm rejects every trick that would force the model to compute (Base36, sequences, dictionaries, suffix reattachment) and keeps only the tricks that de-duplicate structure — schema declared once, object arrays tabulated, nested objects grouped. The dividing line is: does this require execution to read?

When Not to Use LOON

• Deeply nested, non-uniform objects: with no uniform arrays to tabulate, compact mode helps but the win shrinks — compact JSON can be competitive.
• Pure flat tables consumed by code, not an LLM: CSV is marginally smaller. LOON adds structure (schema, length, type guards) that pays off for model reliability, not for a CSV parser.
• You need raw JSON.parse on the wire: use compat mode (valid JSON) and accept a smaller token saving.
• Latency-critical local models: measure first. Token count is not always the bottleneck for quantized/local inference.

Benchmarks

Real billed session — LOON vs TOON vs JSON (Gemini 3 Flash)

Same dataset (fakestore: 1,000 products × id, price, category, rating_rate, rating_count), same ~10 data-retrieval questions, same model (gemini-3-flash-preview), prompt caching on. Each format was fed to the model and billed by the provider. Raw exports: token-usage-31843f03.json (LOON), token-usage-4eedd7fe.json (TOON), token-usage-5ed44cd3.json (JSON).

1,000 products × ~10 retrieval questions · Gemini 3 Flash · prompt-cached
(total billed tokens — lower is better)

   LOON llm   ███████░░░░░░░░░░░░░   272,048 tokens   │  $0.047429   ← best
   TOON       █████████████░░░░░░░   509,094 tokens   │  $0.081574   (+87% tokens)
   JSON       ████████████████████   761,458 tokens   │  $0.118791   (+180% tokens)

[!TIP] LOON llm vs JSON: −64.3% total tokens, −60.1% cost. LOON llm vs TOON: −46.6% total tokens, −41.9% cost. Normalized per call (LOON ran one extra question): $0.0043/call vs TOON $0.0082 and JSON $0.0119 — LOON is ~47% cheaper per call than TOON.

Honest read of the numbers. LOON llm carries the highest output and reasoning tokens of the three (14.7k out, 10.8k reasoning) and ran one extra call — yet still lands lowest on total tokens and cost, because its input is dramatically smaller. The whole game here is input: a 1,000-row payload is re-sent each turn, so a compact schema-once layout compounds across the session.

Payload tokens (vs json-compact, GPT-4o)

Static encoding size across structural patterns:

full is the storage / transmission winner everywhere. llm beats TOON on every dataset and remains LLM-readable without a spec.

full needs a spec; llm doesn't

Single-question retrieval, same payload, billed tokens (Gemini 3 Flash):

full minimizes input by deferring work to the model — output cost then blows up because it must mentally decode every Base36 / sequence / dictionary cell. Fix it with getSpec(), or simply use llm mode when the consumer is an LLM.

Round-trip fidelity

All four LOON modes recover all ten benchmark datasets losslessly (10 / 10). Full reports in ../comprehension-benchmark/benchmarks/results/.

Quick Start

npm install loon-core

import { Loon } from 'loon-core';

const loon = new Loon();

const data = [
  { id: 1, name: 'Alice', dept: 'Engineering', salary: 120000, active: true },
  { id: 2, name: 'Bob',   dept: 'Sales',       salary: 98000,  active: false },
];

// LLM consumption (default for prompts)
loon.toLOON(data, { mode: 'llm' });

// Bulk transmission / storage (not for raw LLM reading)
loon.toLOON(data, { mode: 'full' });

// Small / irregular data
loon.toLOON(data, { mode: 'compact' });

// Decode (auto-detects the mode)
loon.fromLOON(encoded);

If mode is omitted it is auto-selected: compact for empty or non-uniform input, micro for 1–4 rows, full for ≥ 5 uniform rows. Override with { mode: 'llm' } when the target is a model.

CLI

No installation required — use it instantly with npx:

npx loon-core data.json

Or install globally and use the loon command:

npm install -g loon-core

Options

Token statistics

Pass --stats to see a token estimate before and after encoding. Uses a chars/4 heuristic — fast, no API key required.

✔ data.json → output.loon
ℹ Token estimate: ~15,145 (json) → ~8,745 (loon)
✔ Saved ~6,400 tokens (-42%) [strong]

Output is colour-coded in TTY terminals and plain text when piped.

Examples

# JSON → LOON (auto mode)
loon data.json

# JSON → LOON llm mode + token stats
loon data.json -m llm --stats

# JSON → LOON full compression, write to file
loon data.json -m full -o output.loon

# CSV → LOON
loon data.csv -f csv -m llm

# LOON → JSON
loon data.loon -t json

# Pipe from stdin
echo '[{"id":1,"name":"Ada","role":"dev"}]' | loon

# Round-trip: JSON → LOON → JSON
cat data.json | loon | loon -f loon -t json

Wire Format

full mode — maximum compression

The format that lives in a .loon file or rides between two services.

S:@T1[N]=[col:type,...]      schema: row count + columns with type codes
A:fullName,...               column aliases (only if names were abbreviated)
DC:col,...                   integer columns stored decimal, not Base36
C:col=value                  constant column (omitted from every row)
Q:col=start,step             integer arithmetic sequence
QF:col=start,step            float arithmetic sequence
QS:col=start,step,prefix     string sequence (prefix + counter)
FP:d=col,...                 fixed-point: row token ÷ 10^d = value
X:col=suffix                 common suffix stripped, re-appended on decode
D:col={tok:val,...}          semantic dictionary (token → value)
D:__global__={tok:prefix}    shared prefix dictionary (backs `$tok` cells)
D:defaults=col=val,...       per-column default; `~` in a row means "use it"
DL:col=firstValue            delta encoding (row tokens are signed deltas)
NM:col=mean,std,sigmaT,mT    z-score normalization (LOSSY)
LY:NM                        marks payload contains lossy NORM columns
AS:col=k1,k2,...             uniform object-array sub-schema (see below)
@T1:                         start of the data block
F:csv                        rows are comma-delimited
<data rows>

full is not meant to be read by an LLM directly. If you must, prepend getSpec(encoded) to the prompt — and even then expect output-token costs to be higher than llm mode, because the model has to mentally decode every Base36 / sequence / dictionary cell.

llm mode — self-evident, LLM-readable

One header line. Plain decimal rows. JSON-style literals.

@T1[N]{id,name,email,dept,salary,active}
C:status=active                       (optional, when a constant column exists)
AS:items=sku,name,qty                 (optional, for object-array columns)
1,Alice,alice@x.com,Engineering,120000,true
2,Bob,bob@x.com,Sales,98000,false
3,Carol,carol@x.com,Marketing,null,true

Rules the model can apply at sight:

• Numbers bare: 120000. Decoded as Number(token).
• Booleans literal: true / false.
• Null literal: null (same single BPE token JSON uses).
• Strings bare when unambiguous: Alice.
• Strings that look like numbers / bools / null are quote-wrapped: "123", "true", "null". The decoder strips the quotes and keeps the string as a string.
• Absent key in this row (sparse / non-uniform): +.

No :type codes in the header — the model infers types from cell shape. That alone saves ~2 tokens per column. No DC: / @T1: / F:csv scaffolding either; the @…{…} line is its own start marker.

compact mode — small or irregular data

id: 1
name: Alice
tags[3]: a,b,c                    scalar array, length 3
items[2]{sku,qty}: A,1;B,2        uniform object array
---
id: 2
...

Single deep objects (configs) use an indented hierarchy instead of repeated dot-notation keys.

Row tokens (shared)

AS: — uniform object-array sub-schema

A column holding an array of same-shape objects (items: [{sku,name,qty}, …]) would otherwise be inline JSON with the keys repeated on every element. AS: declares the shared shape once; each cell carries values only:

AS:items=sku,name,qty
cell:  A|Mouse|1;B|Cable|2   →   [{sku:A,name:Mouse,qty:1},{sku:B,name:Cable,qty:2}]

Fields within an object are |-separated; objects are ;-separated.

Type codes (full mode)

i integer · f float · s string · b boolean · a array · o object

llm mode omits the codes — types are inferred from cell shape.

getSpec() — when full must talk to an LLM

full is built for parsers. If you need an LLM to read a full payload (for example: you stored data in .loon, you now want a model to query it), getSpec(encoded) returns a minimal decode spec (200–600 tokens) covering only the headers that this specific payload actually uses, plus a worked walkthrough of row 0.

import { getSpec } from 'loon-core';

const encoded = loon.toLOON(data, { mode: 'full' });
const spec = getSpec(encoded);   // { text, sections, estimatedTokens }
// prepend spec.text to the prompt; the model can now decode `full`.

llm mode does not need a spec — that is the whole point of it.

Sessions & Context Caching

For repeated calls against the same data shape, LoonSession separates the cacheable prompt prefix from the per-call data block. LLM providers cache an identical prefix and bill it at a fraction of the normal rate; put the spec and the schema there and the per-call cost shrinks to just the rows.

import { LoonSession } from 'loon-core';

const s = new LoonSession();
s.init(firstBatch, { mode: 'full' });

// System prompt (mark it for prompt caching): s.primer   ← getSpec() + schema
// User message 1:                             s.dataBlock
for (const batch of moreBatches) {
  send(s.encodeRows(batch).dataBlock);          // only the rows
}

splitLoon(encoded) exposes the raw { schema, dataBlock } split for custom integrations.

Prompt caching reduces input cost. It does not touch output tokens. A model that has to reason hard about a format still pays full price on output. That is why llm mode beats full + cached spec for direct LLM consumption: llm's output token cost is small because the format requires no reasoning to read.

API

Options (LoonOptions)

Architecture

Input (JSON / CSV / XML / YAML / tree)
        │
   Mode Selector ──────────────┐
        │                      │ (auto-pick when mode omitted)
   ┌────┴─────┬──────────┐     │
   ▼          ▼          ▼     ▼
 full       llm      compact  compat
   │          │          │     │
   └── Adaptive ─┘        │     │
   pipeline               │     │
        │                 │     │
        └─────────────────┴─────┘
                  │
            LOON output

full and llm share one analysis pipeline. The analyzer gates off every compute-requiring primitive (Base36, sequences, dictionaries, defaults, suffixes, fixed-point, delta, NORM, RLE, anchor rows) when the mode is llm. What survives is structural-only: the schema, C: constants, and AS: sub-schemas. The header emitter also drops :type codes and replaces ^ (null sentinel) with the literal null for llm mode.

License

MIT © LOON Thesis Team