LOON — LLM-Optimized Object Notation

A compact wire format for data pipelines and LLM prompts.

LOON rewrites structured data (JSON / CSV / XML / YAML / trees) into a dense, fully reversible encoding that shrinks token counts without losing a single byte on the way back. Most formats lock you into one trade-off; LOON instead exposes three distinct modes — one built for service-to-service transport, one written to be read straight by a model, and one sized for tiny or uneven objects — so a job never carries compression overhead it has no use for.

The mental model is a boundary codec: hold JSON in your application, and switch to LOON only at the edge where every token shows up on the bill.

[!TIP] Across real retrieval runs (Gemini 3 Flash, o3-mini), LOON llm is the most token-efficient format at 100% accuracy — fewer tokens than TOON, JTON, JSON, JSON-compact, YAML and XML — and round-trips every benchmark dataset losslessly. See Benchmarks.

Table of Contents

• Why LOON?
• Key Features
• The Three Modes
• When Not to Use LOON
• Benchmarks
• Quick Start
• CLI
• Wire Format
• getSpec() — when full must talk to an LLM
• Sessions & Context Caching
• API
• Architecture
• License

Why LOON?

Context windows keep getting bigger, yet every token is still metered — and JSON spends them freely. Each brace, each quote, each field name repeated row after row lands on the invoice for every call. A uniform array of records is charged for its column labels once per element:

[
  { "id": 1, "name": "Alice", "dept": "Engineering", "salary": 120000, "active": true },
  { "id": 2, "name": "Bob",   "dept": "Sales",       "salary": 98000,  "active": false }
]

LOON llm mode states the schema a single time and then emits bare rows — no braces, no echoed keys, no quotes around values that aren't ambiguous — and a model can follow it cold, with no primer attached:

@T1[2]{id,name,dept,salary,active}
1,Alice,Engineering,120000,true
2,Bob,Sales,98000,false

When the target is storage or a service-to-service hop, full mode pushes harder (Base36 integers, sequences, dictionaries) and the decoder rebuilds the original exactly, byte for byte.

Key Features

• Measured, Not Promised: in one fully billed retrieval run, LOON llm dropped total tokens 64% against JSON and 47% against TOON while answering at 100% accuracy (see Benchmarks).
• One Mode Per Job: full covers transport and storage, llm is written for model reading, compact handles small or uneven data — nothing tries to be everything at once.
• Reversible By Construction: the decoder is deterministic and reproduces the source JSON exactly; every mode round-trips the benchmark datasets 10/10.
• Zero Setup For Models (llm): decimal numbers as-is, literal true/false/null, schema stated once. The model simply reads it — there is nothing for it to unpack.
• Format Bridges Built In: a single API moves between JSON, CSV, XML, YAML, and nested trees in both directions.
• Designed For Prompt Caches: LoonSession peels a reusable schema/spec prefix off the per-call rows, so repeat queries are billed mostly as cache hits.
• Ships With A CLI: npx loon-core data.json converts, round-trips, and reports token savings under --stats.

The Three Modes

Plus:

• compat — JSON-hybrid ({S,T,R} arrays). A 100%-valid-JSON escape hatch for environments that must parse with JSON.parse.
• local — deprecated alias of llm (kept for back-compat).

Design rule. full is allowed every compression trick because its reader is a decoder. llm rejects every trick that would force the model to compute (Base36, sequences, dictionaries, suffix reattachment) and keeps only the tricks that de-duplicate structure — schema declared once, object arrays tabulated, nested objects grouped. The dividing line is: does this require execution to read?

When Not to Use LOON

• Deep, irregular object trees: with no uniform arrays to fold into a table, compact mode still helps but the margin narrows — minified JSON can hold its own here.
• Flat tables read by code, not a model: plain CSV edges it out on size. LOON's extra structure (schema, length markers, type guards) earns its keep with model reliability, not with a CSV reader.
• You must call JSON.parse directly on the wire: reach for compat mode (which stays valid JSON) and take the smaller token cut that comes with it.
• Latency-bound local inference: benchmark before you commit. Fewer tokens doesn't always translate to faster decode on quantized or on-device models.

Benchmarks

Three experiments: retrieval accuracy + token efficiency (real API calls), multi-tokenizer consistency, and round-trip fidelity. Datasets: synthetic (@faker-js/faker, seed 12345 — tabular, nested, analytics, event-logs, nested-config) plus a real one (top-100 GitHub repositories). Token efficiency = (accuracy% ÷ tokens) × 1000 (higher is better).

Retrieval accuracy & token efficiency

Each dataset is serialized in every format, embedded in an identical prompt, and queried with the same deterministic questions; answers are validated against a type-aware ground truth. Question sets are reduced (10–12 Q) to bound API cost.

gemini-3-flash-preview — 10 deterministic questions

o3-mini — 12 deterministic questions (DRY_RUN + FAST_FORMAT)

[!TIP] LOON llm is the most token-efficient format at 100% accuracy — −4% tokens vs TOON on Gemini, −19% vs TOON on o3-mini. full minimizes input but its Base36 / sequence / dictionary cells force the model to decode mentally → more output tokens and an accuracy drop (91.7%). Use llm for LLM consumption, full for storage / transport.

Multi-tokenizer consistency

Percent token-count difference vs the o200k_base (GPT-4o / GPT-5) baseline on the GitHub dataset. Lower = the format compresses by a similar amount on that tokenizer, so a savings claim transfers across model families.

Round-trip fidelity

Encode → decode with each format's standard parser → compare to the original. The comparator is type-coercion-aware (123 == "123"); structural diffs — missing keys, dropped nesting, length changes — count as loss.

All four LOON modes round-trip every dataset losslessly — matching JSON / TOON, ahead of XML (lossy on nested), CSV (no standard parser), and JTON (lossy on all).

[!NOTE] The retrieval runs use reduced deterministic question sets (10–12 Q) to bound API cost. At this size every lossless format reaches 100% accuracy, so these results establish token efficiency at parity accuracy — not an accuracy ranking. Larger question sets and weaker models are needed to separate formats on comprehension.

Quick Start

npm install loon-core

import { Loon } from 'loon-core';

const loon = new Loon();

const data = [
  { id: 1, name: 'Alice', dept: 'Engineering', salary: 120000, active: true },
  { id: 2, name: 'Bob',   dept: 'Sales',       salary: 98000,  active: false },
];

// LLM consumption (default for prompts)
loon.toLOON(data, { mode: 'llm' });

// Bulk transmission / storage (not for raw LLM reading)
loon.toLOON(data, { mode: 'full' });

// Small / irregular data
loon.toLOON(data, { mode: 'compact' });

// Decode (auto-detects the mode)
loon.fromLOON(encoded);

If mode is omitted it is auto-selected: compact for empty or non-uniform input, micro for 1–4 rows, full for ≥ 5 uniform rows. Override with { mode: 'llm' } when the target is a model.

CLI

No installation required — use it instantly with npx:

npx loon-core data.json

Or install globally and use the loon command:

npm install -g loon-core

Options

Token statistics

Pass --stats to see a token estimate before and after encoding. Uses a chars/4 heuristic — fast, no API key required.

✔ data.json → output.loon
ℹ Token estimate: ~15,145 (json) → ~8,745 (loon)
✔ Saved ~6,400 tokens (-42%) [strong]

Output is colour-coded in TTY terminals and plain text when piped.

Examples

# JSON → LOON (auto mode)
loon data.json

# JSON → LOON llm mode + token stats
loon data.json -m llm --stats

# JSON → LOON full compression, write to file
loon data.json -m full -o output.loon

# CSV → LOON
loon data.csv -f csv -m llm

# LOON → JSON
loon data.loon -t json

# Pipe from stdin
echo '[{"id":1,"name":"Ada","role":"dev"}]' | loon

# Round-trip: JSON → LOON → JSON
cat data.json | loon | loon -f loon -t json

Wire Format

full mode — maximum compression

The format that lives in a .loon file or rides between two services.

S:@T1[N]=[col:type,...]      schema: row count + columns with type codes
A:fullName,...               column aliases (only if names were abbreviated)
DC:col,...                   integer columns stored decimal, not Base36
C:col=value                  constant column (omitted from every row)
Q:col=start,step             integer arithmetic sequence
QF:col=start,step            float arithmetic sequence
QS:col=start,step,prefix     string sequence (prefix + counter)
FP:d=col,...                 fixed-point: row token ÷ 10^d = value
X:col=suffix                 common suffix stripped, re-appended on decode
D:col={tok:val,...}          semantic dictionary (token → value)
D:__global__={tok:prefix}    shared prefix dictionary (backs `$tok` cells)
D:defaults=col=val,...       per-column default; `~` in a row means "use it"
DL:col=firstValue            delta encoding (row tokens are signed deltas)
NM:col=mean,std,sigmaT,mT    z-score normalization (LOSSY)
LY:NM                        marks payload contains lossy NORM columns
AS:col=k1,k2,...             uniform object-array sub-schema (see below)
@T1:                         start of the data block
F:csv                        rows are comma-delimited
<data rows>

full is not meant to be read by an LLM directly. If you must, prepend getSpec(encoded) to the prompt — and even then expect output-token costs to be higher than llm mode, because the model has to mentally decode every Base36 / sequence / dictionary cell.

llm mode — self-evident, LLM-readable

One header line. Plain decimal rows. JSON-style literals.

@T1[N]{id,name,email,dept,salary,active}
C:status=active                       (optional, when a constant column exists)
AS:items=sku,name,qty                 (optional, for object-array columns)
1,Alice,alice@x.com,Engineering,120000,true
2,Bob,bob@x.com,Sales,98000,false
3,Carol,carol@x.com,Marketing,null,true

Rules the model can apply at sight:

• Numbers bare: 120000. Decoded as Number(token).
• Booleans literal: true / false.
• Null literal: null (same single BPE token JSON uses).
• Strings bare when unambiguous: Alice.
• Strings that look like numbers / bools / null are quote-wrapped: "123", "true", "null". The decoder strips the quotes and keeps the string as a string.
• Absent key in this row (sparse / non-uniform): +.
• Inline object (col:o): the cell is RAW JSON ({"a":1,"b":2}) — read as-is. Commas inside it are data, not separators (the decoder consumes the balanced {…} as one field), so the model reads native JSON with zero escape noise.
• #ex row1 → … (when present): row 1 already decoded, column by column. A #-prefixed line is not data — the decoder skips it. It is emitted only for non-trivial schemas (wide, or with extracted constants / :o / :a columns) to anchor the positional mapping on the first read, cutting the reasoning a model would otherwise spend aligning values to columns.

No :type codes in the header — the model infers types from cell shape. That alone saves ~2 tokens per column (the :a array and :o inline-object tags are the two exceptions — type inference cannot recover a pipe-joined array or distinguish a JSON-object cell from a string). No DC: / @T1: / F:csv scaffolding either; the @…{…} line is its own start marker.

Deep, heterogeneous arrays — subtree folding

Flattening every nested object to dot-notation explodes a heterogeneous array (e.g. GitHub events: PushEvent, ForkEvent, …) into a giant column union where most columns are absent (+) on most rows. That bloats the header and forces the model to count comma positions across +,+,+,… runs — pure reasoning-token burn. LOON folds a nested subtree back to a single inline :o JSON column when it is deep + sparse (present in < 50 % of rows) or when its key set diverges per record (the union of child keys is ≥ 2× the average per record — payload.* is shaped differently for every event type even though every row has one). Dense, uniformly-shaped structure still expands to dot-notation columns, where schema-once dedup pays off. On github.json (30 events) this cut the schema from 100+ columns to 18, absent-cell density from 32 % to 11 %, and the payload now reads as one native JSON cell per row — lossless round-trip, and fewer input tokens than TOON (−13 %).

compact mode — small or irregular data

id: 1
name: Alice
tags[3]: a,b,c                    scalar array, length 3
items[2]{sku,qty}: A,1;B,2        uniform object array
---
id: 2
...

Single deep objects (configs) use an indented hierarchy instead of repeated dot-notation keys.

Row tokens (shared)

AS: — uniform object-array sub-schema

A column holding an array of same-shape objects (items: [{sku,name,qty}, …]) would otherwise be inline JSON with the keys repeated on every element. AS: declares the shared shape once; each cell carries values only:

AS:items=sku,name,qty
cell:  A|Mouse|1;B|Cable|2   →   [{sku:A,name:Mouse,qty:1},{sku:B,name:Cable,qty:2}]

Fields within an object are |-separated; objects are ;-separated.

Type codes (full mode)

i integer · f float · s string · b boolean · a array · o object

llm mode omits the codes — types are inferred from cell shape.

getSpec() — when full must talk to an LLM

full is built for parsers. If you need an LLM to read a full payload (for example: you stored data in .loon, you now want a model to query it), getSpec(encoded) returns a minimal decode spec (200–600 tokens) covering only the headers that this specific payload actually uses, plus a worked walkthrough of row 0.

import { getSpec } from 'loon-core';

const encoded = loon.toLOON(data, { mode: 'full' });
const spec = getSpec(encoded);   // { text, sections, estimatedTokens }
// prepend spec.text to the prompt; the model can now decode `full`.

llm mode does not need a spec — that is the whole point of it.

Sessions & Context Caching

For repeated calls against the same data shape, LoonSession separates the cacheable prompt prefix from the per-call data block. LLM providers cache an identical prefix and bill it at a fraction of the normal rate; put the spec and the schema there and the per-call cost shrinks to just the rows.

import { LoonSession } from 'loon-core';

const s = new LoonSession();
s.init(firstBatch, { mode: 'full' });

// System prompt (mark it for prompt caching): s.primer   ← getSpec() + schema
// User message 1:                             s.dataBlock
for (const batch of moreBatches) {
  send(s.encodeRows(batch).dataBlock);          // only the rows
}

splitLoon(encoded) exposes the raw { schema, dataBlock } split for custom integrations.

Stable header + row batches (llm)

For the llm caching pattern, encode the header once and stream row blocks. toLOONHeader derives a stable schema — every column present in every row, declared order, no per-batch constants / legends / sub-schemas — so any later batch's rows decode against the one cached header:

const { header, columns } = loon.toLOONHeader(firstBatch);   // cache this
// system prompt: header  (llm is self-describing — no getSpec needed)
for (const batch of batches) {
  send(loon.toLOONRows(batch, columns));                     // rows only
}

Why stable matters. Encoding each batch independently lets its layout drift: a column that is constant within batch 1 (dept=Eng) is hoisted to a C: header and dropped from its rows; batch 2, where dept varies, keeps it in the rows — so a rows-only block from batch 2 silently corrupts when decoded against batch 1's header. toLOONHeader / toLOONRows (and LoonSession in llm mode) disable that drift. LoonSession.primer is just the schema for llm — bolting a ~300-token getSpec() onto a small readable payload costs more than it saves.

Prompt caching reduces input cost. It does not touch output tokens. A model that has to reason hard about a format still pays full price on output. That is why llm mode beats full + cached spec for direct LLM consumption: llm's output token cost is small because the format requires no reasoning to read.

Output tokens — answer in LOON

Output tokens cost ~4–5× input tokens, and most of an agent's bill is the model restating data as JSON ({ "id": 53, "name": ... }) plus wrappers ({ answer, confidence, metadata }). Have the model answer in LOON instead. Three accuracy-neutral levers:

1. Output contract — a short, cacheable prompt snippet telling the model how to answer by query type (single value / one field / row subset). This is ~80 % of the win, and it is prompt design, not a new format.

import { buildOutputContract } from 'loon-core';
const contract = buildOutputContract(header);   // put in the system prompt

ANSWER FORMAT — reply in LOON, never JSON:
• Single value:  @ans  /  <value>
• One field of one record:  <col>=<value>   (or 53.email=ana@x.com by id)
• Which / how many match:  @ids 3,7,12      (key column only — not the records)
• Multiple records in full: LOON rows, one per line, same columns as the header

A field lookup drops from {"id":53,...,"email":"ana@x.com",...} (25 tok) to 53.email=ana@x.com (7 tok). A "which records match" answer drops from a JSON array of full records to @ids 3,7,12 — the single biggest output win, since filter/count is the most common query. Use { mode } to emit only the rule for the query type you expect, and { keyCol } to name the id column for @ids.

2. Parse + validate the answer — checkAnswer turns the model's reply back into values and, for the rows shape, validates every row against the header, returning a tiny repair hint (50–150 tok) on a malformed row instead of a silent wrong result. This is what makes output-LOON safe: the arity check runs on the raw text (quote-/escape-aware), so a dropped or extra column is caught even though a positional decoder would fill it silently.

const r = loon.checkAnswer(header, modelReply);
// { kind:'scalar'|'field'|'ids'|'rows', value? , ids? , rows? , ok, hint }
if (r.kind === 'rows' && !r.ok) send(r.hint);   // cheap retry, not a re-ask

3. Constrained decoding (GBNF) — where the provider supports grammars (llama.cpp, vLLM, and some hosted tool paths), buildLoonGrammar(header) emits a grammar that pins output to exactly N comma-separated fields per row. This is the strongest guarantee available: parseable, correctly-aligned output with zero format drift — no validation round-trip needed.

import { buildLoonGrammar } from 'loon-core';
const gbnf = buildLoonGrammar(header);   // pass to the provider's grammar param

Prefill + stop — buildAnswerControls(kind, header?) returns a prefill to seed the assistant turn into the right shape (the model spends no tokens on framing and cannot drift to JSON) and stop sequences that end generation at the answer boundary (capping output). For rows it also returns the grammar.

import { buildAnswerControls } from 'loon-core';
const { prefill, stop, grammar } = buildAnswerControls('ids', header);
// → prefill "@ids ", stop ["\n", "```"]   — the model only emits the key list

Measure the win on your own data with npm run bench:output (token deltas per query shape + a deterministic round-trip check). Real-model accuracy is the next step: feed the contract to your model — cloud or local (GBNF closes the local-model format gap) — and re-run with its actual replies.

Output is not input. On input the model reads and the decoder reverses any compression, so aggressive encodings are safe. On output the model generates and nothing enforces the format, so anything that makes it compute the encoding (Base36, index→key maps, implicit positions, sequences) raises the error rate. The safe output subset is positional-but-named, literal values, no computed encodings — exactly what these helpers assume.

For object edits, toDelta(before, after, { key }) already emits a minimal patch (M53 active=false) — have the model reply in that shape and apply it with applyDelta, instead of re-emitting whole records.

API

Options (LoonOptions)

Architecture

Input (JSON / CSV / XML / YAML / tree)
        │
   Mode Selector ──────────────┐
        │                      │ (auto-pick when mode omitted)
   ┌────┴─────┬──────────┐     │
   ▼          ▼          ▼     ▼
 full       llm      compact  compat
   │          │          │     │
   └── Adaptive ─┘        │     │
   pipeline               │     │
        │                 │     │
        └─────────────────┴─────┘
                  │
            LOON output

full and llm share one analysis pipeline. The analyzer gates off every compute-requiring primitive (Base36, sequences, dictionaries, defaults, suffixes, fixed-point, delta, NORM, RLE, anchor rows) when the mode is llm. What survives is structural-only: the schema, C: constants, and AS: sub-schemas. The header emitter also drops :type codes and replaces ^ (null sentinel) with the literal null for llm mode.

License

MIT © LOON Thesis Team