LOON — LLM-Optimized Object Notation

Token-efficient serialization for data pipelines and LLM prompts.

LOON converts structured data (JSON / CSV / XML / YAML / trees) into a compact wire format. Two distinct problems, two distinct optimizations: moving large volumes of data between systems is one job; feeding data to an LLM is another; serializing a small or irregular object is a third. LOON exposes three modes, one per job.

The three modes

Plus:

• compat — JSON-hybrid ({S,T,R} arrays). A 100%-valid-JSON escape hatch for environments that must parse with JSON.parse.
• local — deprecated alias of llm (kept for back-compat).

What each mode optimizes

full — minimise tokens, full stop. Base36 integers, arithmetic sequences, semantic dictionaries, suffix stripping, fixed-point, delta, RLE, abbreviations. The output is opaque to a human or an LLM, but a deterministic decoder restores the original bit-for-bit. This is the format you write to a .loon file or push across a wire when both ends own the decoder.

llm — self-evidence. One header line, plain decimal numbers, literal strings, null / true / false as JSON does them. No Base36, no sequences, no dictionaries, no sentinels the model would have to learn. The LLM reads it, it never decodes it.

compact — get out of the way. A schema header would cost more than it saves on a 3-row dataset or a single deep config object. compact writes key: value blocks (or indented hierarchy) — cheap for the common small-data case where columnar formats lose.

Design rule. full is allowed every compression trick because its reader is a decoder. llm rejects every trick that would force the model to compute (Base36, sequences, dictionaries, suffix reattachment) and keeps only the tricks that de-duplicate structure — schema declared once, object arrays tabulated, nested objects grouped. The dividing line is "does this require execution to read?"

Quick start

npm install loon-core

import { Loon } from 'loon-core';

const loon = new Loon();

const data = [
  { id: 1, name: 'Alice', dept: 'Engineering', salary: 120000, active: true },
  { id: 2, name: 'Bob',   dept: 'Sales',       salary: 98000,  active: false },
];

// LLM consumption (default for prompts)
loon.toLOON(data, { mode: 'llm' });

// Bulk transmission / storage (not for raw LLM reading)
loon.toLOON(data, { mode: 'full' });

// Small / irregular data
loon.toLOON(data, { mode: 'compact' });

// Decode (auto-detects the mode)
loon.fromLOON(encoded);

If mode is omitted it is auto-selected: compact for empty or non-uniform input, micro for 1–4 rows, full for ≥ 5 uniform rows. Override with { mode: 'llm' } when the target is a model.

Wire format

full mode — maximum compression

The format that lives in a .loon file or rides between two services.

S:@T1[N]=[col:type,...]      schema: row count + columns with type codes
A:fullName,...               column aliases (only if names were abbreviated)
DC:col,...                   integer columns stored decimal, not Base36
C:col=value                  constant column (omitted from every row)
Q:col=start,step             integer arithmetic sequence
QF:col=start,step            float arithmetic sequence
QS:col=start,step,prefix     string sequence (prefix + counter)
FP:d=col,...                 fixed-point: row token ÷ 10^d = value
X:col=suffix                 common suffix stripped, re-appended on decode
D:col={tok:val,...}          semantic dictionary (token → value)
D:__global__={tok:prefix}    shared prefix dictionary (backs `$tok` cells)
D:defaults=col=val,...       per-column default; `~` in a row means "use it"
DL:col=firstValue            delta encoding (row tokens are signed deltas)
NM:col=mean,std,sigmaT,mT    z-score normalization (LOSSY)
LY:NM                        marks payload contains lossy NORM columns
AS:col=k1,k2,...             uniform object-array sub-schema (see below)
@T1:                         start of the data block
F:csv                        rows are comma-delimited
<data rows>

full is not meant to be read by an LLM directly. If you must, prepend getSpec(encoded) to the prompt — and even then expect output-token costs to be higher than llm mode, because the model has to mentally decode every Base36 / sequence / dictionary cell.

llm mode — self-evident, LLM-readable

One header line. Plain decimal rows. JSON-style literals.

@T1[N]{id,name,email,dept,salary,active}
C:status=active                       (optional, when a constant column exists)
AS:items=sku,name,qty                 (optional, for object-array columns)
1,Alice,alice@x.com,Engineering,120000,true
2,Bob,bob@x.com,Sales,98000,false
3,Carol,carol@x.com,Marketing,null,true

Rules the model can apply at sight:

• Numbers bare: 120000. Decoded as Number(token).
• Booleans literal: true / false.
• Null literal: null (same single BPE token JSON uses).
• Strings bare when unambiguous: Alice.
• Strings that look like numbers / bools / null are quote-wrapped: "123", "true", "null". The decoder strips the quotes and keeps the string as a string.
• Absent key in this row (sparse / non-uniform): +.

No :type codes in the header — the model infers types from cell shape. That alone saves ~2 tokens per column. No DC: / @T1: / F:csv scaffolding either; the @…{…} line is its own start marker.

compact mode — small or irregular data

id: 1
name: Alice
tags[3]: a,b,c                    scalar array, length 3
items[2]{sku,qty}: A,1;B,2        uniform object array
---
id: 2
...

Single deep objects (configs) use an indented hierarchy instead of repeated dot-notation keys.

Row tokens (shared)

AS: — uniform object-array sub-schema

A column holding an array of same-shape objects (items: [{sku,name,qty}, …]) would otherwise be inline JSON with the keys repeated on every element. AS: declares the shared shape once; each cell carries values only:

AS:items=sku,name,qty
cell:  A|Mouse|1;B|Cable|2   →   [{sku:A,name:Mouse,qty:1},{sku:B,name:Cable,qty:2}]

Fields within an object are |-separated; objects are ;-separated.

Type codes (full mode)

i integer · f float · s string · b boolean · a array · o object

llm mode omits the codes — types are inferred from cell shape.

getSpec() — when full must talk to an LLM

full is built for parsers. If you need an LLM to read a full payload (for example: you stored data in .loon, you now want a model to query it), getSpec(encoded) returns a minimal decode spec (200–600 tokens) covering only the headers that this specific payload actually uses, plus a worked walkthrough of row 0.

import { getSpec } from 'loon-core';

const encoded = loon.toLOON(data, { mode: 'full' });
const spec = getSpec(encoded);   // { text, sections, estimatedTokens }
// prepend spec.text to the prompt; the model can now decode `full`.

llm mode does not need a spec — that is the whole point of it.

Sessions & context caching

For repeated calls against the same data shape, LoonSession separates the cacheable prompt prefix from the per-call data block. LLM providers cache an identical prefix and bill it at a fraction of the normal rate; put the spec and the schema there and the per-call cost shrinks to just the rows.

import { LoonSession } from 'loon-core';

const s = new LoonSession();
s.init(firstBatch, { mode: 'full' });

// System prompt (mark it for prompt caching): s.primer   ← getSpec() + schema
// User message 1:                             s.dataBlock
for (const batch of moreBatches) {
  send(s.encodeRows(batch).dataBlock);          // only the rows
}

splitLoon(encoded) exposes the raw { schema, dataBlock } split for custom integrations.

Prompt caching reduces input cost. It does not touch output tokens. A model that has to reason hard about a format still pays full price on output. That is why llm mode beats full + cached spec for direct LLM consumption: llm's output token cost is small because the format requires no reasoning to read.

API

Options (LoonOptions)

Architecture

Input (JSON / CSV / XML / YAML / tree)
        │
   Mode Selector ──────────────┐
        │                      │ (auto-pick when mode omitted)
   ┌────┴─────┬──────────┐     │
   ▼          ▼          ▼     ▼
 full       llm      compact  compat
   │          │          │     │
   └── Adaptive ─┘        │     │
   pipeline               │     │
        │                 │     │
        └─────────────────┴─────┘
                  │
            LOON output

full and llm share one analysis pipeline. The analyzer gates off every compute-requiring primitive (Base36, sequences, dictionaries, defaults, suffixes, fixed-point, delta, NORM, RLE, anchor rows) when the mode is llm. What survives is structural-only: the schema, C: constants, and AS: sub-schemas. The header emitter also drops :type codes and replaces ^ (null sentinel) with the literal null for llm mode.

Benchmarks

Measured against json-compact on six datasets. Token efficiency uses the GPT-4o tokenizer; retrieval accuracy is measured on gemini-3-flash-preview. Full reports in ../comprehension-benchmark/benchmarks/results/.

Payload tokens (vs json-compact, GPT-4o)

full is the storage / transmission winner everywhere. llm beats TOON on every dataset and remains LLM-readable without a spec.

Real billed tokens — LLM retrieval (Gemini 3 Flash)

full minimises input tokens by reasoning the model has to do later — output cost blows up because the model has to mentally decode every cell. The remedy is getSpec() (corrects accuracy, partly tames output) or, simpler, use llm mode when the consumer is an LLM.

Round-trip fidelity

All four LOON modes recover all ten benchmark datasets losslessly (10 / 10).

License

MIT © LOON Thesis Team