Storage Format

An immutable table's data lives in time partitions — 4-week (28-day) buckets. Every channel of a partition is stored in one file, a pack, so a read maps one file per partition it touches, not one per (partition, channel). At billions of rows that is the difference between tens of thousands of open files and a few hundred.

On disk

<db>/data/t<table>/p<part>/pack.seg

A pack concatenates one self-contained sub-segment per channel, then a per-partition entity index and a directory:

"XCPK" | version u8 | table uvarint
  sub-segment(channel a)      each is a standalone segment, offsets relative to its own start
  sub-segment(channel b)
  ...
entity index                  count, then each entity — sorted, distinct, across all channels
directory                     per channel: id, kind, byte offset, byte length
CRC(header | entity index | directory) u32 | entLen u32 | dirLen u32 | "XCPK"

A reader mmaps the file once and hands out a per-channel view over a byte slice of that single mapping — so the whole partition costs one memory mapping regardless of channel count. The entity index answers "who is in this partition?" without decoding a single block.

Sub-segment — one channel

Each sub-segment is itself a sealed columnar run of the channel's (entity, ts)-sorted samples, split into blocks with a sparse index:

"XCSG" | version u8 | table uvarint | channel uvarint | kind u8
block*        rowCount | entities col | timestamps col | values col
footer        per block: entity span, ts min/max, value min/max, Bloom filter, offset, length, count

Default 8192 rows per block. Writers enforce (entity, ts) order. Because a sub-segment's block offsets are relative to its own start, the exact same bytes read back whether they sit in a pack or (in an exported bundle) as a standalone file. The sparse index decodes into memory on open; blocks decode on demand — the OS page cache is the only cache, no buffer pool.

Compression. In a pack, each channel's sub-segment compresses per block: every 8192-row block is its own zstd frame, and the sparse index (ts/value zone maps, Bloom filters, entity spans) stays outside the frames, plain in the file. A reader prunes blocks straight off the mmap and inflates only the blocks it actually visits — a point probe or a "newest 100" read decodes a handful of blocks, never a whole channel. Writing streams block by block, so memory stays bounded to one block. On real telemetry this roughly halves the on-disk size: because the store is already sparse (only non-default samples are kept) and then compressed, the footprint lands at or below a dense columnar store's. Packs carry a version: the reader transparently reads every older generation (uncompressed packs and the earlier whole-channel-frame compression), so an upgrade needs no rewrite — old packs convert to the current layout as compaction naturally rewrites them. To convert a whole database at once (and get block-granular reads on cold historical partitions immediately), the offline xcon-db-convert -db <dir> tool re-encodes every pack in place: it refuses to run against a live engine, verifies each converted pack against its source row by row, and renames atomically, so an interrupted run leaves nothing half-written. Older binaries cannot read the current pack format — upgrade is one-way once new data is written.

Encodings

Column	Encoding	Why it wins on IoT data
timestamps	delta-of-delta, zigzag varints	steady cadence → ~1 byte/point
f64 values	Gorilla/XOR bit-packing	slow drift → ~1 bit/point when constant
strings	adaptive: dict+RLE or block-zstd	engine picks by cardinality; state channels collapse to a handful of runs
entities	same adaptive string encoder	few devices per block → dictionary

The user only ever declares f64 or string; encoding choice is the engine's, made per block by looking at the data (a heuristic, not a model).

Skipping a read

Four prunings drop work before a block is ever decoded:

• partition — a time-bounded read touches only the 4-week partitions overlapping its window.
• entity index — an unbounded listing (SELECT * with no time filter, SHOW of the devices) reads each pack's index, O(entities), with no block decode.
• block by time — inside a channel the sparse index skips blocks whose [minTS, maxTS] misses the range.
• block by value — for i64 channels each block also carries a value [min, max], so a value predicate (a geo covering, an integer comparison) skips blocks the way the time window does.
• block by Bloom filter — text/bytes blocks carry a small Bloom filter (~10 bits per distinct value), so a string equality rules a block out from the footer. "Maybe" still decodes and exact-compares: false positives cost speed, false negatives are impossible.
• postings index — a column declared INDEX adds value→block postings to each pack's tail: an equality probe jumps straight to its candidate blocks (and the entities they name) without walking anything else.

Filtered reads

A WHERE on ordinary columns does not reconstruct rows to test them. The planner scans only the compared column first and derives the timestamp intervals where the predicate can hold — an event (FILL NONE) column contributes exact points, a state (FILL HOLD) column contributes the carry intervals between samples, staleness-clipped. Rows are then reconstructed exclusively inside those intervals (and every filter is re-checked on the assembled row, so the intervals are a hint, never an authority). The single-column scan runs as tight loops over decoded block arrays, uses every skipping above, and fans out across CPU cores by entity range; an unordered LIMIT stops the walk the moment it is satisfied.

Legacy layout

Databases written before packs stored one c<channel>.seg per (partition, channel). The engine still reads that layout transparently — no migration — and writes packs going forward; the first compaction of a legacy partition rewrites it into a pack.