Merge & Compaction

The HTAP bridge: a background worker folds frozen write generations into columnar packs, and a compactor keeps the number of runs per partition bounded — all without ever letting memory grow with the table.

Fold

1. Freeze & rotate — the active generation (WAL + memtable) is frozen and a fresh one takes over. The frozen memtable keeps serving reads until its rows are published.
2. Group by partition — frozen rows are split into 4-week partitions; every channel of a partition is written together into one pack.
3. Write a fresh run — the fold never rewrites a sealed pack. Each partition it touches gets a new run, holding only the delta's rows, resolved on the way out (immutable: keep every sample, drop exact duplicates; mutable: newest version per entity, tombstoned entities vanish). Cost is O(delta), not O(table) — the fold stays cheap however large the table.
4. Publish — the pack renames atomically into place and is registered under the store lock, dropping the frozen rows in the same critical section, so a reader never sees a partition's rows twice or not at all. One rename per partition means there is no half-published, per-channel state to roll back.
5. Prune — once every covered pack is durable, the old WAL files are deleted.

Compaction

Because folds only ever add runs, a hot partition accumulates them. A compactor merges runs within one (partition, level): when a level holds K runs (8 for immutable tables, 6 for mutable) they k-way merge into a single run at the next level, and a level whose run outgrows its size threshold merges down. Merging is always bounded to one partition — never the whole table — and the merged pack renames atomically, its inputs unlinked only after the new run is durable.

Bounded memory

The write path never lets memory scale with the table. A frozen delta is capped by mem_budget (default 512 MiB): once the honestly-accounted resident bytes reach it, ingest applies backpressure instead of buffering unboundedly — the engine refuses, it does not OOM. The fold streams one channel at a time, so a wide table (hundreds of columns) never materializes a whole partition at once. On the read side, query_mem_budget (default 1 GiB) caps a single query's materialized result, read_source_budget (default 4 GiB) caps the decompressed source bytes one statement may hold live — shared across all of its parallel workers, credited back as buffers release — and read_node_budget (default 8 GiB) backstops every concurrent read on the node, so a runaway SELECT * returns an error rather than exhausting the box. statement_timeout (off by default) additionally cancels any single statement past a wall-clock bound, and a client that disconnects mid-statement has its scan cancelled rather than left running.

Triggers & throttle

A fold fires on size (merge_bytes, default 8 MiB of delta) or time (merge_every, default 30 s), whichever comes first. Writing is paced by merge_throttle (bytes/s; 0 = unlimited) so a merge never starves foreground ingest on small hardware.

Retention

Declared per table with RETENTION, enforced by the same worker: whole partitions older than the window are dropped — one directory unlink per partition, O(1), with no rewrite of surviving data. Mutable tables have no retention.