Skip to main content
Version: 0.1.33

The Timeseries Sync Feed

Rows tables are born with a change feed — x_rev rides every write. A timeseries table can opt in to the same contract: every ingested fragment (SQL INSERT and ILP alike) is stamped with a hidden monotone sequence, x_seq, and a client can drain exactly what arrived since it last looked — resumable after a crash, never skipping a concurrent write. This is the door the rule engine's TableTrigger consumes; any client that wants a replica, a mirror, or an audit trail can speak it directly.

CREATE TABLE sensor (p00 DOUBLE, p01 DOUBLE) WITH (SYNC);
-- or later, on a live table:
ALTER TABLE sensor SET SYNC; -- and SET SYNC OFF to stop stamping

Off by default — a table that never feeds a runner pays nothing.

The pull

SELECT * FROM sensor WHERE x_seq > 41250 LIMIT 500;
-- resumable form, guarded against rebuilds:
SELECT * FROM sensor WHERE x_epoch = 3 AND x_seq > 41250 LIMIT 500;
  • Pages come back in ascending x_seq — arrival order, not time order: late data lands in the feed where it arrived, so a downstream consumer never misses a backfilled reading.
  • LIMIT is required — the feed pages, it does not stream unbounded.
  • SELECT * in a drain gains x_seq as the last column (the puller needs its cursor); name columns explicitly to choose your own shape.
  • Each row is the coalesced row at the fragment's (entity, ts), delivered at least once per fragment — re-applying a page is idempotent by construction, so "apply, advance the cursor to the page's max x_seq, repeat" is the whole client.
  • A concurrent slow writer is never skipped: the feed stops below the oldest in-flight write and the next pull picks it up (the same watermark rule as the rows feed).

The handshake

SHOW SYNC FROM sensor;
-- enabled | epoch | seq | watermark
-- true | 3 | 41250| 41250

seq is the write-sequence high-water; watermark is the highest seq a pull may answer right now. A resuming client compares its stored (epoch, cursor) and drains — or resyncs.

Epochs: rebuilds are loud

TRUNCATE (and re-enabling sync) bumps the table's epoch. A cursor that names a stale epoch is refused with resync in full — a rebuilt table can never silently swallow the gap between an old cursor and its new life. The sequence itself never runs backwards, ever: a crash burns a small gap in x_seq and re-mints nothing (gaps are fine — the cursor is an order, not a count).

Two more refusals, same spirit: a cursor beyond the table's write sequence (a dropped-and-rebuilt table) and a drain on a table with sync off. Errors, never empty silence.

Bootstrap

Fragments written before sync was enabled carry no x_seq and are invisible to the feed. Starting a mirror on a table with history means: copy once by time range, then drain from (epoch, 0). Creating the table WITH (SYNC) from day one makes drain-from-zero the complete story.

x_seq and x_epoch are engine-owned (the x_ prefix is reserved): never writable, absent from SELECT * outside a drain and from SHOW COLUMNS.