Running Pipelines in Production

The daemon is production machinery: cursors, leases and registry all live in the database, every write is idempotent, and a crash replays from the last checkpoint. What follows is the operating envelope — the rules and numbers that keep a deployment inside it.

The concurrency model

Every armed pipeline runs in its own worker: one goroutine, one connection, one private cursor loop. A slow pipeline — say, a scheduled full-table LATEST ON recompute that takes minutes — cannot delay another pipeline's drain; the daemon's main loop only routes wakeups. Failure is isolated the same way: a broken run drops its own connection, waits a poll tick and redials, while every other pipeline keeps serving.

Two consequences worth knowing:

• Each pipeline holds one idle connection. A handful of pipelines is nothing; if you publish dozens against one database, budget the engine's connection limit accordingly.
• Runs and schedules interleave freely. Correctness comes from the at-least-once cursor protocol and idempotent sinks, not from any ordering between pipelines.

The one scheduling rule

A scheduled pipeline's cadence must exceed its own run time. The daemon will not start the same pipeline twice concurrently, but a Schedule of 30s on a query that takes 90 seconds runs back-to-back forever — a full-time job occupying a core and, on small boxes, starving everything else of CPU. Measure the run (run-end carries elapsed_ms in the journal), then set the cadence with real headroom:

{"id": "sched", "type": "Schedule", "config": {"every": "5m"}}

is a sane floor for anything that scans a large table.

Monitoring lag

A mirror pipeline's health is one number: how far its cursor trails the table's write sequence. Both ends are SQL:

SHOW SYNC FROM sensor;                          -- seq = write high-water
SELECT key, cursor FROM x_re_cursors;           -- storm-mirror/sensor = drained

seq − cursor is the backlog in rows. In steady state it hovers near zero (fresh rows mirror within seconds). Alert when it grows and keeps growing — that means ingest is outrunning the drain, and the sections below tell you what happens next. The daemon also reports every run and failure to LOGOR when LOGOR_DSN is set — put the alert there.

The throughput envelope

Two different speeds matter, and they are not close:

• The write path (ingest into the source table) sustains north of 10,000 rows/s on modest hardware.
• The drain's read side — picking each new row back out and pushing it through the pipeline — runs at roughly 500 rows/s per pipeline once rows have left the in-memory tail. Reads of very recent rows are much faster.

So a mirror keeps up with sustained ingest in the hundreds of rows per second — the typical fleet/sensor profile, with real headroom. It does not live-track a bulk load or a migration-scale firehose: the cursor falls behind, and past the tail window (the most recent ~500k rows) the feed's row-by-row recovery is seek-bound and slow.

Falling behind is not data loss. The feed is durable and the sink is idempotent; the mirror is merely late. But the fast way back is never to wait for the feed:

Recovery: bootstrap, not the feed

Whenever a mirror is far behind — after a bulk import, an outage, or a misconfigured schedule that ate the box — resync it with the sweep:

sudo systemctl stop xcon-db-rule-engine@acme
xcon-db-rule-engine bootstrap graph.json --pg "…"   # minutes, not hours
sudo systemctl start xcon-db-rule-engine@acme

bootstrap reads the source entity-by-entity in storage order — sequential reads instead of a seek per row — and leaves the cursor at the high-water so serve resumes cleanly. Millions of rows take minutes. It is safe to repeat: the sink converges.

Topology limits (today)

• One daemon per database. Leases hand a dead daemon's pipelines to a successor, but a run that outlives the 45-second lease TTL can briefly execute on two daemons at once — wasted work, not wrong rows, yet not a supported HA mode. Run one daemon; let systemd restart it.
• One writer per calculated table. Two pipelines writing stateful columns into the same target race on the shared formula state (last-writer-wins). Give each calculated table exactly one pipeline.
• Size for the scans you schedule. The engine is comfortable in 2 GB for the steady-state profile; scheduled full-table scans over many millions of rows want memory of their own. If the OOM killer visits, the fix is the scheduling rule above, more memory, or both.

The production checklist

1. Cadences exceed run times (check elapsed_ms in the journal).
2. Lag alert wired: SHOW SYNC seq vs cursor, plus LOGOR DSN.
3. One daemon per database, one pipeline per calculated table.
4. bootstrap rehearsed once — it is the recovery plan.
5. The instance conf is 640 root:xcondbre and holds the only copy of the password.