Version: 0.1.33

Rule Engine

The rule engine runs pipelines: "when data arrives in table X, run this graph" — transforms, calculated mirrors, alarms, derived tables. A pipeline is a dataflow of nodes (a trigger, transforms, a sink) described as JSON, published into the database, and executed by a small daemon.

The one rule: the runner is a client of the database, never a part of it. It logs in over pg-wire like any client, wakes on LISTEN x_data, drains the sync feeds, and writes through ordinary INSERT. The consequences are the point:

a broken user script can never take the database down — ingest does not stop, ever (Lua even runs in its own throwaway subprocess with a hard memory ceiling);
the daemon scales out to more boxes with zero engine changes;
the engine's user/database walls apply to pipelines unchanged.

Install

sudo apt install xcon-db-rule-engine

(from the same apt repository as the engine). One systemd instance per served database:

# 1. the instance's connection — it holds a password, keep it 640
sudo tee /etc/xcon-db/rule-engine/acme.conf >/dev/null <<'EOF'
XCON_PG=host=127.0.0.1 port=5432 user=acme_rw password=... database=acme default_query_exec_mode=simple_protocol
EOF
sudo chown root:xcondbre /etc/xcon-db/rule-engine/acme.conf
sudo chmod 640 /etc/xcon-db/rule-engine/acme.conf

# 2. run it
sudo systemctl enable --now xcon-db-rule-engine@acme

The daemon needs nothing else: the pipelines it runs, the cursors it advances and the leases it holds all live in the database itself, in x_-prefixed tables it declares on first start. Telemetry streams as ndjson to the journal.

The same binary is also a CLI:

xcon-db-rule-engine validate graph.json          # check a blueprint
xcon-db-rule-engine run graph.json               # debug: dry-run + previews
xcon-db-rule-engine run graph.json --write --pg "…"
xcon-db-rule-engine publish graph.json --pg "…"  # store it in the registry
xcon-db-rule-engine bootstrap graph.json --pg "…" # backfill an existing table, then hand off
xcon-db-rule-engine serve --pg "…"               # what the systemd unit runs

bootstrap is a one-shot backfill for a table that already holds data when you first publish a pipeline. Instead of draining the change feed row by row (which pays a storage seek per row, scattered across entities), it sweeps the source table entity-by-entity — each entity is contiguous on disk, so this reads sequentially and runs orders of magnitude faster. It fills the sink and leaves the steady-state cursor exactly where serve should resume, so the daemon picks up new rows without re-processing the backlog. Run it once, then start serve.

Continue with pipelines for the moving parts, jump straight to the features most pipelines exist for — calculated fields, table triggers or buckets for anomaly detection — or go to running pipelines in production for the operating envelope: concurrency, scheduling, lag monitoring and recovery.

Install​

Install