xcon-db

xcon-db is a lean HTAP database engine for IoT. It combines a synchronous, flexible-schema write path with compressed columnar reads in a single process, on modest hardware — no cluster machinery, no reserved RAM, no external sync pipelines.

What It Does

Devices write sparse, fragmented telemetry; dashboards and analytics read dense time ranges. xcon-db serves both from one engine:

CREATE TABLE telemetry (
  speed DOUBLE FILL HOLD STALENESS 60 s,
  event TEXT   FILL NONE
) WITH (RETENTION 90 d);

INSERT INTO telemetry (entity, ts, speed) VALUES ('veh-1', 1700000000000, 52.5);

SELECT * FROM telemetry LATEST ON ts PARTITION BY entity;   -- device shadow
SELECT avg(speed) FROM telemetry
  WHERE entity = 'veh-1' AND ts BETWEEN 1700000000000 AND 1700086400000
  SAMPLE BY 5 m FILL(PREV);                                 -- downsample

Telemetry is one of three table families: timeseries for the streams, rows for current-state metadata (device registries, catalogs — the document-store shape, sync-native), and kv for key→value state (sessions, leases, caches — the Redis shape, with TTL). The stack an IoT platform usually runs as Mongo + ClickHouse + Redis lives in one engine.

Every language's PostgreSQL driver, Grafana's pg datasource and any ILP-speaking collector work out of the box — xcon-db speaks standard protocols instead of inventing its own.

Design Principles

• The engine knows shape, not meaning — channels are nameless streams to the engine; semantics live in your catalog and code
• Frozen type system: f64 + string — the core never grows; flexibility lives in code
• Store the sparse truth, fill at read time — a filled value is never written to disk
• Writes are synchronous and durable — fsync'ed (group commit) before the ack
• Reads always combine the delta and the columnar — correctness, not an optimization
• db-per-tenant — one self-contained directory per database; move, back up or delete a tenant by moving a directory

Architecture

 ingest (pg-wire / ILP)
        │
        ▼
 delta log (WAL, group-commit fsync)  ──►  merge worker (throttled)
        │                                        │
        ▼                                        ▼
    memtable            +            columnar segments (mmap, per channel)
        └───────────── read path ────────────────┘
                (always both sides)

Layer	Role
Delta log	Append-only WAL; the write buffer is the log
Memtable	In-memory image of the unmerged delta, split per channel
Merge worker	Background delta→columnar folding, size+time triggered, throttled
Segments	Immutable, compressed, per-channel columnar files with sparse indexes
SQL layer	Time-series SQL subset over pg-wire

Status

All engine phases are implemented and tested (delta log, columnar core, merge worker, read path, catalog, SCRAM auth, TS-SQL, pg-wire, LISTEN/NOTIFY, ILP, TLS). Known limitations:

• No bind parameters over pg-wire (inline literals; use simple query mode)
• No pg_catalog emulation — IDE schema browsers stay empty; SHOW TABLES works
• One process per database directory (no lock file yet)
• ILP carries no credentials: bind an ILP listener to one database and firewall it