Designing a Platform that Remembers

Much of my working life has been spent with machines that have already failed. Electronic control modules arrive on the bench dead or misbehaving, and the job is to work out what happened inside them. Failure analysis teaches one discipline above all: you can only reason from what the object kept. A module that spent nine years in a truck arrives with no account of any of it.

Most monitoring software has the same problem one level up. It stores readings, sometimes for years, and shows you the latest one. What it doesn't keep is what an experienced engineer carries in their head: this gauge drifted like this last spring, and it turned out to be the sensor, not the structure. When that engineer retires, the knowledge leaves with them.

Vigil is the platform I'm building at Continuis Labs to fix that for physical infrastructure: bridges, dams, substations, treatment works. It collects sensor data, and it also writes down what the data meant, as durable, checkable claims I call memories. This post is about how that data layer is designed, why, and the bugs that taught me the most.

One note on scale before anything else. The numbers below come from Vigil's demo environment, a simulated fleet of about 10,000 devices across 156 sites, with 39,255 channels. The data is simulated. The software, the database and the problems are real.

Two kinds of data

The design rests on one split. Readings are what a sensor measured at an instant. Memories are claims about a monitored thing over a period of time, with the evidence that justifies them. Readings are many, cheap, and worth less as they age. Memories are few, valuable, and meant to last. They get different storage, different schemas and different rules.

Readings: a time-series store in three tiers

Readings live in PostgreSQL with the TimescaleDB extension. I chose it over a dedicated time-series database because everything else in the platform (devices, sites, alert rules, memories, users) is relational, and joining a reading to its device and site in one SQL query is worth more to me than a few percent of ingest speed.

The schema is narrow: one row per reading per channel, holding `time`, `device_id`, `binding_id`, one of `value_float`, `value_text` or `value_json`, and a `quality` flag. A binding is one channel of one device, mapped to a quantity in a registry such as `temperature.contact`, `pressure.gauge` or `vibration.velocity_rms`. A wide table with a column per quantity would have needed hundreds of mostly empty columns and a migration every time someone connected a new kind of sensor. With a narrow table, a new kind of sensor is a new row in the registry, not a schema change.

Readings are split across three hypertables by how fast they arrive:

• measurements_low for slow channels: temperatures, levels, pressures.

• measurements_high for fast summaries, such as vibration levels.

• measurements_ultra for burst waveforms, each stored as an array of samples with its sample rate and duration, and kept for 30 days.

Everything is stored in SI base units: kelvin, pascals, ratios for humidity. Statistics, detectors and forecasting models never have to ask which unit a number is in; conversion to °C, kPa or %RH happens only at the edge, when something is shown to a person.

On top of the raw tables sit four continuous aggregates, TimescaleDB's incrementally refreshed materialized views: one-minute, fifteen-minute, hourly and daily buckets holding the mean, minimum, maximum and count for each channel. A query picks its tier from the span it asks for: raw data up to six hours, one-minute buckets up to two days, fifteen-minute buckets up to two weeks, hourly up to three months, daily beyond that. A small cache table keyed on (device, binding) holds each channel's latest value, upserted on every write, so "what is everything reading right now?" never touches the hypertables.

Data arrives through several doors: a push API that accepts plain names (`{"temperature": 22.4}` becomes the right quantity, stored in kelvin), MQTT for our own nodes, read-only Modbus TCP and OPC UA connectors for existing plant equipment, and a historian CSV import that writes straight to the tables and then refreshes the aggregates over the imported span. The live paths share one store function. I hold every door to one rule: if nothing was stored, say so. The push API returns a 400 rather than a cheerful 200 when none of the fields matched a channel.

Memories: claims with evidence and provenance

A memory is a row in an ordinary Django model, but four decisions make it more than a log line.

It's an interval, not an instant. Every memory has `period_start` and `period_end`, with a database check constraint that the end isn't before the start. A note someone writes about a moment is the degenerate case where the two are equal. Getting this right changed everything downstream: "what did we know about this dam during the week of the storm?" becomes an interval-overlap query, not a timestamp lookup.

What it says has two dimensions. `kind` is the shape of the claim, one of 18: an anomaly, a regime change, a trend, an exposure (a running total against a budget, like fatigue cycles), a response (Y follows X with a gain and a lag), a deviation from its peers, a calibration or data-quality problem, and so on. `mechanism` is the physical process, from a catalogue of 70 across 12 domains, such as bearing outer-race defects, scour progression or thermal faults in a transformer. I split them because "bearing outer-race defect" and "weld seam fatigue" are the same shape of claim about different physics. A single list would need an entry for every failure mode in every domain and could never be finished. Mechanism is free text on purpose, so a customer can name something the catalogue has never seen.

It carries its evidence. A JSON field holds the numbers that justify the claim, enough to re-derive it from the raw series: the window's statistics, the baseline it was compared with, the thresholds that applied. The detail page has a "re-derive from raw measurements" button that recomputes them and flags the memory if they no longer agree.

It says who made it and when. `source` (sensor, model or human), `method` (a versioned routine name such as `rolling_baseline_v1`, so a change in method can be traced), `author` and `created_at`, plus `verified_by`, `verified_at` and `disputed`. One rule holds across the whole interface: every memory shown anywhere says who or what made it, and when. A claim without an author and a date can't be weighed against a later one.

The indexes follow the access pattern, which is almost always "this thing, this period, most important first": composite indexes on (organisation, period end), (device, period end) and (site, period end), one on (organisation, kind, salience), and GIN indexes on the quantity and tag arrays.

Detectors form memories. I started by writing them one at a time, until it became clear that 70 mechanisms are really about a dozen algorithms configured many times: periodicity, deviation from peers, limit exceedance, exposure, response, data quality and a few more. They now live in a registry, configured per mechanism, with settings resolved from organisation down to site, device and channel, the most specific winning. One rule isn't negotiable: a detector whose required numbers aren't set doesn't run. A fatigue budget belongs to one asset; guessing it produces confident claims against the wrong threshold.

The memory timeline counts how many memories overlap each time bucket rather than how many were written in it. It's one SQL query, `generate_series` joined on interval overlap, and takes about 125 ms for 6,000 memories over 30 days. Individual periods are drawn only when a filter brings the set under 600. Past that they smear into something unreadable, so the interface says so and asks you to narrow it.

What broke

Each of these was caught by measuring rather than by reading code, and most now have a test named after them.

A claim that disagreed with its own evidence. Look closely at the screenshot below: the headline says 8.5σ, and the evidence underneath says 13.54σ. I noticed it while choosing pictures for this post. The cause was in how memories merge. When a detector finds the same event in overlapping windows, the findings merge into one memory whose period grows. The merge kept the strongest sentence but the newest evidence, so the two drifted apart. When I counted, about half of the anomaly memories in the demo fleet disagreed with their own evidence. The fix is a rule: a sentence and its evidence always come from the same finding. The original detectors now write each sentence from the numbers their evidence stores, and only those, and a repair command uses the same code to rewrite existing sentences from their own evidence, keeping a backup.

The period finder that preferred the wrong peak. To find a channel's cycle, the periodicity detector looks for peaks in its autocorrelation. An early version took the highest peak. But a tank with an 18-hour fill cycle and a 24-hour thermal cycle correlates best at 72 hours, where the two line up again. On real tank data, r(18 h) was 0.40 and r(72 h) was 0.61. The fundamental is the weaker peak. The detector now takes the first local maximum after the first negative crossing.

A fleet that was mostly never examined. The detector runner sorted devices by name and took the first 150 each pass. That's the same 150 devices every time, and 9,850 were never looked at. It now walks the fleet with a cursor.

Five thousand false alarms a day. Adaptive alert bands learn what normal looks like for each channel. On the demo fleet they first produced 5,518 excursions a day, and every cause was a question of resolution or scale. Rates depended on the sampling interval; the fit read 52-minute averages while the check read raw samples with five times the spread; seasonal buckets fitted from six samples were hypersensitive. Fixing those three brought it to 16 to 23 excursions a day across 570 channels. Two more turned up in what remained, including a binary channel that had only ever read 0: its band had zero width, so a single 1 was "a trillion band widths out".

Rollups bigger than the data. In early October the continuous aggregates took 166 GB, more than the 116 GB of raw data they summarize, and they didn't appear in the usual hypertable size views at all. By the time I trimmed them they had reached 185 GB. Compressing every rollup segmented by channel, keeping the one-minute rollup for only 30 days (older windows now read the fifteen-minute one, which the tier selection knows about) and shorter chunks brought that to 112 GB, still falling as chunks age.

Thirty-second page loads. Every page opened a server-sent-event stream for the alert badge, each holding a worker thread for up to twenty minutes. After about eight clicks the threads were gone, and the next page waited exactly 29.x seconds. Separately, Django's gzip middleware buffered streaming responses, so every live stream reached browsers in one lump at the end; curl looked fine because curl doesn't ask for gzip. The badge now polls, streams are capped, and event streams are never gzipped (the test sends `Accept-Encoding: gzip`, the condition that hid the bug). The same audit cut the trace viewer's channel list from 17.9 MB of JSON and 197,000 DOM nodes to 6 KB and 1,327 nodes, by searching on the server instead of shipping all 39,255 channels.

What I'd keep

If I started again, three decisions would survive unchanged. Store base units and convert at the edge. Make every claim carry its evidence and its provenance, and show both everywhere it appears. And measure before believing: almost every bug above produced output that looked reasonable until someone counted.

---

How this was built. I'm the founder of Continuis Labs. I set Vigil's architecture and requirements, made the product and engineering calls, and reviewed, tested and measured the work; AI coding agents (Claude Code among them) wrote much of the code under my direction. Every figure here comes from the platform's own tests, audits and logs, and the demo fleet is simulated.

Next in the series: Argus, an AI agent that cites its evidence and asks before it acts.

Connecting the digital to the physical.

Contact

Links

wadesthomas1@gmail.com

+12155019211

© 2026. All rights reserved.