Forecasting Every Channel, and Checking Whether the Forecast was any Good
How Vigil forecasts thousands of channels with measured error: honest backtests, conformal bands, a fine-tuned Chronos-2, strict promotion gates, and an audit that caught a model learning its own simulator.
Tom Wade
9/9/20267 min read
A forecast without a measured error is a guess with a chart. When I added forecasting to Vigil, the requirement I wrote for myself was short: every forecast must rest on a method I can cite, be validated on data it never saw, and be tracked against what actually happened. This post is about how that became an engine, a fine-tuned foundation model and, most usefully, an audit that caught one of my own models cheating.
All the numbers below come from the demo environment (simulated channels) unless I say otherwise, and every error is measured on held-out data: channels and time windows the models never trained on.
Preparing the series
Forecasts read the fifteen-minute rollup described in part 1. Gaps up to an hour are interpolated linearly, longer ones are filled from the channel's daily profile, and a series that would be more than half filled in is refused rather than forecast. Counters are converted to increments, and state channels (on/off, open/closed) are skipped.
One data-engineering wrinkle mattered more than it looks. The fifteen-minute aggregate is refreshed on a schedule and runs about 40 minutes behind, so the loader tops it up with complete buckets computed from the raw table. Without that, every forecast would start 40 minutes in the past.
A statistical engine first
Before any neural network, I built the baseline every forecast should have to beat, using Nixtla's `statsforecast`: seasonal naive, AutoETS, AutoTheta, and MSTL decompositions with ETS or ARIMA. (Seasonal AutoARIMA over a daily season of 96 fifteen-minute steps took about 40 seconds per series; decomposing first is far faster.)
Each channel gets a rolling-origin backtest: three windows, each as long as the forecast horizon. Prediction intervals are rescaled by split-conformal prediction on the backtest errors, so an "80% band" is calibrated against that channel's own record rather than a model's assumptions. Methods are compared by scaled CRPS (continuous ranked probability score, which rewards accuracy and honest uncertainty together), and the forecast is a quantile average of the best three.
The first mistake I caught was a classic. I chose each channel's method on the same windows I reported its error on, which flatters every result. Selection now happens leave-one-window-out: the method is chosen on the other windows and scored on the held-out one. On the first three demo sites the honest engine scored 0.393–0.401 against seasonal naive's 0.537–0.540, about 26% better, and beat it on 96–97% of channels.
Even then, the bands came out about three points short of their stated coverage at every lead time, partly from the selection itself and partly from a hard first hour. A small adaptive calibration, in the spirit of Gibbs and Candès' adaptive conformal inference, learns a target for each block of lead times from every validation run. Checked on other channels, the 80% and 95% bands held 79.6% and 95.1%.
Then a foundation model
Chronos-2, a pretrained time-series model run zero-shot on the GPU, changed the picture. With it in the engine, held-out error was 0.330 against seasonal naive's 0.530: 38% better, and better on 99% of channels. A pretrained model beat every model I had trained from scratch, including a quantile neural network I'd spent real time on. An LSTM autoencoder that was on the list never got built, because there was no longer a case for it.
Chronos became the default through a shrinkage prior: a channel moves to another method only if that method beats Chronos by more than 15% on the channel's own backtest. It's empirical Bayes in miniature (Efron and Morris's point that the overall winner is usually a better bet than a noisy local one), and it improved held-out error by a further 2.9% (95% interval: 2.0% to 4.0%).
Fine-tuning, with a promotion rule
Vigil-TM is Chronos-2 fully fine-tuned on Vigil's channels. Channels are split by hashing their IDs, 70% for training, and the test windows come later in time than anything the model trained on. The first champion, v2, cut scaled CRPS by 10.2% compared with zero-shot Chronos-2 (95% interval: 8.8% to 11.7%) and was better on 74% of channels. With it, the engine scored 0.281 against 0.319 without, and the 80% and 95% bands held 81.8% and 96.4%.
Two failures from that work are worth passing on. A learning rate of 1e-5 for 1,500 steps made the model 15.6% worse, as fine-tuning overwrote what pretraining had taught it. And adding calendar features made the zero-shot model 11% worse.
Training runs every night, and a candidate is promoted only if it beats both zero-shot Chronos and the current champion by at least 1%, with the whole 95% confidence interval of the improvement (a paired bootstrap over hundreds of held-out channels) below zero. A test on a single window per channel had once called a real improvement "no gain"; judging on hundreds of channels with an interval guards against both kinds of mistake.
The GPU is shared with the local language model, which keeps about 10 GB resident, so training uses micro-batches of 32 with four steps of gradient accumulation. A file lock decides who goes first: training is exclusive, predictions share, and web requests never wait.
The audit that caught my model cheating
The most useful thing I built for forecasting isn't a model. It's an audit command whose job is to prove the numbers wrong.
Some of its checks passed. Replaying history with the future scrambled gave identical forecasts, so nothing was leaking from the future, and live errors matched validation errors for the same engine.
One check failed badly. Model v5 scored 19% better than zero-shot Chronos on the demo fleet, the best result yet. On series from outside the simulator it was 6% worse, worst of all on weekly rhythms, and on real data it was 18% worse. It had learned the simulator. The demo fleet's signals were closed-form: an exact 24-hour sine plus noise, a mean-reverting walk, occasional spikes. There was no working week in them at all. Every model looks good on data like that, and a fine-tuned one learns to exploit it.
The fixes were in the data and in the rules. The simulator now gives each site weather, a working week, machine duty cycles and sensor artefacts. I added a frozen real-data benchmark (open weather data at 24 locations, and Great Britain's electricity demand), a separate real training set, and a stress suite of ten signal families. Promotion now also requires the candidate to be no worse out of distribution and on real data. v6, trained on simulated, realistic and real series together, passes every gate and is the champion.
What forecasts are for
A forecast is only useful if it changes a decision, so a few features sit on top of the engine:
Predictive alerts fire when the chance of crossing a limit within the horizon passes a threshold, update with each new forecast, withdraw with hysteresis (below 0.8 times the firing probability), and settle as a hit or a miss against what actually happened.
The forecast doctor triages forecasts that scored badly, diagnoses a cause (a level shift, changed seasonality, a stuck sensor, an unseen driver), tries one remedy at a time on a backtest, and reviews it a day later: kept if it's at least 5% better, rolled back if not.
The scenario lab fits how one channel responds to another, with a lag, and answers "what if". Its bands carry the uncertainty in the fitted gain, with the standard error corrected for autocorrelation so it isn't overconfident.
Foresight takes a question in plain language, plans which forecasts answer it, and streams the result.
What broke elsewhere
A forecast sitting on a limit was "97.5% likely" to cross it. Tied quantiles at the limit made the crossing probability jump. The distribution is now right-continuous for "above" and left-continuous for "below", with each tail fitted separately.
Bridge accelerometers were "100% likely" to cross every day. They did, every day: seven days out of seven. Crossings now carry how many days they happened on, and three or more days in seven counts as routine. Routine crossings are shown, but never become alerts or memories.
Unscored forecasts vanished from scoring. `.exclude(score__complete=True)` reads like "everything not yet complete". In SQL, a row with no `complete` key gives NULL, NOT NULL is still NULL, and the row disappears. The query now asks for "never scored, or scored and incomplete" explicitly. The same trap turned up later on a different field, which is why it's written into the project notes.
What I'd keep
Judge every change by held-out error with an interval, never by training loss. Keep a baseline that every model has to beat, channel by channel. And build the audit before you need it: the model that most needed catching was the one with the best number.
---
How this was built. I'm the founder of Continuis Labs. I set Vigil's architecture and requirements, made the product and engineering calls, and reviewed, tested and measured the work; AI coding agents (Claude Code among them) wrote much of the code under my direction. Every figure here comes from the platform's own validation runs and audits; the demo fleet is simulated, and the real-data benchmark is public weather and electricity-demand data.
Next in the series: from CAD to the browser, designing a modular sensor node.
