A Trillion Witnesses

In October of 2016 a large portion of the American internet became unreachable for most of a day. The cause was not a sophisticated state operation. It was a botnet assembled almost entirely out of home security cameras and digital video recorders, devices that had shipped with default credentials their owners never changed and that no longer received updates from manufacturers who had, in many cases, ceased to exist.

The interesting thing about that event is what it revealed about the shape of the problem. Nobody had done anything unusually wrong. Each individual device was cheap and unremarkable and behaved exactly as designed. The failure was in the aggregate: a few hundred thousand unremarkable objects, each irrelevant alone, turned out to constitute critical infrastructure that no one had decided to build and no one was responsible for.

Those devices could only send packets. In my other writing I have argued for a world with vastly more instrumentation than we have now — sensing dense enough to find every leak in a water system, watch every transformer, and give a robot a building that can describe itself. I believe in that future and I am working on it. This essay is about the fact that it is also a world with vastly more devices that observe, retain, and eventually act, and that the honest version of the argument has to reckon with what that means when it goes wrong.

I want to be careful about how I do this, because there are two failure modes in writing about technological risk and both are common.

• Avoid reflexive technophobia. The genre of warning about ubiquitous computing has a long history of being simultaneously right about the mechanism and useless about the response, because it treats the technology itself as the problem rather than particular design choices within it. That framing produces no actionable engineering, only a mood. It also gets ignored by the people actually building, which is exactly backwards, since they are the only ones positioned to fix anything.

• Acknowledge real uncertainty. I do not know that the scenarios below will materialize. Some may be prevented by ordinary market dynamics, some by regulation that already exists, and some may simply not happen. I am not making predictions. I am identifying failure modes that seem plausible enough that someone should be thinking about them now, while the architecture is still soft.

• Prefer architectural fixes to prohibitions. The most durable answer to most of these problems is a design decision made early, not a rule imposed late. Rules can be evaded, repealed, and jurisdictionally arbitraged. An architecture where the customer holds the trust anchor is harder to reverse than a regulation saying vendors should behave. Where I do think rules are needed I will say so, but my bias is strongly toward getting the structure right while it is still cheap.

The premise I am working from is the one I laid out in the companion essay: measurement nodes cheap enough to deploy by the thousand, lasting a decade unattended, carrying substantial local memory and their own coherent timeline, describing themselves in a verifiable machine-readable form, deployable by machine, and increasingly capable of acting rather than only observing. Call it pervasive instrumentation. I think it is coming, on a timescale of one to two decades, whether or not I build any of it.

Here is the thing that makes it different from previous waves of computing. The devices in question are physically distributed through the environment rather than held by users, they are numerous enough that no one can enumerate them, they outlive the companies that made them, and they are attached to things that matter. That combination produces failure modes with no good precedent.

I. The room remembers you too

Ambient observation and the collapse of consent

The first and most obvious risk is that instrumentation dense enough to be genuinely useful is instrumentation dense enough to be genuinely invasive, and the boundary between the two is not where most people assume.

The intuitive model of surveillance involves cameras and microphones, and the intuitive defense is to avoid deploying them. But the concerning inferences do not require either. Electrical current draw on a circuit reveals which appliances are running and therefore when someone woke, showered, cooked, and left. Vibration in a floor slab reveals footfall, and footfall reveals occupancy, gait, and eventually identity. Carbon dioxide concentration in a room is a direct proxy for how many people are in it and for how long. Water flow tells you who is home. Thermal patterns tell you which rooms are used.

Every one of those measurements has an unambiguously legitimate purpose. Every one of them is also a behavioral record. There is no version of this technology that provides the first without producing the second, and any design that pretends otherwise is deceiving itself.

The consent problem is worse than the observation problem. Existing privacy frameworks are built around a user who interacts with a device and can be presented with terms. Ambient instrumentation observes people who are not users: employees in a workplace, tenants in a building, visitors, children, contractors, the neighbor whose wall abuts the sensor. None of them agreed to anything, most do not know the system exists, and the party who did consent — the building owner, the employer, the landlord — has interests that frequently diverge from theirs.

The asymmetries this creates are predictable, and some are already visible. Employers monitoring workers under the banner of safety or efficiency, where the instrumentation was justified by equipment condition and quietly repurposed. Landlords with fine-grained occupancy data about tenants, which is useful for enforcing lease terms and for a great deal else. Insurers pricing on continuous behavioral observation, where declining to be observed becomes itself a signal. In each case the technology was deployed for a defensible reason and the observation was a byproduct that turned out to be more valuable than the original purpose.

Defenses

I think there are four things worth doing, in descending order of how confident I am.

First, aggregate at the edge by default. The single most effective privacy measure available in this architecture is that nodes have enough local memory and compute to answer questions without exporting raw data. A node can report that a zone is occupied without exporting the footfall waveform that would identify who is walking. This is not a policy; it is a default that engineers choose, and choosing it costs almost nothing because the local processing capability is there anyway. Raw high-resolution data should be something you deliberately request, not the normal product of the system.

Second, make retention a property of the data. Every observation in this architecture already carries structured metadata about its provenance. Retention policy belongs in the same envelope: this class of measurement expires in thirty days unless something specific happens. Enforcement at the storage layer, applied automatically, is worth far more than a policy document nobody reads.

Third, treat the distinction between instrumenting a place and monitoring a person as a first-class design constraint, and be honest about which one you are doing. A great deal of ambiguity here is not technical but rhetorical, and the discipline of writing it down changes designs.

Fourth — and this one does require law — the people observed by ambient instrumentation need standing that does not depend on being the customer. Employees and tenants have almost no rights over data generated about them by systems they did not purchase. That gap is a policy problem and I do not think architecture solves it. I would rather the industry propose something workable than have something unworkable proposed for it after the first serious scandal.

II. The enclosure of ordinary fact

Ownership, lock-in, and abandonment

The second risk is subtler and I think it is the one most likely to actually happen, because it requires no bad actors at all — only ordinary business incentives operating as designed.

The measurements produced by pervasive instrumentation are facts about physical reality: the pressure in a pipe, the temperature of a transformer, the strain on a beam. Those facts are generated on someone's property, about their equipment, frequently at their expense. Under current industry norms they end up owned by whoever manufactured the sensor, held in that vendor's cloud, accessible through that vendor's API, on that vendor's terms, for as long as that vendor exists.

Scale that up and you arrive at private ownership of the empirical record of the physical world. Not because anyone conspired, but because every individual vendor made the locally rational choice to retain their customers' data, and because the data model is the moat.

The consequences compound in unpleasant ways. A facility that wants to correlate across systems cannot, because each vendor's terms prohibit the export that would make correlation possible. A researcher who could answer a public-interest question about infrastructure condition cannot get the data. A city that instrumented its water network discovers at renewal that the pricing has changed and the switching cost is the entire deployment. This is not hypothetical; it is the current state of building management systems, industrial historians, and agricultural equipment telematics, and it will simply be reproduced at a hundred times the scale unless something changes.

Then there is abandonment, which is the version that keeps me up at night, because devices in walls outlive the companies that made them by decades.

It is worth being concrete about the timescale, because software people routinely underestimate it. A commercial building has a fifty-year service life and its mechanical systems turn over on roughly twenty-year cycles. A water main lasts a century. A bridge lasts longer. Anything embedded in those assets will outlive not only the vendor but the procurement officer, the integrator, the standard it was built to, and in many cases the operating company. We are proposing to distribute computers through infrastructure on timescales where the relevant comparison is not a phone or a laptop but a building code.

Consider what happens when a sensor vendor fails. The devices keep working, or they do not, depending on whether the vendor was foolish enough to make them cloud-dependent. Updates stop. Security vulnerabilities go unpatched permanently. The historical data is in a cloud account that stops being paid for. And the devices are in the ceiling, behind drywall, under a road, in a substation — physically expensive to remove and therefore left in place. We are on a path toward a built environment full of abandoned, unpatchable, unowned computers embedded in things that matter. The Mirai devices are the small version of this problem.

Defenses

The architectural answer here is more satisfying than in the previous section, because it is achievable unilaterally by anyone building this equipment.

Local-first operation, meaning the system's full functionality is available without any vendor service. This is the single highest-leverage decision in the entire category. If a device is completely useful when disconnected, then vendor failure degrades the experience rather than destroying the asset, and lock-in has to be earned through quality rather than imposed through dependency.

Open, documented data at rest, so the historical record is readable by the owner with standard tools rather than through an API that can be revoked. The measurements are facts about the customer's property; the vendor's product is the instrument and the analysis, not custody of reality.

Customer-held trust anchors. In an architecture where devices verify signed identities, whoever controls which signing keys are accepted controls the ecosystem. If that is always the manufacturer, verification becomes lock-in with a security justification. If the operator can hold their own trust anchor and choose to trust additional parties including themselves, the same cryptography becomes a tool of ownership. These two designs look nearly identical on a block diagram and produce opposite worlds.

And a genuine answer to end-of-life. I would like to see it become normal for firmware, tooling, and specifications to be escrowed and released under an open license when a product line is discontinued or a company dissolves. Software abandonment is annoying. Abandonment of ten thousand devices grouted into a bridge deck is a public liability, and it should be treated as one.

III. When the botnet has hands

Security at physical scale

The Mirai devices could only generate traffic. The systems I have described observe, retain, and in their mature form actuate. A compromise of a sensing network is a data breach with unusually intimate contents. A compromise of an actuating network is a physical event.

Three properties of this domain make the security problem structurally harder than in conventional computing.

The devices are constrained. Cryptography that is routine on a phone is expensive on something that wakes for milliseconds a day. Certificate validation, revocation checking, and key rotation all assume power and connectivity that these devices do not have. It is easy for well-intentioned engineers to make small compromises that individually seem reasonable and collectively leave a fleet defenseless.

The lifetimes are absurd. A node deployed in 2028 with an expected fifteen-year life must remain secure against 2043's attackers, using cryptography chosen in 2027, with an update mechanism designed by people who will have moved on. Almost nothing in modern software engineering prepares anyone for that timescale. It is closer to civil engineering, and the industry does not think like civil engineers.

Monoculture creates correlated failure. The economics of this business push toward a small number of platforms deployed identically at enormous scale. That is exactly the condition under which one vulnerability becomes a simultaneous, global event rather than a series of local ones. The diversity that limits blast radius in biological systems is the opposite of what manufacturing efficiency wants.

Layer on the supply chain. A modular architecture where probes attach to nodes creates a physical interface anything can be plugged into, by anyone, over a fifteen-year service life, frequently by a contractor nobody supervised. A counterfeit probe reporting false calibration produces measurements that are wrong in a way nothing downstream detects, which is a worse outcome than an obvious failure.

Defenses

Verified identity at every interface, checked before power is committed rather than after — which is the central argument for building cryptographic identity into the connector itself rather than layering it on later. Post-quantum signatures for the assertions with long lifetimes, chosen now, because the expensive part of a cryptographic migration is never the algorithm but the installed base that has no field for one.

Physical constraints that survive software compromise. An actuator that cannot exceed a limit because of how it is built is more trustworthy than one that will not exceed a limit because of what it was told. This is old, well-understood safety engineering, and it is being quietly abandoned in favor of software interlocks by people who have not thought carefully about adversaries.

Deliberate diversity in critical infrastructure, even at a cost. Multiple implementations, multiple vendors, multiple key hierarchies, so that one compromise is not universal. This is an argument for open specifications with several independent implementations rather than a single excellent product, and it cuts against the commercial interest of anyone building one, including me.

And a serious answer to update over a fifteen-year horizon, which almost certainly means the ability to update devices that are physically unreachable, over a network that may not exist at the moment the update is needed, with authorization that does not depend on a company still being solvent. I do not think anyone has solved this. I think it is one of the more important unsolved problems in the field.

I want to add one more property that makes this domain unusual, which is that the consequences of compromise are not evenly distributed and the people bearing them are not the ones who chose the equipment. A homeowner whose thermostat is conscripted into a botnet has a minor problem. A city whose pressure management system is manipulated has a major one, and the residents whose basements flood did not select the vendor. Security decisions in this space are made by people who will not bear the cost of getting them wrong, which is the standard precondition for systematic underinvestment.

IV. Nobody decided that

Action without accountability

The transition from sensing to acting is where the risk profile changes category, and it will happen gradually enough that nobody notices the threshold being crossed.

It starts sensibly. A system that detects a leak closes a valve, because waiting for a human costs water. A system that sees a thermal excursion sheds load. Each individual automation is defensible, each is faster than a person, and each removes a human from a loop they were mostly rubber-stamping anyway. Aggregate enough of them and a large share of physical actions in the built environment are initiated by software acting on inference, with no human in any particular loop.

The failure mode is not dramatic. It is that when something goes wrong, nobody can reconstruct why. The state of the world at the moment of decision was assembled from a hundred devices, some of which were miscalibrated, some of which had stale data, some of which were reporting from a network partition. The inference was made by a model whose training data nobody has retained. The action was taken by a controller three vendors deep in an integration nobody has documented since the person who built it left.

This is not primarily a legal problem, though the liability questions are genuinely unresolved. It is an epistemic one. A system whose decisions cannot be reconstructed cannot be improved, because you cannot learn from a failure you cannot explain. And organizations respond to that opacity in a predictable and corrosive way: they stop asking. The alarm becomes weather. Nobody remembers why the setpoint is what it is. The institutional knowledge of why the system behaves as it does evaporates within a few staff turnovers, and what remains is a machine that everyone obeys and nobody understands.

There is a related loss I want to name because it is easy to dismiss as nostalgia and I do not think it is. Skilled technicians build intuition through diagnosis. If diagnosis is automated away entirely, the pipeline that produces people capable of handling the situations the automation does not cover is quietly severed. Aviation has learned this lesson expensively, more than once. We should assume the same dynamic applies wherever expertise is displaced rather than augmented.

Defenses

Receipts, by which I mean a tamper-evident record of what was commanded, by which system, on what evidence, at what time, with what confidence. Not a log file, which is mutable and usually rotated away before anyone needs it, but a signed, retained artifact treated as part of the action rather than a side effect of it. If a system can act, it should be architecturally incapable of acting without leaving one.

Uncertainty that propagates. A control decision made from a measurement whose timestamp is uncertain to four hundred milliseconds should be able to know that. Systems that strip uncertainty at ingestion — which is nearly all of them — make confident decisions from data that did not deserve confidence, and this is a solved problem in metrology that industrial software has simply declined to adopt.

Human-legible explanations as a requirement rather than a feature. If a technician cannot look at an automated decision and disagree with it in specific terms, the system has produced compliance rather than understanding. The test is whether an experienced person can argue with the output using the same evidence.

And deliberate preservation of skill. If diagnosis is automated, someone has to think about how the next generation of diagnosticians is trained, because it will not happen incidentally anymore. This is an organizational problem that technology creates and cannot solve.

V. Ground truth is a kind of power

Concentration, states, and who gets to know things

The risks so far have been about systems behaving badly. This one is about systems behaving exactly as designed, for someone whose interests are not yours.

Whoever operates a pervasive sensing substrate holds something that has not really existed before: authoritative, continuous, verifiable knowledge of physical reality across a territory. Not inference from satellites, not statistical sampling, but the actual state of the actual pipes, roads, fields, buildings, and machines, timestamped and attributable. That is an enormous asset, and the question of who accumulates it is a question about power rather than about technology.

There are three concentrations worth worrying about, and they are not equally likely.

Commercial concentration is the most probable. The economics of this business favor scale hard: certification costs, security engineering, and manufacturing volume all reward the largest player, and the value of the data grows superlinearly with coverage. The natural end state is a small number of firms holding the empirical record of the built environment, with the ability to price access to facts about your own property. If that sounds abstract, note that it is already the situation in agricultural telematics and building management, just at a smaller scale.

State use is less probable in democracies and close to certain elsewhere. A dense sensing network deployed for legitimate infrastructure purposes is, with no hardware modification whatsoever, a national-scale observation system. The measurements described in the first section — occupancy from carbon dioxide, movement from vibration, activity from current draw — are not surveillance until someone queries them that way, and the query is free. I would note that the countries most likely to build this out fastest are the ones with the most centralized infrastructure planning, which is not a comforting correlation. And unlike cameras, this equipment does not look like surveillance, which means it does not attract the political attention that would otherwise constrain it.

Export of the capability is the version that receives the least attention and may matter most. Surveillance technology has a well-documented history of being developed under civilian justification in one country and sold as an instrument of control to another. There is no reason to expect pervasive instrumentation to be exempt, and quite a lot of reason to expect the opposite, since the civilian justification here is unusually strong and therefore unusually good cover.

I want to flag an asymmetry that I find genuinely troubling. In every previous wave of computing, individuals had at least the theoretical option of non-participation — you could decline the device, close the account, leave the platform. Ambient instrumentation removes that option structurally. You cannot opt out of the building you work in, the road you drive on, or the water system you depend on. The observed party has no exit, which means the usual market correction of customers leaving is simply unavailable, and the only remaining corrections are regulatory or architectural.

Defenses

Federated by default, meaning the architecture should make it natural for many parties to each hold their own instrumentation and their own data, and unnatural for one party to aggregate everything. This is largely a consequence of the local-first and open-data choices already described, which is a useful property: the same decisions defend against three different failure modes.

Public ownership of public infrastructure data. When a municipality instruments its water system, the resulting record is a public asset and should be held that way, with the vendor supplying instruments and analysis rather than custody. This is a procurement decision more than a technical one, and it is winnable right now while the deployments are small and the contracts are being written for the first time. In ten years it will be a fight rather than a choice.

Explicit limits on secondary use, ideally contractual and eventually statutory. Data collected to maintain a chiller should not be lawfully available for assessing employee productivity, and the way to establish that is to write it down before anyone has built a business on the alternative.

And a norm within the industry against building the capability itself, not merely against using it. I am aware that this is the weakest item on the list, because norms fail under commercial pressure and someone always defects. But norms are what exist before law, and the engineers who will be asked to build the first national-scale ambient monitoring system have more leverage in that moment than they will ever have again.

VI. The weight of a trillion things

Material and environmental cost

There is an irony at the center of this technology that deserves direct treatment rather than a footnote.

The case for pervasive instrumentation is substantially environmental: less water lost, less energy wasted, less food spoiled, less equipment discarded. But the means is the manufacture of an enormous number of physical objects, each containing a battery, a radio, semiconductors, plastics, and a variety of metals whose extraction is neither clean nor ethically simple.

If we are talking about hundreds of billions of devices, the embodied cost is not negligible and the disposal question is severe. A decade-life primary cell is a decade-life primary cell, and multiplying it by a very large number produces a waste stream that current recycling infrastructure is not remotely prepared for. It would be an unpleasant outcome to instrument the world for sustainability and produce a landfill problem of comparable magnitude to the savings.

There is a second-order version too. Cheap deployment encourages disposable deployment. If nodes cost very little, the rational behavior for an operator is to abandon them in place rather than pay to recover them, and the drone-deployable temporary instrumentation I find so exciting is exactly the case where abandonment is easiest and least visible.

Defenses

Design for recovery, which mostly means designing for retrieval as explicitly as for deployment. If a drone can place a node it can retrieve one, and whether that happens is determined by whether anyone made it easy and whether anyone is accountable for it.

Energy harvesting wherever it is remotely feasible, because the battery is the dominant environmental term and eliminating it eliminates most of the problem along with the maintenance cost. This is one of the places where the commercial and environmental incentives align cleanly.

Modularity as a lifecycle strategy rather than only a scaling one. If the expensive, materially intensive part is a long-lived core and the part that gets replaced is a simple probe, the replacement cycle consumes far less. The architecture I advocate for other reasons happens to be the right one here, and I should be honest that this is a convenient coincidence rather than the original motivation.

And a genuine reckoning with the number. If a proposal implies a trillion devices, someone should do the material arithmetic before rather than after. I have not seen anyone in this industry do it seriously, including me, and it should be done.

VII. Seeing like a sensor

What measurement does to what we value

The last risk is the most abstract and possibly the most important, and it is not about the technology failing. It is about the technology working.

Measurement is not neutral. What gets measured gets managed, which also means that what does not get measured gets neglected, and that the choice of what to instrument is a choice about what will be considered real. Organizations optimize what appears on the dashboard. If the dashboard shows equipment efficiency and not the experience of the people in the building, the building will be run for the equipment.

This is a well-documented failure mode of quantification in general, and there is no reason to expect a dense sensing substrate to be exempt. If anything it is more susceptible, because the measurements will be numerous, precise, and continuous enough to feel comprehensive when they are merely abundant. Precision is very easily mistaken for completeness.

The version of this that concerns me most is in work. Instrumentation aimed at equipment can be redirected at people almost trivially, and the resulting metrics — task duration, path efficiency, idle time — are precise, comparable, and almost entirely disconnected from what makes someone good at maintaining complex systems. A technician who spends twenty minutes standing and listening before touching anything is doing the highest-value part of the job, and it looks like idle time in every metric a system like this would naturally produce. We should expect that to be optimized away by managers acting on data they trust more than they trust the technician.

There is also a dependency question. A world that runs on continuous measurement is a world that degrades in an unfamiliar way when measurement fails, and the more capable the instrumented version becomes, the less anyone retains the ability to operate the uninstrumented one. This is a general property of infrastructure and not a reason to avoid building it, but it does argue for graceful degradation as a first-class requirement rather than an afterthought.

Defenses

Measure the thing you care about, not the thing that is easy, and be suspicious when those diverge. This sounds trite and it is the single most common failure in applied instrumentation.

Preserve qualitative judgment explicitly. Systems should be designed so that a human assessment can be recorded alongside quantitative data with equal standing, because otherwise the quantitative record becomes the only record and the judgment disappears from the institutional memory.

Draw a hard line on worker monitoring, ideally before the capability exists rather than after it is deployed. I would like this to be an industry norm and I suspect it eventually has to be law, because the commercial pressure runs entirely one direction and individual firms declining to build it will simply be outcompeted by firms that do.

And build for graceful degradation, meaning the instrumented system should fail toward the manual one rather than toward paralysis. If losing the sensing network means nobody can operate the plant, the design was wrong.

VIII. The test we are actually taking

Writing this essay was less pleasant than writing its companion, which is probably a sign it needed writing. I want to close by saying clearly what I think the situation is.

None of the risks above are arguments against building this. They are arguments about how, and nearly all of them are decided by architecture rather than by intent. Whether the customer or the manufacturer holds the trust anchor. Whether raw data leaves the node by default or by request. Whether the system works when disconnected. Whether an action is architecturally required to leave a signed record. Whether the specification is open enough that several independent implementations exist. Whether the probe is designed to be retrieved.

Every one of those is a decision an engineer makes, usually early, usually without much ceremony, and usually without realizing that it is the decision that determines which world we get. That is the part I find sobering and also motivating: the leverage is enormous and it sits with people who are not currently being asked to think about it.

There is a version of this technology that makes people's lives quietly better — less waste, fewer failures, safer work, a physical world that can finally explain itself. There is another version that is a surveillance substrate owned by a handful of firms, running on unpatched hardware embedded in the walls, making unaccountable decisions about physical systems nobody understands anymore. The technical content of those two worlds is nearly identical. They differ in choices that are being made right now, in products that are shipping this year, by people who mostly have not read anything like this.

I am building in this space, which makes me an interested party, and you should weigh what I say accordingly. But being an interested party is also why I would rather have this conversation early than be defending choices later. The architecture is still soft. In fifteen years it will not be, and the installed base will make most of these questions academic.

I should also say what I think the likeliest bad outcome actually is, since it is none of the dramatic ones. It is not a catastrophic attack or an authoritarian sensing state. It is mundane enclosure: a built environment gradually filled with proprietary, cloud-dependent, unpatchable devices, each individually reasonable, collectively constituting a layer of infrastructure that nobody owns, nobody can audit, nobody can replace, and everybody depends on. That outcome requires no villain. It is simply what happens if every vendor optimizes normally and nobody decides otherwise. It is also, I think, the default.

The thing I would most like from anyone who reads this is disagreement of a specific kind. Not whether the risks are real, which seems clear enough, but which ones I have miscategorized, which defenses will not survive contact with commercial reality, and what I have missed entirely. That last category is the one that worries me, because it always turns out to be occupied.

We do not get many chances to think carefully about infrastructure before it exists. This is one of them, and the window is narrower than it looks.

Connecting the digital to the physical.

Contact

Links

wadesthomas1@gmail.com

+12155019211

© 2026. All rights reserved.