Field Note #10 ∷ On Trusting Trust, Revisited
"Such blatant code would not go undetected for long. Even the most casual perusal of the source of the C compiler would raise suspicions."
— Ken Thompson, 1984
I. A lecture worth rereading¶
In 1984, Ken Thompson — co-creator of Unix, recipient of the Turing Award — gave an acceptance speech that is now one of the most cited and least carefully read lectures in computing.
It is short. You can read it in under ten minutes. Most people who cite it clearly have not.
The lecture is usually summarised as a clever proof: you can hide a backdoor inside a compiler, and that backdoor can survive even if the compiler's source code is inspected and rebuilt from scratch.
The cleverness is real. It is also a distraction.
Thompson's actual point is not the trick. It is the structural observation underneath it.
Everybody remembers the backdoored compiler. Almost no one remembers what Thompson said about paycheques.
Consider the line above. Such blatant code would not go undetected for long. Even the most casual perusal of the source of the C compiler would raise suspicions. This is the sentence that does the work. The demonstration that follows — the self-reproducing compiler, the backdoor that survives recompilation — simply shows that the observation is true.
The observation is this: the obvious version of any attack would be caught. The successful version operates at a layer casual perusal does not reach.
That layer always exists.
And that is the point.
II. What Thompson actually said¶
Thompson's thesis, stated plainly: you can't trust code that you did not totally create yourself.
The parenthetical that follows, which most readers forget: (especially code from companies that employ people like me).
Read that twice.
Thompson is not warning about a hypothetical adversary outside the system. He is pointing out that the people building the trusted tools are themselves part of the world in which adversaries exist.
The threat is not at the perimeter. It is in the supply chain of trust.
The compiler that compiles your compiler was written by someone. That someone works for an organisation. That organisation exists within a set of incentives, constraints, and interests that change over time. That organisation exists within the confines of a nation-state. Every layer of the trust chain was authored by someone, configured by someone, and ultimately depends on hardware that has not been fully verified.✦
Thompson's lecture is not telling you to be paranoid about your tools.
It is telling you that the trust commitment was always there — that you have been making it your whole career — and that the question is whether you have made it visible enough to defend.
III. The example that arrived forty-one years late¶
In April 2026, researchers at SentinelLabs published an analysis of a piece of malware they call fast16.❈
The framework was compiled in 2005. It targets a narrow class of high-precision engineering and scientific simulation programs — the tools used to model car crashes, structural loads, fluid dynamics, and, on a plausible reading of the evidence, nuclear weapons physics modelling.
It does not destroy the programs. It does not steal their data.
It modifies the floating-point calculations they perform, in memory, as they run.
The outputs look like real outputs. Engineers using the affected systems would have no reason to suspect that anything was wrong.
What matters is not the sophistication of the code.
What matters is the layer it operates on.
Defenders were looking at files on disk, network traffic, and known-bad signatures. fast16 was none of those things. It intercepted the moment a program was loaded into memory, recognised a narrow set of targets, and substituted slightly different mathematics into the running system.
A sample sat on a public malware-analysis site for nearly a decade before anyone understood what it was.
A large part of the problem is how frameworks like MITRE ATT&CK are used in practice. They train analysts to map observations to known categories. What falls outside those categories is easily treated as noise.
You cannot ask questions you do not yet have a frame to imagine.
It was not hidden.
It was below the threshold of what defenders thought to look at.
This is Thompson's lecture, made concrete.
A note from the author (2026-05-01)¶
Nobody asked me to write this. I spotted Vitaly Kamluk's report on LinkedIn over coffee on a Monday morning and went down the rabbit hole. The piece you are reading is the result. It is the first public CTI writing I have done in years — I broke a long-standing silence as a trained analyst because I think this category of attack is structurally larger than HAFNIUM, not necessarily in immediate operational impact, but in what it reveals about a class of capability that has been present for at least two decades and was invisible to the analytical field until this week.
I have no particular geopolitical dog in this fight. I am, however, firmly on the side of defending science. No one should be using this technique — it undermines the fundamental trust layer that upholds the availability of the scientific method itself. It is naïve to assume the technique circulated for twenty-one years unobserved by anyone, which means the wiser working assumption is that the Golden Rule of Cyberwarfare has been quietly in operation: do unto others as they have done unto you. If multiple capable actors have spent the past two decades subtly corrupting one another's foundational science and engineering calculations, this needs to be surfaced and dealt with. That is a hard problem. It is also, now, everyone's problem.
This analysis was not commissioned and no party had input into it.
IV. The map and the territory¶
There is a softer version of this problem, and it is worth naming, because most working analysts encounter it long before they encounter anything as exotic as fast16.
Modern security operations are organised around shared frameworks for describing adversary behaviour. The most widely used of these, MITRE ATT&CK, is a careful, openly maintained catalogue of tactics and techniques observed in real-world attacks. It has done enormous good. It gives analysts a common vocabulary, makes detection engineering tractable, and lets defenders compare notes across organisations.
The framework was built as a shared description of what has been observed.
It was not built as an exhaustive ontology of what can happen.
This distinction is small on paper and load-bearing in practice.
A generation of analysts has now been trained to think inside the framework — to map every observation onto a known technique, to treat un-mappable observations as noise, analyst error, or low-priority anomaly.
The framework trains a parser that recognises only what it has been trained to see.
The framework becomes the imagination. The imagination becomes the limit.
This is not a flaw in the framework. It is the limit any sufficiently useful map carries with it.
The categories that are not on the map are precisely the categories where the next fast16 will operate.
☞ An honest reflection that came up while drafting this Field Note. I have been giving security advice to organisations small and large for more than twenty years. The shape of that advice has changed across the period — the tools, the threats, the regulatory environment — but one assumption ran through all of it: that nation-state-grade capability generally stayed where nation-states aimed it, and that the security needs of ordinary organisations could be reasoned about separately. Reading the SentinelLabs analysis, I am not sure that assumption was ever true in fact. fast16 was not specifically aimed at any of my clients, to my awareness. But the category it represents was already loose in the world, shaping what security meant, before any of us had a name for it. I do not think this category has a name yet. It needs one. I have been calling it "epistemic-layer attacks" (or "Sophon attacks") in my own notes, for what that's worth.
V. The practical problem¶
Most readings of Thompson stop at the philosophical conclusion: therefore you can't fully trust anything.
This is true.
It is also useless as a working stance.
The useful question is different.
If trust commitments cannot be eliminated, can they at least be made visible?
Can you know which layers of your computational stack you have decided to trust, who you are trusting at each layer, and what conditions would make those commitments worth revisiting?
This is the discipline.
It is the same discipline Gödel forced on mathematics in 1931, when he showed that no sufficiently powerful formal system can certify its own foundations.
Gödel's work did not make mathematics impossible. It made it honest about its limits.
Thompson's lecture does the same thing for computational trust.
It does not make trust impossible.
It asks that trust be honest about its dependencies.
The fast16 case shows what happens when that discipline is absent. The trust commitments were invisible. The defenders did not know which layers they had implicitly chosen to trust. The adversary did.
For twenty years, the adversary's view of the trust structure was clearer than the defender's.
That is the asymmetry.
☞ In principle, subscription-based software ecosystems with automated update mechanisms make untraceable, per-user, per-file manipulation trivially achievable in ways that are essentially unobservable. Ken Thompson would have to analyse the entire source of Excel each time he ran it. The static-binary world had bounded trust commitments. You compiled once, you trusted the toolchain, the artefact on disk stayed where you put it. fast16 sat on a public analysis site for nearly a decade because it was static — there was a sample to collect, eventually share, and eventually decode. The subscription model removes the artefact. The binary that ran for one user, on one machine, for one calculation, may never exist anywhere else. There is no sample to upload. There is no public site. There is the moment of execution, and then there is nothing. This is the infrastructure our epistemic layer is already running on.
VI. A working heuristic¶
If there is one thing to take from Thompson's lecture, applied to ordinary working life, it is this:
For every important conclusion you reach through a computational tool, ask: which layers of the stack did I decide to trust, and why?
You will not be able to verify those layers.
That is not the point.
The point is that knowing where your trust is placed is the precondition for noticing when something has changed — when a vendor is acquired, a maintainer leaves, a dependency shifts, or a layer that was previously below your threshold of attention becomes worth examining.
There is a harder version of the same heuristic:
Which layers did I not think to enumerate at all? What would tell me they were there?
This question cannot be answered from inside the working frame alone. It requires reading beyond it — old papers, adjacent disciplines, and the tradition of demonstrations that exist precisely to expand what the field considers possible.
This is not paranoia.
Paranoia assumes hidden hostility everywhere.
This discipline assumes hidden dependencies somewhere — and asks you to find them.
A third note — on multioccupancy and the failure of causal modelling under concurrent substrate compromise (2026-05-10)¶
fast16's most operationally significant property is one the SentinelLabs analysis names but does not fully develop: the framework discriminated per-target. Most callers of the affected floating-point operations received correct mathematics. A narrow set of targeted callers received subtly modified mathematics. This is the property that matters more than the floating-point modification itself.
Imagine an adversary — call them Larry — who has modelled a specific perturbation to a target system nominally owned by the proprietor. Larry has done the work. Larry knows that a particular floating-point calculation, shifted from .9999 to .9998 in a stress simulation, will mask a fatigue point in a critical structure, or shift a yield estimate, or alter an investment decision. Larry's model of the substrate's compromised state is coherent with Larry's operational intent. He knows what he is doing.
Now consider that Larry is not alone in the proprietor's system. A second adversary — Curly — has also achieved a substrate-layer foothold in the same infrastructure, perhaps through entirely different means, perhaps targeting entirely different calculations, perhaps not even aware that Larry is present. Curly is also running his own perturbation. Curly has also modelled what his perturbation buys him. A third adversary — Moe — may be present as well, having arrived through yet another route, running yet another perturbation, similarly uncoordinated with the other two.
None of the three has audited or modelled the others' playbooks. None of them has modelled the real-world impact of their concurrent operations on the proprietor.
The actual behavior of the target system is the composite of all three perturbations operating concurrently. From the proprietor's perspective, the system produces output that does not match any single adversary's playbook. From Larry's perspective, his perturbation is not producing the operational outcome he modelled — but Larry cannot tell whether the discrepancy is environmental noise, defender countermeasures, or another adversary he cannot see. From Curly's perspective, the same. From Moe's perspective, the same. No party in the proprietor's system — least of all the proprietor — has a model of what the system is actually doing. The target is suffering damage that no single adversary intended. The adversaries are receiving operational signals they cannot reliably interpret.
Substrate-trust commitments are not bilateral commitments between a single occupant and a trusted substrate. They are multioccupancy commitments — implicit guarantees that the substrate behaves coherently in conditions where multiple parties, authorised and unauthorised, share it. fast16 demonstrated that this coherence can be falsified by a single adversary on a per-target basis. The harder structural fact, which the public record on adversary co-occupancy strongly suggests is already operationally common, is that multiple uncoordinated adversaries operating concurrent substrate-layer attacks against the same target produce emergent behaviour that no participant has the information to model — including the attackers.
This is the regime in which causal modelling fails for all participants. The defender cannot detect from inside. The attackers cannot detect each other from inside. The system's actual behaviour is determined by the composite of multiple uncoordinated adversarial intents that did not represent each other. The output is not what any single adversary modelled. It is not what the defender expected. It is the emergent product of operational intents that did not know about each other.
The implication for defenders is paradoxical: in a regime of widespread substrate compromise by uncoordinated adversaries, the most important diagnostic signal is whether the system is behaving incoherently — producing outputs that match no single coherent threat model, because the actual cause may be the chaotic composition of multiple incoherent threat actors running ops simultaneously.
The discipline this Field Note advocates — make trust commitments visible — must therefore include the multioccupancy commitment explicitly, and must extend to the multi-adversary case. The substrate is not merely falsifiable; it is falsifiable concurrently by multiple uncoordinated parties, and the resulting behaviour is unpredictable from any given participant's reference frame. This is the operational condition we must now defend systems under.
VII. The harder version¶
There is a harder version of this argument still.
The trust commitments we make in our computational tools are not separate from the trust commitments we make in our institutions.
Every layer of the computational stack is embedded in a social and political context.
The compiler was written by someone, employed by an organisation, operating under a legal regime, within a set of incentives and constraints that evolve over time.
The operating system, the libraries, the simulation tools, the frameworks we use to describe attacks — each layer reflects not just technical decisions, but the conditions under which those decisions were made.
Thompson's lecture, read carefully, does not stop at code.
It is a claim about the structure of trust in any complex system.
This does not mean institutions are untrustworthy.
It means the trust we place in them is of the same kind Thompson described: chained, ungroundable, and worth making visible.
The wisdom is not in eliminating that trust.
The wisdom is in knowing it is there.
VIII. Where this leaves us¶
Limits are structural.
You cannot verify your way out of them.
You can only locate them, name them, and work honestly within them.
This is true for mathematical systems, as Gödel showed.
It is true for meaning and interpretation, as earlier Field Notes have argued.
It is true for the computational tools we use to think, as Thompson showed in 1984 — and as fast16 has now made concrete.
The discipline is the same in each case.
Make the trust commitments visible.
Read the trusted layers carefully when you can.
Remember that the list of layers you can name is not the list of layers that exist.
Notice when conditions change.
A fourth note — on AI-mediated development cost-reduction and noticing when conditions change (2026-05-22)¶
Symantec / Albright / Zetter (May 2026) have now confirmed attribution and targeting in the public record. The analytical implications about category-of-attack cost-reduction in the age of LLM-assisted development remain unsurfaced. Fast16 in 2005 required state-level resources because the analytical work — reverse-engineering target simulation software, understanding the physics, identifying the specific intervention point — was expensive. LLM-assisted development collapses each of these cost components. The category-of-attack surface widens to actors who could not have executed Fast16-class operations in 2005.
The inverse move is also worth registering. The same disclosure incentivises target states to migrate away from commercial simulation software toward bespoke vibe-coded implementations. The attacker’s prior-knowledge advantage — predicated on the predictability of commercial software versions, interfaces, and intervention points — collapses against bespoke targets. The defender’s substrate-trust commitments do not disappear; they migrate, from the commercial-software layer to the LLM-training, inference, and code-review layers. The verification problem changes character: known attack surface is traded for unknown verification surface. The strategic landscape becomes more complex for attackers and defenders alike. The implications run in multiple directions and warrant careful consideration.
Treat conclusions reached under invisible trust as provisional.
Build systems — and institutions — that can hold these limits without pretending to eliminate them.
None of this is comfortable.
None of it is supposed to be.
But it is the work.
And it has been the work for a long time.
The only question is whether we choose to do it consciously, or continue doing it accidentally.
A second note — on the scope of the question (2026-05-01, revised 2026-05-10)¶
[Editorial note: An earlier version of this aside enumerated a list of domains plausibly affected by the structural vulnerability fast16 demonstrates. On reflection, I have replaced the list with the structural argument that animated it. The list was doing rhetorical work the argument did not require, and risked being read out of context as a catalogue of likely compromise rather than as a prompt for honest threat modelling. The earlier version remains in the git history for anyone interested in the editorial trajectory. I would rather be slightly less vivid than slightly misread.]
Sitting with the SentinelLabs analysis for several days, an obvious question surfaces and I want to put it on the record while it is still uncomfortable to ask.
fast16 demonstrated a deployed capability for kernel-level, target-software-recognising, in-memory tampering of floating-point arithmetic, with output indistinguishable from clean computation by downstream consumers, persisting for years undetected. The published analysis names three target software families in engineering simulation. The capability itself is not domain-specific. It would operate, in principle, against any software that performs trusted numerical computation feeding downstream decisions on platforms where kernel-level driver installation is achievable.
The structural property worth naming is this: trusted numerical output, no independent verification against ground truth, and downstream consumption by systems that propagate the output forward into decisions. Wherever that property holds, the substrate is in principle falsifiable in the manner fast16 demonstrated. The reader working in any technical discipline can supply their own assessment of where this property holds in their own working environment. That assessment is more useful coming from inside the domain than from me.
It is worth distinguishing two questions that the structural argument tends to collapse. The first is whether a domain is structurally vulnerable to this class of capability. The second is whether deployment against that domain is operationally attractive to a capable adversary. The two are not the same. A domain may be structurally vulnerable and operationally unattractive — wide attack surface, many independent eyes on the output, low per-incident value, frequent reconciliation against external sources. Another domain may share the same structural vulnerability and be operationally attractive for opposite reasons — narrow target, high-value output, low cross-checking, downstream propagation into decisions that cannot easily be unmade. Structural vulnerability is therefore a necessary but not sufficient condition for concern. The sufficient condition is the conjunction of structure and deployment economics, and that conjunction is domain-specific.
The load-bearing claim is narrower, and harder to dismiss. fast16 was named in the 2017 ShadowBrokers leak. The framework itself was compiled in 2005. It is not credible that a capability of this class, sitting in a public sample database for nearly a decade and referenced in one of the most carefully studied intelligence leaks of the modern era, went unexamined by every nation-state with serious reverse engineering capability across a twenty-one-year window. The wiser working assumption is that capable adversaries have had equivalent or better capability for some time, and that what SentinelLabs has now made public is one specimen of a class. The Golden Rule of Cyberwarfare applies: do unto others as they have done unto you, and assume they have done it first. The question is therefore not whether this category of attack has been deployed elsewhere. It is whether the engineering and analytical disciplines that depend on trusted numerical substrates have caught up to a class of capability that has been quietly available to capable adversaries for two decades.
We have no way to bound the answer from inside the affected systems. Capability of this class is designed to be undetectable from within. The historical record of computations performed on potentially affected platforms across the last two decades cannot be cleanly verified retrospectively. Whatever happened, happened.
What can be done now is to design forward — to build verification, redundancy, and external attestation into the computational layers that downstream systems depend on, with the explicit working assumption that the computational layer may have been compromised in the past and may be again in the future. This is not paranoia. It is the responsible engineering response to a confirmed capability whose deployment scope cannot now be determined. Researchers in any domain that depends on trusted numerical computation may also benefit from revisiting past cases of unexplained anomaly with this class of capability in their threat model. Many anomalies that have been attributed to model error, measurement uncertainty, or implementation bugs are now plausibly worth reconsidering. We have less time than we would have had with earlier disclosure. The work of incorporating this into engineering practice is therefore more urgent than it might otherwise have been.
One further observation worth registering. Formal verification — the discipline of proving that implementations correctly satisfy their specifications — would not have prevented fast16. Neither would a purist LANGSEC approach. The compromise occurred at a layer below where these disciplines operate. The verified simulation software was running its specified algorithm correctly; the parsers were parsing correctly; the implementations were correct against their specifications. fast16 tampered with the execution substrate — the floating-point operations the verified software was specified to call — and modified the values the operations returned. The proofs remain valid. The proofs simply no longer say what they appeared to say, because the trust commitment they implicitly depended on (that the underlying hardware-and-OS layer is honest) has been demonstrated to be falsifiable in deployment. This does not make formal methods useless. It locates precisely where their sufficiency ends.
I am not claiming any specific deployment beyond the one SentinelLabs documented. I am noting that the structural class is broad, the adversary capability is almost certainly distributed, and the question of scope is now an open question that public engagement is the necessary first step toward addressing — regardless of what the answer turns out to be.
— Trey
✦ The hardware layer is the one most readers will instinctively skip past as solid ground. It is not. The silicon was designed by someone, fabricated in a facility located somewhere, packaged through a supply chain that crosses jurisdictions, and shipped through logistics networks whose integrity assumptions are also load-bearing and also unverified. The 2038 mitigation problem will eventually require replacing a great deal of embedded silicon at scale, and the question of where that silicon comes from — and what trust commitments come with it — is one of the load-bearing assumptions of long-horizon timing-resilience work. A future Field Note will examine this properly. ↩
❈ The original SentinelLabs analysis is published at sentinelone.com/labs under the title "fast16 | Mystery ShadowBrokers Reference Reveals High-Precision Software Sabotage 5 Years Before Stuxnet" by Vitaly Kamluk and Juan Andrés Guerrero-Saade, dated 23 April 2026. The piece is technically dense but worth reading in full. The framework was first surfaced through a deconfliction note in the 2017 ShadowBrokers leak of NSA tooling, where the operator instruction read: "fast16 *** Nothing to see here – carry on ***". The note was correct on its own terms. Nothing to see — until someone bothered to look at a layer the analytical tradition had not been trained to examine. ↩
If you received this via email, the canonical archive version lives here: propertools.be/fieldnotes/field-note-10-on-trusting-trust-revisited/