Why architecture and governance functions need an accurate model of what AI systems actually are before they can be trusted, controlled, or blamed.
By Stuart Dee
There is a seductive habit, in boardrooms and server rooms alike, of speaking about large language models as though someone is home. We say a model “understands” a brief, “decides” on a course of action, “knows” when it is uncertain. The vocabulary of mind slips in unnoticed, and with it comes a set of assumptions that architecture and governance functions can ill afford that the system has intent, that it can be reasoned with, that its errors are lapses of judgement rather than features of its design.
They are not. At a high level, a large language model is a probabilistic engine given a sequence of tokens, it computes a distribution over what token is likely to come next, samples from that distribution, and repeats. The arithmetic underneath is deterministic (fixed weights, fixed maths), but the mechanism it drives, and the behaviour it produces, are probabilistic by design. There is no inner deliberation weighing options, no persistent self, evaluating whether an answer is true, no moment at which the system “decides” anything in the sense a person decides.
What looks like reasoning is the surface pattern of a function trained to produce plausible continuations of text, at extraordinary scale and with extraordinary fluency. The fluency is precisely what makes the illusion so durable: we are pattern-matching creatures ourselves, and a system that produces coherent, confident, well-formed prose triggers our instinct to attribute a mind behind it.
This matters enormously for architecture and governance because governance frameworks are built on assumptions about the entities they govern. Human decision-makers can be held accountable because they possess intent, can explain their reasoning, and can be expected to learn from correction in a way that generalises. None of that applies straightforwardly to a probabilistic engine. When a model produces a plausible sounding but false citation, that is not a lie in any meaningful sense, nor is it a bug in the colloquial sense of a mistake the system “knows” it made. It is the expected behaviour of a system optimised to produce statistically likely text, occasionally decoupled from truth. Calling this “hallucination” is itself a symptom of the same anthropomorphic drift – it borrows a term from human psychiatry to describe a mathematical property of sampling from a probability distribution.
The practical risk is that organisations design governance around the wrong model of the system. If leaders believe an AI assistant “understands” the compliance implications of a decision, they may under-specify the guardrails around it, trusting to a comprehension that does not exist. If architects treat a model’s outputs as reflecting stable internal knowledge, they may skip the verification steps that a purely statistical process demands. Sound governance starts from an accurate model of what is actually running in production: a function that maps inputs to probable outputs, shaped by training data and fine-tuning, with no ground truth, no persistent memory of its own reasoning, and no capacity to be genuinely surprised by its own conclusions.
This is not an argument for dismissing these systems, nor for stripping the language of all convenience: “the model suggests” is a harmless shorthand. It is an argument for architectural and governance discipline that does not quietly borrow the assumptions of human cognition when designing controls. Three implications follow directly.
First, accountability must sit with people and processes, not with the model. Every deployment needs a named owner responsible for outputs because the system itself cannot bear responsibility in any sense that governance can act upon.
Second, verification has to be structural rather than incidental. Because plausibility and accuracy are different axes for a probabilistic engine, and can diverge without warning, architectures should treat every generated output as a draft requiring independent checking wherever it touches a consequential decision, rather than as a considered judgement that merely needs occasional review.
Third, monitoring should track distributional behaviour, not intent. Since the system has no intent to audit, the meaningful governance questions are statistical: how does output quality shift as inputs drift from the training distribution, where does confidence miscalibrate against accuracy, and which classes of query reliably produce degraded outputs. These are engineering and data questions, not questions about the state of a mind, and they demand different tooling and different expertise than traditional conduct-based governance.
There is an architectural analogy worth drawing out. Enterprise architects have long known how to design around unreliable components: a flaky network link is wrapped in retries and circuit breakers, not trusted to always deliver. A probabilistic engine deserves the same treatment. It should sit inside a decision architecture with deterministic checks either side of it – validation on the way in, verification on the way out – and clear boundaries around where its output is permitted to act autonomously versus where it must be reviewed before it touches anything consequential. Treating the model as a component to be engineered around, rather than a colleague to be consulted, is not a diminishment of the technology. It is simply the correct category for it.
None of this is to say consciousness is settled science, or that it never will be relevant. But architecture and governance functions do not need to resolve the philosophy of mind to do their job well; they need to resolve what is actually computing the outputs their organisation relies upon. The honest description is unglamorous: a very large, very capable statistical engine, trained to continue text in ways humans tend to find useful and convincing. Building governance on that description, rather than on the comforting fiction of a mind at work, is the difference between controls that hold under pressure and controls that quietly assume a competence nobody has ever demonstrated. The illusion of consciousness is not dangerous because it is philosophically wrong. It is dangerous because it is operationally misleading, and operational accuracy is precisely what governance exists to protect.