The System Is Not the Interface

A user presses “approve”. The screen dims for a fraction of a second, a check mark appears, and the decision seems to have happened inside that small rectangle. In the background, records may have changed, permissions may have taken effect, notifications may have been sent, a fin...

A button rarely triggers only one action

A user presses “approve”. The screen dims for a fraction of a second, a check mark appears, and the decision seems to have happened inside that small rectangle. In the background, records may have changed, permissions may have taken effect, notifications may have been sent, a financial obligation may have been created, another person’s workflow may have shifted, and an audit entry may have recorded who did what.

The interface is not necessarily deceiving anyone. Its job is to make the complexity required for use manageable. The problem begins when ease of use is mistaken for simplicity of the system.

The same mistake appears around public-service portals, corporate AI assistants, marketplaces and hospital booking tools. What is visible consists of fields, statuses and controls. What is operating consists of people, rules, data, contracts, permissions, exceptions and points of responsibility.

An interface is a compressed promise made by the system. It says that a certain action can be performed here and that a certain visible result should follow. It does not, by itself, establish that every role behind the screen is clear, every failure path is governed, or every decision can be reconstructed later.

A prototype proves one route

A successful demonstration is persuasive. A user enters a request, the system responds, a chart moves, a recommendation appears. The scene is complete. The prototype has genuinely shown something: under one configuration, with a known input and a controlled environment, the intended output was produced.

That is useful evidence. It answers a different question from durable operation.

A production system must survive incomplete data, concurrent use, abandoned sessions, repeated clicks, incorrect permissions, unavailable dependencies and cases that were not imagined during the demonstration. Prototypes usually display the successful route. Institutional services must also govern contested, failed and borderline cases.

Machine-learning components add further dependencies. Model behaviour can be shaped by data pipelines, preprocessing, surrounding code, serving infrastructure and feedback loops. Sculley and colleagues described this silently accumulating dependency structure as technical debt in machine-learning systems [4]. The impressive model is one small box inside a much larger diagram.

A demonstration is therefore development evidence. Operational, safety or institutional evidence requires additional work.

Permission is not a menu setting

On screen, permission often appears as a switch: administrator, editor, partner, customer. The label looks precise while compressing four separate questions: who may act, on what object, under which conditions, and who bears the consequence.

A marketplace administrator may be able to remove a profile. May that happen while a contract is active? Can private messages be inspected? Can a financial status be changed? May an automated outcome be overridden? Must a reason be recorded? Does the action leave an audit trail? Who reviews the dispute if the administrator is also involved?

These are not cosmetic design details. They expose the structure of institutional responsibility. Norman’s work on design emphasised that interfaces should make available actions and their effects understandable [1]. That is only the user-facing side. The other side must also establish whether an action is authorised, reviewed and reversible.

A role matrix is therefore not enough. The same role behaves differently during normal operation, an incident, a legal dispute or a data-access request. A system becomes more real when conditions, exceptions and responsibility chains are attached to the role names.

An AI response is not yet a decision system

A language model can summarise a complaint, locate possible risks in a contract or suggest the next step in a case. This makes it tempting to speak as though “the AI” were the system. In practice, the model is a component that receives an input, produces an output and carries characteristic patterns of error.

A decision system requires more. Who selected the input? What was omitted? Which model version ran? Under which instruction? Was there human review? What can be done when the output is wrong? Was the advisory status clear to the user? Can a log be retained in a form that supports review without spreading sensitive data unnecessarily?

The NIST AI Risk Management Framework separates governance, mapping, measurement and management activities rather than reducing risk to a single technical score [6]. The issue is not simply whether a model is “good”. It is how that model fits a particular organisation, purpose and field of consequence.

Research on sociotechnical systems points to the same difficulty. A clean, computable fragment can be extracted from a social situation while losing the conditions that give the decision its meaning. Selbst and colleagues discuss this through abstraction traps: a model may be internally consistent and still be inadequate for the setting into which it is introduced [5].

An AI output can therefore be evidence, advice or a trace in a review process. It becomes decision authority only when an institution attaches power, responsibility and a route of appeal to it.

Invisible human work is also a design decision

Many ‘automated’ services contain substantial human work. People resolve exceptions, repair data, moderate content, bridge incompatible processes or call the customer when the interface has reached its limit.

This can be entirely reasonable. The risk appears when the organisation itself no longer knows how much of the service depends on hidden manual labour. Capacity, cost and error rates then sit somewhere other than the management dashboard suggests. Adding one customer appears to scale an automated process while actually adding thirty minutes to three different people’s day.

Lucy Suchman’s work challenged the idea that technical action is merely the execution of a prewritten plan. In real situations, people interpret circumstances, adjust and improvise [2]. That flexibility need not be eliminated. It should be made visible, otherwise human correction disappears from the system’s account of its own cost, risk and knowledge.

The quality of automation appears in the clarity of its human roles: where people are needed, what they may see, what they may change and when authority returns to the governed process.

Failure may not originate where it appears

The interface displays an error. The user sees a failed action. The underlying cause may be corrupt data, an outdated business rule, a broken integration, an overly narrow permission or a conflict the design had no state for representing.

In complex systems, harmful outcomes often emerge from interactions among decisions that look acceptable in isolation. Leveson’s systems-safety approach therefore examines control and feedback structures alongside component failures [3]. The same perspective is useful for digital services. After locating the failed module, the investigation must also identify the constraint, signal or review that was absent from the wider process.

A ticket stating “the button does not work” is often too small. After the local repair, the button may work while the user still cannot proceed because the permission state and the contractual state remain inconsistent. The visible defect has gone; the system tension remains.

Symptoms, local technical causes and operating conditions should therefore be kept distinct.

Where does the system begin and end?

There is no single boundary that serves every investigation. For a developer, the system may be a codebase and its dependencies. For a customer, the brand, support channel and payment process form one system. A legal review may include controller, processor, contract and complaint route. A security analysis may extend further into suppliers and user workarounds.

The boundary is an analytical decision. It becomes defective when consequences relevant to the question repeatedly fall outside it.

An AI-based assessment tool can be measured as a model: accuracy, error patterns and latency. If it supports employee classification, the relevant system must also include who defines the objective, how the score is used, who can request review and what happens to a disputed case. The model boundary may be technically clean while the decision boundary is misleading.

A useful boundary makes the responsibility under examination visible. Different questions require different maps, but switching maps should not be presented as though the object had remained unchanged.

Documentation is part of the control structure

Documentation is often treated as something written after the product is complete. A system description is also a method of testing the system. When states, roles, data movements and failure routes cannot be explained clearly, they are often not fully governed in operation either.

A usable description does not need to force everything into one enormous diagram. Separate views may be needed for user journeys, responsibility, data, integrations, permissions and incidents. The relationships among those views, however, cannot remain assumptions.

The strongest sentences in technical documentation are often conditional. If this external service fails, this process stops. If this role is revoked, these active cases become unowned. If an AI recommendation is overridden, the original output and the reason are retained for this period. Such statements do not weaken a system. They reveal where an actual decision point exists.

A simple interface and an honest system

Users do not need to see every database, service and responsibility dispute. A good interface is usable precisely because it selects the information needed for the task at hand.

The organisation cannot afford the same blindness.

Internally, the construction of simplicity must remain visible: what is automated, where people intervene, what happens to exceptions, who may override, what evidence supports a decision, and where a process can be stopped or reversed.

An interface is honest when it does not promise more than the system can carry. A system is mature when background complexity does not become an excuse for blurred responsibility.

The check mark may remain simple. We still need to know what happened behind it.

References

[1] Norman, Don A. (2013): The Design of Everyday Things. Revised and Expanded Edition. Basic Books.

[2] Suchman, Lucy A. (2007): Human-Machine Reconfigurations: Plans and Situated Actions. 2nd edition. Cambridge University Press.

[3] Leveson, Nancy G. (2011): Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press.

[4] Sculley, D. et al. (2015): Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems 28, 2503-2511.

[5] Selbst, Andrew D. - Boyd, Danah - Friedler, Sorelle A. - Venkatasubramanian, Suresh - Vertesi, Janet (2019): Fairness and Abstraction in Sociotechnical Systems. Proceedings of FAT '19, 59-68. DOI: 10.1145/3287560.3287598.

[6] National Institute of Standards and Technology (2023): Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. DOI: 10.6028/NIST.AI.100-1.