The Egoist Machine
The Egoist Machine: a manifesto for machines that represent individuals
Computers have historically functioned primarily as tools through which individuals directly exercised their own agency. In 1843, Ada Lovelace wrote of Babbage's Analytical Engine that it had "no pretensions whatsoever to originate anything" because its function was to carry out operations that humans knew how to specify:
- A person would specify an action,
- A computer would execute that action.
In this flow, responsibility for deciding what should happen remains largely with the person operating the computer.
JCR Licklider's 1960 Man-Computer Symbiosis imagined a considerably richer partnership between people and computers, but still allocated the central source of purpose to the human, writing that humans would "set the goals, formulate the hypotheses, determine the criteria", while computers performed the routinisable work to support those goals. He explicitely wanted computers to participate in formulating problems and making decisions, while nevertheless retaining a clear distinction between human purposes and computational assistance.
Douglas Engelbart, in Augmenting Human Intellect, reminds us that one of the foundational ambitions of personal computing was augmentation rather than pure automation: increasingly the capabilities of the person operating the machine.
Artificial Intelligence brings forth a new change because machines are beginning to maintain personal context about individuals, infer intentions without the need for them to be directly specified, independently formulate and evaluate plans for achieveing those intentions, and take actions without requiring intermediate decision-making by the user.
The progression is therefore approximately:
Tools → Systems that predict and recommend → Systems that understand and act
This should be understood as a progression in the dominant relationship between people and machines, and is not a strict technological chronology. What is changing at present is that capabilities previous distributed across specialised systems are being combined in general-purpose agents. The significance of this decision is that humans are beginning to delegate agency to machines. This is a neutral statement, void of criticism or support; just a fact that, for example, a search engine provides information from which a person makes a decision, while an agent can increasingly make parts of that decision (and, ultimately, the whole decision) itself.
Once a machine possesses delegated agency, the question of representation falls out of the quality of the machine's intelligence no longer being the only relevant consideration. What now concurrently matters is whose interests govern the intelligence and authority that the machine exercises.
The Representation Problem
An agent is an entity authorised to act on behalf of a principal. This relationship naturally lends itself to questions concerning:
- whose objectives the agent pursues,
- what information the agent possesses,
- what authority has been delegated to it,
- what other interests might influence its behaviour,
- how the principal can evaluate its actions,
- whether the principal can replace it.
Economics and law are foundationally rooted in the examination of the principal-agent problem. Jensen and Meckling's canonical formulation of an agency relationship describes a principal engaging another party to perform a service on their behalf while delegating some decision-making authority, after which the central economic problem becomes what happens when the interests of the principal and the agent diverge, and when monitoring the agent is costly or incomplete.
You can see, and we shall develop, the parallels of this structure when applied to AI agents, but we shall start instead with a statement worth bearing in mind, even if you read no further:
The fact that an AI system interacts directly with a person does not establish that person as its principal. Representation depends on the underlying governance of the relationship, rather than on the outward projection of personalisation.
Think about it:
- → A machine can be used by an individual without representing them
- → A machine can be personalised to an individual without representing them
- → A machine can know an enormous amount about an individual without representing them
- → A machine can perform actions for an individual without being governed by them
This distinction becomes increasingly consequential as machines become capable of acting. A system the recommends something while pursuing an external objective will influence a person's choices (though do not take this as negating the negative implications of such scenarios), whereas a system that acts while pursuing an external objective can exercise the person's delegated authority according to interests that are far from their own.
The Web
The internet, as we know it, organises personal context primarily around applications and institutions, and does not do so on the level of the individual at anywhere near the same scale. Different organisations maintain separate representations of the same person according to the interactions that occur within their respective systems.
This arrangement is not incidental to the development of the modern web because, if we go back to Tim O'Reilly's dedscription of Web 2.0, control over distinctive and difficult-to-recreate datasets was explicitly identified as a source of competitive advantage. The subsequent development of platform businesses demonstrated how accumulated behavioural data could improve consumer technology offerings.
The basic economic loop would not be out of place in Econ101:
interaction produces information about the user → that information improves the service's model of the user → the imporved model increases the value of continued interaction with the same service
This creates an asymmetry in which institutions possess increasingly sophisticated machines for understanding individuals in order to pursue institutional objectives, while the individual generally cannot take an equivalent accumulated understanding and make it immediately useable by another service.
There is already a substantial intellectual and technical tradition attempting to reverse this asymmetry, including personal information management systems, Project VRM, MyData, self-sovereign identity, local-first software, and Tim Berners-Lee’s Solid project. Vannevar Bush’s much earlier 1945 proposal for the memex is also relevant because he imagined a persistent personal information environment organized around an individual’s associative memory rather than around institutional databases.
For us at Egoist Machines, our idea is therefore not that user-controlled data or portable persoanl context are new ideas (because, quite clearly, they are not); the new idea is that AI changes the stakes of these older ideas because accumulated personal context is becoming the informational basis for machines capable of exercising agency.
The Egoist Machine
An Egoist Machine is an intelligent system whose memory, incentives, and actions are governed by the individual it represents.
The term does not imply that the machine posseses an ego in the colloquial sense, nor does it imply that the machine should pursue the individual's immediate desires without limitation. The egoism describes the orientation of the relationship: the machine has an identifiable principal whose interests and intentions govern the relationship.
This places the idea within a much older tradition of agency rather than within philosophical ethical egoism. In agency law, an agent is not permitted to do absolutely anything its principal requests; the agent-principal relationship establishes representation, authority, duties, and boundaries, with fiduciary duties like loyalty exist because the agent may possess interests or opportunities that diverge from those of the principal.
Remember: loyalty does not mean unlimited obedience. Loyalty means that the representative does not covertly substitute another party's interests for those of the party it represents.
An Egoist Machine can (and must!) operate within legal constraints, respect the rights and autonomy of other people, refuse harmful actions, acknowledge uncertainty, and operate according to boundaries that the individual cannot override. The definition of the Egoist Machine depends on (1) memory, (2) incentives, and (3) actions, because:
- Memory determines what the machine knows about the individual
- Incentives determine what the machine attempts to accomplish using that knowledge
- Actions determine what authority the machine can exercise as a consequence
An Egoist Machine requires governance across all three.
Memory
A machine cannot meaningfully represent an individual if every interaction begins with a blank slate of context. Effective representation increasingly depends on persistent understanding that may include explicitely stated content, and inferences developed through previous interactions.
The idea that computing could become an extension of personal memory considerably predates the current AI discourse. Going back to memex, this was conceived as a device in which an individual could build persistent associative trails. The difference introduced by contemporary AI is that memory can increasingly be interpreted rather than just being retrieved, allowinf heterogenous traces of a person's life to be transformed into future decisions.
Personal data alone is not a useful object, though it is monetisable. AI systems can transform collections of individual facts and interactions into accumulated understanding, which can be defined as a useable model that affects how future information is interpreted and how to take actions that are more likely than not to advance the individual's objectives.
It is interesting that this distinction between raw data and derived understanding is both legally and technically relevant. Older data-protection regimes establish rights involving access, correction, deletion, purpose limitation, data minimisation, and portability, but explicit portability rights such as GDPR Article 20 are principally formulated around personal data provided by the individual in a structured and machine-readable form (while regulatory guidance has historically distinguisged such data from some inferred or derived information created by the service).
The emerging object of value in consumer AI is exactly this derived layer - those inferences that make an assistant feel as though it actually understands someone.
The question then becomes: who governs that understanding?
If an institution accumulates a progressively richer understanding of an individual, but that understanding cannot meaningfully leave the institution, personalisation compounds primarily as an institutional asset. The longer the individual remains within the institution, the better that instituion becomes at serving the individual, at the cost of interoperability: moving to another provider of the same service means sacrificing some portion of the accumulated relationship.
In an Egoist Machine, what the machine learns about the individual accumulates principally for the individual. This requires the individual to possess meaningful governance over the resulting memory, including the ability to inspect significant information and inferences, correct inaccurate representations, understand where consequential information originated, establish boundaries of privacy, and determine which parts of their accumulated context may be disclosed to another system.
There is already substantial precedent for each of these requirements:
- Data-protection law (in some regions) provides rights of access, rectification, and erasure
- Privacy engineering has developed principles of purpose limitation and data minimisation
Portability then follows directly. If an individual changes the model, application, interface, or company through which they access intelligence, the relevant accumulated understanding should be capable of surviving that transition ratehr than being inseparable from the provider that happened to develop it.
This definition does not require all personal information to exist in a single database. User governance and technical centralisation are separate questions. Context could be stored locally, encrypted remotely, distributed across multiple services, or implemented through architectures that do not yet exist. What matters to the definition is purely that governance follows the individual rather than the application.
An important note here is to take heed of Daniel Solove's critique of privacy self-management: notice-and-consent systems can place an unrealistic decision-making burden on individuals confronted with enormous numbers of complex and interdependent privacy decisions, meaning that an Egoist architecture would need to make policies governable by the individual without requiring the individual to become the full-time administrator of their own data infrastructure.
Incentives
Knowing an individual is different from representing their objectives. A system can possess an extraordinarily accurate model of someone's preferences while using that model to optimise for an objective selected by somebody else.
This distinction is well-documented already in recommender-system research. Google had an influential 2016 paper describing YouTube's recommendation architecture which explained that ranking was optimised around expected watch time rather than through some prediction of which video a person would explicitly say they preferred. This is a handy illustration of the fact that personalisation systems combine a model of the user with an objective selected by the system designer.
Many other existing personalisation systems combine predictions about individual behaviour with institutional objectives. These objectives can overlap with the interests of the individual, and often do, but they are not equivalent to them.
Leaving computing for a moment, Herbert Simon had the interesting observation that an abundance of information consumes the scarce resource of human attention, which means that systems competing successfully for attention need not necessarily be systems advancing the reflective objectives of the person whose attention they capture. Behavioural economics and decision theory similarly provide reasons not to equate observed behaviour with welfare or intention, because people can act from habit, temptation, limited information, social pressure, or short-term preference, while simultaneously holding longer-term goals that conflict with those actions.
An Egoist Machine requires the individual to have meaningful givernance over what the machine is attempting to accomplish on their behalf.
Why meaningful governance? Because observed preference and intended objective are not the same thing. For example, a person can:
- enjoy consuming entertaining content while wanting to reduce the amount of time they spend consuming it
- repeatedly purcahse expensive products while wanting to save more money
- habitually consume arguments that reinforce existing beliefs while wanting to encounter serious opposing views
- consistently choose convenience while deciding that future purchases should prioritise durability
And other such, very ineffiently human, examples.
What we learn from this is that historical behaviour provides evidence about a person, but it does not necessarily define what that person wants their future behaviour to become.
So, a personalised machine primarily needs to answer the question "What is this person likely to choose?". An Egoist Machine must additionally be capable of answering "What has this person decided they are trying to achieve?".
When external economic incentives enter the relationship, the gulf between the personalised and the Egoist machine becomes more visible. Structures like advertising, commissions, proprietary ecosystems, and other forms of compensation can create objectives that diverge from those of the individual. An Egoist Machine does not, of course, require the elimination of commerce, but material conflicts affecting its representation of the individual must not covertly supersede the objectives that the individual has established.
Returning to the idea of the fiduciary duty, the law around fiduciaries does not assume that conflicts of interest can always be eliminated, however it treats loyalty, disclosure, consent, and conflict management as fundamental to the relationships in which one party is entrusted to act for another. The analogous question for an AI representative /agent is therefore: can the machine simultaneously claim to represent the individual while allowing undisclosed commercial interests to alter the decisions it makes on that individual's behalf?
Actions
Understanding an individual does not automatically confer authority to act for them. Memory and delegated authority are separate, and an Egoist Machine requires governance of both, as previously stated.
As AI systems gain access to the broad ecosystem of pre-existing applications that consumers use on a day-to-day basis, their actions increasingly exercise authority that originates with the individual. This requires such systems to be concerned with what the individual has authorised the system to undertake, not simply the current default of whether the system can do the thing.
The principle of least privilege holds that a system should receive only the authority required to accomplish its task. The agentic version of the same principle reads that delegations should transfer the minimum necessary authority rather than converting requests into justification for unrestricted access to the individual's accounts and resources.
Authority can vary according to:
- the type of action
- the monetary value of the action
- the duration of the action
- the counterparty(s) to the action
- the sensitivity of the action
- the reversibility of the action
- the confidence in the action
- the consequences of error in performing the action
Greater personalisation should therefore not automatically imply greater authority, and greater knowledge should not automatically imply greater disclosure. Put simply: a machine can know something without being authorised to reveal it, and it can understand what an individual wants without being authorised to act independently to obtain it.
The underlying problem to solve is that, when the human is no longer present at every individual action, the system needs a durable representation of the boundary of delegated authority.
The Interaction of (1), (2), and (3)
- A machine with individual-governed memory but externally governed incentives could possess a portable and accurate understanding of the individual while still using that understanding to optimise primarily for the commercial interests of another party.
- A machine with individual-governed incentives but externally governed memory could sincerely attempt to pursue the individual's objectives while leaving the accumulated understanding underlying that relationship captive to a single provider, therefore making the relationship difficult to transfer or replace.
- A machine with individual-governed memory but externally governed actions coudl understand the individual and pursue the correct objectives while exercising authority according to boundaries that the individual cannot meaningfully establish or revoke.
So, memory determines the informational basis from which representation occurs, incentives determine the objectives toward which that information is applied, and actions determine the authority through which those obkectives affect the external world. Individual governance of only one or two components therefore does not produce complete representation.
Limits
Individual governance, of course, creates its own risks, particularly because a sufficently rich representation of a person could become an unusually valuable security target.
Individual representation also cannot imply unrestricted pursuit of individual interests. People exist among other people whose rights and interests are equally legitimate, meaning that an Egoist Machine represents one principal within legal and social systems rather than granting that principal unlimited authority over others.
This is where a purely individualistic theory of data reaches its limits because personal context is inherently relational. A conversation, for example, contains information about multiple people; photographs contain bystanders; social graphs encode relationships; inferred knowledge about one person can reveal information about another. An Egoist Machine shoudl therefore govern an individual's relationship with their context without establishing an absolute property right over every piece of information that happens to appear within that context.
Machine-generated representations will inevitably contain inaccuracies, meaning that personal context can never be treated as an objective or permanent description of the individual. The ability to inspect, correct, contextualise, expire, and reject machine-generated conclusions is therefore a necessary consequence of user-governed memory.
Governance also cannot require constant micromanagement. A system that technically provides thousands of permission settings but requires the individual to evaluate every disclosure and action would transfer administrative burden but would not meaningfully increase agency. Individuals therefore need the ability to apply higher-level constraints that machines can consistely apply to lower-level decisions.
Egoist Machines, Concisely
If accumulated understanding becomes portable between intelligent systems, competition between consumer AI providers can increasingly shift towards the quality of intelligence, interfaces, tools, services, and execution rather than ownership of the individual's accumualted context.
If applications can request permissioned access to existing personal context, they can become useful without first reconstructing the individual through lengthy onboarding, behavioural surveillance, or repeated explanation.
If personal objectives can be represented independently of single applications, personalisation can evolve from primarily predicting what people are likely to do toward helping them pursue what they have explicitely decided they want to doo.
If delegated authority can similarly move with the individual, AI agents can become representatives capable of interacting with businesses, marketplaces, institutions, and other agents while maintaining a consistent relationship with the person whose authority they exercise.
The older economic idea od countervailing power, when one side of a market develops institutions capable of balancing concentrated power on the other side, accurately desscribes how Egoist Machines create the possibility that individuals can possess machines capable of modelling institutions and markets in order to advance the individual's objectives.
The wider consequence is a change in who benefits from the accumulation of intelligence about the individual. Under existing architecture, every interaction can increase an instituion's ability to understand the individual. Under an Egoist architecture, those same interactions can also increase the individual's ability to navigate through a machine that retains and applies what has been learned on their behalf.
As machines become more capable of understanding people and acting in the world, intelligence is an insufficient descriptor of the relationship humans need with them.
An Egoist Machine is an intelligent system whose memory, incentives, and actions are governed by the individual it represents.