Breaking into the Memory Walled Garden

There is a fantastic substack titled The Memory Walled Garden, which argues that memory is becoming the most important source of lock-in in consumer AI. This is because, in line with the current model, the longer somebody uses a given application, the richer that application's model of the user becomes, and the more the user stands to lose by switching to a competitor which does not share the same history of user context.

The authors do an interesting job of distinguishing between:

1. The explicit memory profile an AI system maintains about a user, and
2. The much richer context contained in the user's complete conversational history...

...and they argue that existing forms of data portability (such as allowing users to download an archive of their chats) do little to create competition between vendors because the users' information is not continuously available in a form that competing applications could actually use.

Their proposed alternative, written from a policy- rather than technology-focused angle, is something akin to the AI Passport we have developed at Egoist Machines: infrastructure for third-party memory through which a user's accumulated context can exist independently of any particular model or service provider, and be accessed by different applications.

Think of this model similar to how a file in Dropbox can be opened and read by many different applications.

The economic argument beneath this proposal lines up nicely with many of the sentiments we expressed in our manifesto. Separating memory from the applications that consume it would:

• Reduce switching costs
• Lower barriers to entry for smaller consumer AI competitors
• Force AI companies to compete on the quality of their products rather than on the accumulated user context they have managed to capture.

Beyond the Walls

As AI applications learn more about the people who use them, memory itself becomes a source of product quality and therefore, as noted above, a source of switching cost. Intuitively, an AI application with which someone has spent two years discussing their work, relationships, preferences, recurring decisions, and ambitions is not economically equivalent to an otherwise identical they encounter for the first time.

The obvious danger is that a market in which intelligence itself becomes increasingly substitutable, while accumulated understanding of users remains proprietary, leaves users notionally free to choose between models but practically stuck inside whichever platform they already use.

Switching costs are nothing new, however what is new is that the valuable asset in consumer AI is not something you can purchase directly from a company as it consists of the summation of history of the interactions between you, as a user, and the services a company offers. With conventional consumer-targeted software, years of use will principally produce knowledge in the human. The experienced Photoshop user, for example, will know Photoshop better than the novice, and much of the switching cost for the experienced user comes from abandoning their accumulated product-specific knowledge. Consumer AI is somewhat reversing this relationship because the product is also learning the user.

The human becomes better at learning the machine while the machine becomes better at serving the human

This means that a portion of the value experienced by the user in, for example, year 2 did fundamentally not exist in the product that was purchased in year 1 as it has been created through the repeated interaction between the human and the machine.

Albert Hirschman has an interesting distinction between the two primary ways people react when they experience a decline in quality or satisfaction within a group, organisation, business, or state: exit and voice:

• Exit: involves leaving the organisation or switching to a competing product/service (often called "voting with one's feet")
• Voice: involves expressing dissatisfaction and trying to repair, improve, or change the organisation from within

AI systems create a constraint on exit because the better an incumbent service has become at understanding a user, the greater the sacrifice involved in disciplining it by leaving. A theoretical right to move from one consumer AI provider to another therefore means relatively little if exercising that right destroys the accumulated surplus of the relationship.

The authors of The Memory Walled Garden are also right, therefore, in saying that allowing someone to download their chat history is an insufficient way of 'scaling the walls'. OpenAI and Anthropic do allow users to export their conversations, however they argue that this is a "commendable feature but not a market enabling one" because digital markets are created by the infrastructure through which software can use information, rather than by some humans manually carrying archives between databases. So to get a 14GB JSON file of context is pointless (and economically useless as portability) because, while technically the individual can possess this information (and therefore it is technically portable), other systems cannot actually use this information well enough to preserve the value of the context that has accumulated around the individual.

Memory is MORE than a file

The point around some sort of memory.md file being insufficient is that it only contains specific facts or preferences that are relevant to the architecture of the model in which those facts or preferences were revealed so is inherently non-transferable in a fully operational way. It has already been mentioned that a full data export of all chat history is also not especially useful, but there is a third consideration that is exceptionally important when thinking about AI. We must consider the representation that a machine constructs from the record of its relationship with an individual.

Let's consider someone who has been discussing whether they should leave their job with an AI assistant. The conversations themselves constitute a record of what has happened, while an inferred "fact" about the person such as "this person is dissatisfied with their employer" constitutes a representation derived from that record, and an instruction that this information should never be disclosed to work-related applications constitutes a policy governing how that representation may be used. These three are not interchangeable, meaning that transporting the record does not necessarily transport either the representation or the rules according to which the representation should matter. So we have three layers:

1. Record: what happened
2. Representation: what the machine believes those events imply about the individual
3. Policy: how that representation may be used

The problem involved in making personal context a portable entity involves the need to maintain a persistent, machine-readable representation of a changing human without allowing either the custodian of that representation, or every application that access it, to become the final authority on whom that person is.

Your AI's "knowledge"

There is considerably more philosophy hiding in the phrase "your AI's knowledge" than might first appear.

Memory has occupied an unusual place in theories of personal identity for centuries. Locke grounded his account of personal identity in continuity of consciousness, arguing that the identity of the person extends backward insofar as consciousness can extend to previous thoughts and actions. Contemporary philosophy (as is the case with much of philosophy and, increasingly, technology) has disputed almost every element of Locke's account, but the principle to pull out in discussion of AI is that memory is not generally treated as just another category of information because it helps create continuity between the person who acted yesterday and the person deciding what to do today.

Clark and Chalmers made discussion of the relationship between memory and technology stranger still in The Extended Mind. Their famous example concerns Otto, a man with Alzheimer's disease who relies upon a notebook to record information that biological memory would ordinarily provide. Because of the facts that; (1) the notebook is reliably available, (2) the notebook is automatically trusted and consulted in Otto's life; Clark and Chalmers argue that there is no reason for all 'thinking' to occur exclusively inside Otto's head, and therefore Otto and his notebook may be seen as part of one extended cognitive system. The moral implication of this thought experiment is that interference with a sufficiently integrated external cognitive source (for example, if someone were to maliciously write things down in Otto's notebook) can be direct interference with the person whose cognition depends on that external cognitive source.

Personal AI can be seen as a real world manifestation of this psychological experiment because an AI assistant can become (1) continuously available, (2) immediately accessible, (3) routinely consulted before decisions, and (4), unlike Otto's notebook, use past histories of user decisions to inform an action without the end user needing to consciously remember that all the relevant information exists. One does not need to claim that AI memory is literally a part of the human mind but it does lead to the implication that control over AI systems and control over ordinary application databases are qualitatively different when these AI systems are increasingly involved in how intentions are translated into actions.

An AI system with better memory of its users can therefore have a cognitive moat.

A Garden needs Walls

If siloed memory creates cognitive as well as economic lock-in, surely we should aim for maximum portability, with every application able to draw on a common and complete representation of the user. Unfortunately, walled data performs useful functions too.

Suppose a coding assistant could access the complete history of a person's conversations with their general-purpose AI chatbot. The assistant might become dramatically better at its job because it would understand factors such as:

• the blue sky vision of the project
• previous technical decisions
• the user's preferred programming style
• the purpose of the product being built

Unless context distribution is highly scoped, the same conversational history might also contain financial difficulties, arguments with a partner, political beliefs, private correspondence, insecurities, medical questions, life goals, etc. The fact that information improves the completeness of a model does not establish that every system capable of benefiting from that information should receive it.

Human life depends on information constantly moving. Your doctor might know things about you that your friends do not; your employers know things that shops do not; your partners know things that your bank does not. We are intuitively aware that the same fact may be perfectly appropriate to reveal in one relationship while being deeply inappropriate to reveal in another. When discussing privacy therefore, whether information flows is less important than between whom information is flowing. The fragmentation of the present internet may be economically inefficient but it does preserve many of these contextual boundaries. To simply destroy every silo without introducing a new system of boundaries would replace fragmented surveillance with integrated surveillance. The resulting world would be one in which every application can query an extraordinarily rich model of the individual simply because the information exists somewhere in their personal context.

The objective should be user-governed memory fluidity in which context can become more portable between providers while becoming more deliberately bounded between purposes. So, the problem with the memory walled garden is not ultimately that personal memory has walls. Personal memory needs walls. The problem is that somebody else is choosing who gets to go through the gates.

Who opens the gates?

The authors of the original essay are entirely right that the memory walled garden must be opened. A future in which every relationship with intelligence ends when you swap to a different provider will leave users captive to whichever providers they choose to first use. The ability to leave one provider for another while maintaining the useful consequences of that accumulated relationship should therefore become a basic property of consumer AI technology.

It would be relatively easy to imagine solving portability by replacing one centralised memory system with another. Instead of OpenAI, Anthropic, Google, and every application maintaining separate models of the same person, some universal memory provider could maintain the canonical model and then expose it through an API, or similar. Applications and assistants would be able to use it to request context, then the memory provider could return context, and the technical issue of interoperability could be, in effect, solved.

Unfortunately, this would just serve to reproduce the original problem one layer down. This is because the danger of the memory walled garden is not really contained in ChatGPT knowing something that Claude does not, but rather that increasingly consequential representations of a person are being constructed inside systems over which that person has little to no editorial authority. Moving those representations into some third-party database without giving the individual more agency over their construction, interpretation, and, most importantly, their distribution would create a more portable garden, but it would not give the individual control over the gates to that garden.

Therefore, the third-party memory model proposed in The Memory Walled Garden solves only the important problem of location: rather than ChatGPT's understanding of me living inside ChatGPT and Claude's understanding of me living inside Claude, some representation of me can persist independently of either application and be made accessible to both. But, this third-party memory provider could itself become a new walled garden if it opts to be closed, which is arguably the lesser of two evils because an entirely open memory layer would make extraordinarily intimate context broadly available, to the detriment of the privacy of the individual.

This is why the Dropbox analogy used the essay is useful, but falls short of explaining the full picture. A file is comparatively passive. If I move a photograph from Dropbox into Photoshop, there is relatively little ambiguity about what has moved because it is just the photograph. Personal context is significantly different because, as discussed above, what an AI system "knows" about me includes facts but also includes interpretations of those facts, and those interpretations may be appropriate for one application while being inappropriate for another. Portability, in a user-first sense, cannot end at the same representation universally accessible.

A better way of looking at this might be that memory requires something resembling permissions as well as portability. When a new AI application wants to know something about me, it should not follow that its ability to make use of some piece of context gives it an automatically granted claim to receive it. A coding assistant might reasonably need to know the architecture of the company I am building and how I like code explained; it probably does not need the conversation in which I complained about my co-founder six months ago, even if that conversation happens to contain information about the company too. The difficult technical problem is in preserving useful context while maintaining the boundaries between the different relationships in which that context was created.

This slightly changes the competition argument in The Memory Walled Garden where the goal was aligned with making memory maximally open. Instead, the goal should be to make the user's relationship with their memory independent of the user's relationship with any particular AI app or assistant. While those sound somewhat similar, they produce very different architectures. The first requires constructing a universal profile which every sufficiently authorised application can consume and use to alter the outcome for the individual; the second treats the individual as the point through which access to context is mediated before outcomes can be altered.

This is the part of the problem we have become particularly interested in while building AI Passport at Egoist Machines. We agree with the essay's central contention that memory should become infrastructure rather than an application-level moat, but we think that infrastructure has to do more than make context portable. It has to make context portable on behalf of the person it describes.

The persistent representation of a user should exist independently of the applications that contribute to it and consume it. We envision a world in which an application can learn something useful about you without acquiring permanent ownership over that knowledge; and then another application can benefit from what has already been learned without requiring you to rebuild the relationship from zero. Most importantly, we envision a world where the individual can determine what enters that representation of them, what leaves it, and which applications are allowed to use which specific parts of it. You as the individual should be the one opening and closing the garden gates.

In that sense, we are trying to break into the memory walled garden without destroying the garden itself. We want to separate the compounding value of being understood from the company that happened to accumulate that understanding first. If we can do that, switching from one AI product to another no longer has to mean becoming a stranger and starting again from first principles. Economically, new applications can compete for users on the quality of what they do with context rather than on whether they happened to capture that context first.

As models improve and the marginal differences between sources of intelligence become smaller (especially for the average user), the enduring relationship will increasingly be between the person and their context, rather than between the person and any particular model.

Models will change and applications will come and go, but the accumulated understanding of the individual should survive them. This is the world of Egoist Machines.

The memory walled garden should not belong to OpenAI, Anthropic, Google, or, for that matter, us at Egoist Machines. It should belong to you.