Teaching Machines to Understand Humans

Working draft. This is the argument of a paper we are writing, Toward a Foundation Model of Human Behavior, put down in plain language while the paper is finished. It talks about how we can teach machines to understand humans and why that is an important mission. It is also a formalized direction for how you might build a model of this nature, and a survey of several spaces.

It is quite difficult to understand the human mind. We have long been pretty bad at it.

Basically all human attempts to understand the human mind have failed, or made modest progress in comparison to the ambition. We can go back as far as time itself: to Socrates and Aristotle, to the great philosophers, to Nietzsche, to Freud, to modern cognitive science. Each got somewhere. None got where it was trying to go. Behavior is high-dimensional and depends on context, we observe it only through sparse and fragmented traces, and the theories written on top of those traces do not predict very much.

Meanwhile, we have watched deep learning succeed across domains that had resisted every other approach, chemistry and physics among them, and in 2024 the Nobel Prizes in both were awarded for work in or enabled by neural networks. The winning formula across these looks the same each time: large amounts of data, and growing models that learn the patterns in that data on their own, in a way that ends up superhuman. AlphaFold learned the structure of proteins from solved structures. The models that followed it learned structure from sequence alone.

This leads us to ask what we can achieve, and whether we can see similar success in understanding and modeling humans: ourselves and our own psychology. Perhaps this direction holds the answer to questions our species has never been able to answer.

Contributions

Our contributions are fourfold.

1. We roll up the disparate definitions industry and academia have offered for machines that aim to understand humans into one definition of a foundation model of human behavior.
2. We survey the fields building these models relatively independently of one another: their methods, their results, and their trajectories.
3. We argue that these fields are converging, and we formalize their convergence into a common formula: next behavior prediction.
4. We identify future directions by asking what must be true for this field to progress, or, put differently, what must be true for us to build a machine of this nature. The seeming answer is data in a common language, which would allow the training of one general model, a foundation model of human behavior, that transfers across domains and tasks.

What it means to understand

It is not actually clear that anyone understands what the word understand means. Seemingly, it means to disentangle and clearly outline the explanatory or causal factors behind something.

In machine learning the Chinese Room is the standard citation here, and the discussion tends to devolve. Turing's position was that arguments of this kind are meaningless, and that what matters is whether the machine can do the thing: can it predict, can it perform the task, without merely memorizing? If it can reason, if it can predict or perform across a wide range of tasks, at some point you assume the machine understands, even if not in the anthropomorphized sense of the word.

We stick to a definition based on evaluation. If a machine can reliably predict or simulate what a human mind would do in any situation, it understands that mind.

Whether a computer can ultimately understand a person then hinges on its ability to learn to represent that person, which is the closest thing to the definition above that we know how to build. Bengio and colleagues describe what a good representation is: one that captures the underlying explanatory factors of the data, that stays useful across many downstream tasks rather than one, and that disentangles what is stable from what is situational. Read against a person, that is a representation which captures why they do what they do, serves any task that asks about them, and separates who they are from where they happen to be.

We formalize this as a representation formed from a chain of tokens. Let h be the stream of a person's record, everything that has been observed about them and the situations they were in. The model compresses it into a state,

and predicts from that state. Nothing about z is designed. It is whatever summary of the person the model needs in order to predict well. The stream h can be decomposed, and the decomposition is what people now call context engineering: anything from the environment, to the situation or scenario, the history, the memory of the user, their prior actions and the situations they took them in. We dive deeper into this later. The picture to hold is streams of events flowing into a representation that then serves as a prior for prediction. Ideally the definition asks for something that is not purely memorizing, but that understands, and so generalizes across scenarios that were never in the training distribution.

Foundation models

The term foundation model was coined by Stanford researchers in 2021: any model that is trained on broad data, generally using self-supervision at scale, that can be adapted to a wide range of downstream tasks. Three checkable clauses: broad data, generic self-supervision, adaptation to many tasks. The same report names the two properties that make these models matter. Emergence, where behavior is induced rather than explicitly constructed, and homogenization, where one model comes to underlie many applications.

A model of this kind can then be used across a wide variety of tasks, or adapted to them. The power is that there is shared structure across all of these different domains, and if you can compress them together, the model learns to transfer and generalize. The whole is greater than the sum of the parts.

Which models count

How do we tell a general model of today, something like GPT, from a foundation model of human behavior? And what about the awkward case where a model is optimized explicitly for human behavior and the general model performs better anyway? It seems the best way to decide whether something is a foundation model of human behavior is not definitional. In the same sense that a model can be said to understand exactly as well as it can reliably predict, the question of what to call it is close to meaningless. What matters is a core set of evaluations and a representation's capacity to perform on them. Many models are optimized for code or for mathematics, and there is no free lunch: that optimization does not help them model people, and can cost them the breadth of human behavior their pretraining absorbed.

Three fields

Three fields are building models that meet the definition, in whole or in part, and they barely cite one another. Human-computer interaction, simulation, and prediction.

Human-computer interaction observes a single user deeply and maintains a standing model of them in order to assist them. Its user-modeling tradition is as old as interactive computing. Today the models take three forms: the curated memories that commercial assistants keep from chat history, general user models like GUM that turn screenshots of computer use into confidence-weighted propositions about the person, and next-action predictors like LongNAP that train a vision-language model, by reinforcement, to retrieve from a per-user memory and predict what the user does next on screen. The stream is one person's, consented and deep, and the published models train on tens of users.

Simulation takes an open-weight language model and trains it to respond as a particular person, or a particular population, would. The prompted line supplies the person in context, as a demographic profile or as a two-hour interview transcript; the interview agents of Park and colleagues reach 0.83 of the accuracy with which people replicate their own answers two weeks later. The trained line moves the person into weights: HumanLM aligns a latent user state with the person's actual responses rather than imitating their surface text, HumanLLM fine-tunes on the public traces of 282,000 accounts, and OdysSim builds a foundation model for behavior simulation outright. Training is displacing prompting, and the yardsticks are the most person-grounded of the three fields.

Prediction, which in industry means recommendation, trains on billions of users' interaction sequences to predict the next item they will engage with. It is the one field that pretrains on behavior at population scale, and its pipeline now mirrors the language model's: build a tokenizer, pretrain by next-token prediction over the event stream, then post-train with rewards. HSTU recovered power-law scaling on raw event streams, TIGER replaced item IDs with semantic IDs a generative model can write, and PLUM, OneRec, and rankers at Netflix and JD put generative recommenders into production against mature baselines. Its own description of itself is the one to hold onto: constrained by isolated data, today's recommenders operate as domain specialists.

The underlying models are converging. If that is true, then this field should benefit from universal user representations, where the full understanding of a person comes with them across many different tasks, and the whole is greater than the sum of its parts. The proof is close to right in front of us: many modern recommender systems start from a pretrained language model, which means a generalized foundation already transfers and is useful in predicting a person's next action. A question that is not asked well enough is the reverse. Does what you, the individual, or what a population in the abstract, chooses to buy hold representational power that can transfer back and be taught to a large language model?

The common formula

The common formula is predicting the next behavior from the stream:

Each field writes its own version. Recommendation predicts the next item from the items so far. HCI predicts the next screen action from a memory of the user and the current screen. Simulation predicts a response to a situation given a supplied persona. Write the stream out and each collapses into Eq. 2 with part of h removed, summarized, or handed in from outside. That is the only way the fields differ.

The generality is what matters, but h can be decomposed, and it is worth being specific about what it holds. The environment E, the setting and its rules. Prior actions, and the context and situations those actions were taken in. A persona that has been written out. Metadata. The options available, which constrain the output space. We can assume infinite and rich context, which is helpful because it makes the formula clean. In reality that is not tractable, and most of the engineering in each field is deciding which parts of h to keep and how to summarize the rest.

Picture two circles. The outer circle is performance across a wide range of tasks, and its size is set by the model's generalized knowledge, its intelligence, its understanding of human nature. Inside it sits a more practical circle that tries, at every point, to reach the outer bound. That inner circle is memory and context engineering, and it is held back by constraints of its own: context rot, the need for the right amount of context, and the right context at the right time.

You can do a lot with memory today, and that is remarkable by itself. But to expand the outer bound, the outer circle of what is possible at all, we need to train better foundation models of human behavior. Netflix's own recommendation team describes the shift in their terms: feature engineering is becoming context engineering. We are seeing the same trend in each of these fields.

Future directions

We simply need broader, richer, longer data. Better data. It almost certainly needs to include collaborative signal, what people like you did, and it needs to be much richer than what existing language models are trained on: not documents with no person attached, but sequences with a person at the center of each.

Borrowing from the fact that recommender systems today have to learn behavior as a token and content as a token, we also need to think deeply about tokenization. Each field has solved one part of the stream its own way. Recommendation tokenized the action, HCI captions the screen into language, simulation writes the persona and the situation as text. A common language for the whole stream is what would let the silos connect: the domain specialists inside one company, and then across companies and across external datasets, open or not. The precedents are encouraging. Medicine got cross-country transfer once every event was an ICD code, and pharma federated pretraining across ten competing companies once every molecule was one fingerprint.

Scaling

Scaling is the meta of today. We have observed scaling laws in this field, and they are kind of incredible: where behavioral papers publish fitted exponents, the numbers sit close to the language ones, on medical events and on interaction sequences alike. We need to get the data mixture right. But ultimately, if you assume we have enough computational resources, what matters is that we have enough data for the model to learn these patterns on its own. That is the condition for the mission here: machines that understand humans far better than we understand ourselves, in a way that is seemingly godlike.

Recap

How could we teach a computer to understand people in a seemingly godlike way, in parallel to how it came to understand chess better than we do? We explored three paths and argued that they converge and generalize into a surprisingly simple one: next behavior prediction. Not next action, which leaves out the next word you say or the next thing you write. The term has to be general enough to encapsulate all of human behavior.

But simple does not mean easy. The path ahead is largely bottlenecked by data, and by creating a common language across different data types and modalities so that the data can be trained on together. That is what we are building.