Publications

Vision-Language models (VLMs) can now produce fluent, socially appropriate dialogue, leading to interest in whether they have Theory of Mind (ToM). While recent work suggests that VLMs still lack coherent mental-state reasoning, this work has focused on classical propositional belief representations. Here we test a complementary, communication-relevant form of ToM: attention-based social micro-processes that support referential communication in the here and now (i.e., selecting efficient descriptions to identify a particular object for someone else). In face-to-face communication, people strategically add redundant color adjectives to facilitate the listener’s visual search, and omit them when they provide no benefit. We evaluate whether VLMs show the same strategy in a referential communication paradigm where the usefulness of redundant color words varies based on the set size and color distribution of objects. We find that VLMs can produce successful referential expressions but lack the attention-guiding strategies that make human communication so efficient. This suggests that VLMs lack the more implicit representations of attention people use in everyday communication.

Do Large Language Models (LLMs) possess a Theory of Mind (ToM)? Research into this question has found that LLMs succeed on a range of benchmark tasks. However, these evaluations do not test for the actual representations posited by ToM: namely, a causal model of mental states and behavior. Here, we use a cognitively-grounded definition of ToM to develop and test a new evaluation framework. Specifically, our approach probes whether LLMs have a coherent, abstract, and consistent model of how mental states cause behavior – regardless of whether that model matches a human-like ToM. We test our evaluation against GPT-4o and find that even though it succeeds in approximating human judgments in a simple ToM paradigm, GPT-4o fails at a logically-equivalent task and exhibits low consistency between its action predictions and corresponding mental state inferences. As such, these findings suggest that GPT-4o’s social proficiency is not the result of a ToM.

Large Language Models (LLMs) have recently shown success across a range of social tasks, raising the question of whether they have a Theory of Mind (ToM). Research into this question has focused on evaluating LLMs against benchmarks, rather than testing for the representations posited by ToM. Using a cognitively-grounded definition of ToM, we develop a new evaluation framework that allows us to test whether LLMs have a mental causal model of other minds (ToM), human-like or not. We find that LLM social reasoning lacks key signatures expected from a causal model of other minds. These findings suggest that the social proficiency observed in LLMs is not the result of a ToM.

Language understanding in the real world occurs through noise — often, lots of noise. What makes language understanding so robust? Here, we address this challenge with a new approach. We cast language understanding as Bayesian inference in a generative model of how world states arise and project to utterances. We develop this model in a case study of action understanding from language input: inferring the goal of an agent in 2D grid worlds from utterances. The generative model provides a prior over agents’ goals, a planner that maps these goals to actions, and a — ‘language-renderer’ that creates utterances from these actions. The generative model also incorporates GPT-2 as a noisy language production model. We invert this process with sequential Monte Carlo. In a behavioral experiment, the resulting model, called the Generative Semantic Transformation Process, explains evolving goal inferences of humans as utterances unfold.