Publications

Large Language Models (LLMs) have recently shown success across a range of social tasks, raising the question of whether they have a Theory of Mind (ToM). Research into this question has focused on evaluating LLMs against benchmarks, rather than testing for the representations posited by ToM. Using a cognitively-grounded definition of ToM, we develop a new evaluation framework that allows us to test whether LLMs have a mental causal model of other minds (ToM), human-like or not. We find that LLM social reasoning lacks key signatures expected from a causal model of other minds. These findings suggest that the social proficiency observed in LLMs is not the result of a ToM.

Language understanding in the real world occurs through noise — often, lots of noise. What makes language understanding so robust? Here, we address this challenge with a new approach. We cast language understanding as Bayesian inference in a generative model of how world states arise and project to utterances. We develop this model in a case study of action understanding from language input: inferring the goal of an agent in 2D grid worlds from utterances. The generative model provides a prior over agents’ goals, a planner that maps these goals to actions, and a — ‘language-renderer’ that creates utterances from these actions. The generative model also incorporates GPT-2 as a noisy language production model. We invert this process with sequential Monte Carlo. In a behavioral experiment, the resulting model, called the Generative Semantic Transformation Process, explains evolving goal inferences of humans as utterances unfold.