Scaling Up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

We explore an approach to scale up test-time compute by training a language model that can perform iterative inference via a recurrent block. Unlike chain-of-thought reasoning, this model reasons in latent space, without expanding the context window or producing interpretable intermediate steps. We train a large-scale instance of this architecture—a 3.5 billion parameter model trained on 800 billion tokens—and show that this model can trade computation for performance at test time in a nearly compute-optimal way, improving performance on reasoning tasks by iterating the recurrent block beyond the number of steps seen during training.