All podcasts / Dwarkesh Podcast / Summary

What rebuilding AlphaGo teaches us about self-play, RL, and future of LLMs - Eric Jang

2026-05-15 - 157 min - source - Read full transcript

Dwarkesh Patel (host)Eric Jang

Key insights

A 10-layer neural network can amortize an almost intractable tree search into a single forward pass, and that compression is AlphaGo's real breakthrough.

Jang argues the profound thing about AlphaGo isn't beating humans at Go, it's that roughly 10 sequential steps of distributed neural computation can approximate a search space larger than the number of atoms in the universe to very high fidelity. He connects this to AlphaFold's protein folding and speculates it hints our understanding of NP-hardness in practice (versus worst case) may be incomplete.

self-play-and-search

MCTS gives AlphaGo a dense, low-variance supervision signal because it relabels every action taken, not just the actions in games that were won.

Instead of upweighting all moves from a winning game and downweighting all moves from a losing one (which wastes most of the signal on redundant labels), MCTS runs a search from every move and tells the policy network a strictly better action it could have taken there, similar to the DAgger algorithm in imitation learning for robotics.

reinforcement-learning

LLM policy-gradient RL is dramatically more sample-inefficient than supervised learning because of a 'bits per FLOP' problem: sparse per-trajectory rewards plus a low pass-rate regime.

Dwarkesh's framework splits efficiency into samples-per-FLOP (which falls as trajectories get longer, since two days of agent work yields one scalar reward) and bits-per-sample (which for RL follows the entropy of a binary pass/fail variable, versus supervised learning's negative-log-probability curve). Early in training, when the pass rate is near zero, RL extracts almost no signal per sample, unlike supervised learning.

reinforcement-learning

Soft-label distillation carries far more information per sample than one-hot correct-answer labels, which is why AlphaGo trains its policy on the full MCTS visit-count distribution rather than the single best move.

The entropy of a one-hot label is zero bits; the entropy of a full probability distribution over legal moves is much higher, so training on the distribution (dark knowledge) teaches the network more per training example than training on the argmax action alone.

reinforcement-learning

MCTS-style search likely won't transfer directly to LLM reasoning because language's action space is too broad and rarely revisited for PUCT's exploration bonus to make sense.

PUCT relies on repeatedly visiting the same child node to shrink its exploration bonus over time, but in language an LLM will almost never sample the exact same next-token sequence twice, so the visit-count-based exploration heuristic that works for a 361-move Go board breaks down for a combinatorially vast token space. Jang doesn't rule out some future form of forward search returning for reasoning, but the current instantiation doesn't map cleanly.

llm-reasoning

Rebuilding a strong Go bot from scratch cost about $10,000 in 2026 versus the original AlphaGo's outsized compute budget, because being first to solve a problem costs far more than catching up once strong opponents and distillation targets already exist.

Jang got a $10K compute donation from Prime Intellect (roughly $4K exploratory, $3K on the final run) and used best-response training against existing KataGo models rather than fully tabula rasa learning, borrowing 'crutches' the original AlphaZero team didn't have access to.

scaling-laws

Many of KataGo's algorithmic compute-multiplier tricks may be transitory: their benefit shrinks as GPUs get faster, and different tricks likely have correlated rather than additive benefits.

Jang found that with modern Blackwell-class GPUs, several tricks that mattered on V100-era hardware stopped contributing meaningfully, and simplifying the training setup (e.g., a synchronous collect-train-collect loop instead of a distributed async pipeline with replay buffers) worked nearly as well, suggesting most algorithmic cleverness compensates for a specific compute regime rather than being fundamentally necessary.

scaling-laws

Off-policy training helps or hurts depending on whether the relabeled states are ones the deployed policy would actually visit, not on off-policy-ness per se.

If the replay buffer only contains states far outside the trajectories the current policy would ever reach, the model wastes capacity learning to act well in situations it will never encounter. The DAgger framing (train mostly on optimal-trajectory states plus a bounded 'tube' of drift states around them) explains why AlphaGo's replay buffer works despite training on stale, off-policy-generated data.

reinforcement-learning

Current AI coding agents are strong at open-ended hyperparameter and architecture search but weak at 'lateral thinking': stepping back to abandon an unpromising line of experimentation.

Jang used Opus 4.6 and 4.7 throughout the project and found the models excellent at grad-student-level iterative tuning (rewriting data loaders, adjusting layers based on gradient diagnostics) but poor at recognizing when an entire experimental track (what he calls a 'row') isn't worth pursuing and pivoting to first principles; he had to catch several infrastructure bugs himself by prompting the right diagnostic question.

automated-ai-research

Go, scored by the unambiguous Tromp-Taylor rules, is a candidate outer-loop verification environment for training and evaluating automated AI research agents on broader domains.

Jang argues an outer loop that's fast and impossible to cheat (win rate against a strong open-source Go bot) paired with an inner loop of real research engineering (distributed systems, predicting whether ideas will work) could teach an automated scientist skills that transfer to harder-to-verify domains like biosciences or robotics, though he flags that scaling laws and truly novel discoveries (like Chinchilla in 2018) resist this kind of automated outer-loop verification.

automated-ai-research

Media referenced

As Rocks May Think - article - Eric Jang's blog post exploring the grander thesis of thinking as a computational primitive; Dwarkesh recommends it as connected to the episode's themes.
Building AlphaGo from scratch (AutoGo tutorial) - article - Eric Jang's blog writeup (evjang.com) of the rebuild project, including an experiment tree diagramming every failed, successful, or mixed result and its follow-on.
AlphaGo (documentary) - movie - Referenced when discussing the TPU pod used during the Lee Sedol match versus the laptop shown on screen.
Scaling Scaling Laws with Board Games - paper - Andy Jones's 2021 paper showing test-time compute can substitute for training compute in board games, anticipated LLM inference scaling.
Generalized Advantage Estimation - paper - John Schulman's paper, recommended reading for how to compute advantage estimates that reduce RL gradient variance.
KataGo paper - paper - David Wu's 2020 open-source project achieving a 40x reduction in compute needed to train a strong Go bot from scratch; the baseline Jang benchmarks against.
Reiner Pope blog post on cryptography and neural networks - article - Referenced for the structural similarity between cryptographic diffusion and neural network layers, both needing sensitivity to all input information.
Ilya Sutskever Dwarkesh episode - podcast - Referenced for Ilya's point that a strong prior belief in an idea helps a researcher tell bugs apart from wrong ideas.

Companies

Google DeepMind - Original developer of AlphaGo; Jang was previously a senior research scientist there in the robotics group.
1X Technologies - Jang was most recently VP of AI here before taking a sabbatical to rebuild AlphaGo.
Jane Street - Employer of David Wu, author of the KataGo open-source Go engine.
Anthropic - Jang used Opus 4.6 and 4.7 as his automated coding assistant throughout the project; also discusses upcoming Mythos-class models.
Prime Intellect - Donated roughly $10K in compute that funded Jang's from-scratch AlphaGo rebuild.
Nvidia - GPU generational improvements (V100 to Blackwell) discussed as a major driver of why old compute-saving tricks matter less today.
OpenAI - Cited alongside DeepMind's AlphaStar for using neural fictitious self-play in Dota.

Techniques and frameworks

Monte Carlo Tree Search (MCTS) with PUCT - The core search algorithm: select via Predicted Upper Confidence bound for Trees, expand, evaluate with the value network, and back up averaged values.
Self-play policy improvement (DAgger-style relabeling) - MCTS treats every action taken in a game as fixable: it relabels each move with a better one, regardless of whether the game was ultimately won or lost, giving a dense per-move supervision signal.
The Bitter Lesson - Framing for whether compute-scaling alone, versus algorithmic tricks, drives most of the gains in a compute-optimal Go bot.
Distillation onto soft targets - AlphaGo trains its policy network on the full MCTS visit-count distribution rather than just the argmax action, because soft labels carry far more bits of information per sample.
TD learning / Q-learning - Discussed as the model-free analogue to MCTS backup for domains, like StarCraft, where you can't cheaply construct and search a game tree.
Off-policy vs. on-policy replay buffers - Jang explains why AlphaGo's replay buffer can safely include somewhat stale, off-policy states as long as they lie near the optimal trajectory, similar to DAgger in robotics.

Summary

Eric Jang, formerly VP of AI at 1X Technologies and a senior research scientist at Google DeepMind Robotics, spent his sabbatical rebuilding AlphaGo from scratch and used the project as a lens for understanding both classical self-play RL and the RL methods now used to train LLMs. The episode opens with a hands-on demonstration of Go's rules (capture, territory, Tromp-Taylor scoring) before Jang walks through the AlphaGo architecture in detail: Monte Carlo Tree Search with the PUCT selection criterion, a policy/value network that replaces exhaustive search with a fast, intuitive guess, and the self-play loop that repeatedly distills search-improved move distributions back into the raw network. Central to his explanation is that MCTS doesn't just reward wins - for every single move, it recomputes a better action via search and trains the policy to imitate that better action directly, a DAgger-like relabeling scheme rather than a coarse win/loss signal spread thinly across an entire trajectory.

A long middle section contrasts this dense, per-move supervision with how LLMs are trained via policy-gradient RL today. Dwarkesh lays out his "bits per FLOP" framework from a prior blog post: LLM RL suffers both because trajectories are getting longer (fewer learning signals per unit of compute) and because, especially early in training when pass rates are near zero, a binary win/lose reward carries almost no information compared to a full cross-entropy supervised-learning signal. Jang connects this to distillation, arguing that training AlphaGo's policy on the full MCTS visit-count distribution (a soft label) rather than just the top move captures far more bits per training example than one-hot supervision. They also discuss why classic MCTS likely can't transplant directly onto LLM reasoning: Go's exploration heuristics depend on revisiting the same node many times, which almost never happens across language's vastly larger and less-structured action space.

The conversation turns to practical compute economics: Jang rebuilt a strong Go bot for roughly $10,000 using a Prime Intellect compute donation, versus the far larger budget the original AlphaGo required, because being first to solve a problem is inherently more expensive than catching up once distillation targets (KataGo, in this case) already exist. He also questions how durable KataGo's various algorithmic compute-multiplier tricks really are, suspecting many of them mattered more on older, slower GPUs and stack poorly with each other or with newer hardware. A related digression covers off-policy versus on-policy training, where Jang uses a DAgger framing to explain why AlphaGo's somewhat stale replay buffer doesn't destabilize training, as long as most sampled states remain close to the policy's actual trajectory distribution.

The episode closes on automated AI research: Jang describes using Claude Opus 4.6 and 4.7 as a coding-assistant loop throughout the project, finding the models very strong at grad-student-style hyperparameter and architecture search but weak at "lateral thinking," i.e., recognizing when an entire line of experimentation isn't paying off and should be abandoned in favor of first-principles rethinking. He proposes Go, with its fast and unambiguous Tromp-Taylor-scored outer loop, as a candidate sandbox for training and evaluating future automated-researcher agents, with the hope that research taste developed there might transfer to harder-to-verify domains like biosciences or robotics.

Notable Quotes

"10 steps of neural network parallelized distributed-representation thinking is able to amortize and approximate to very high fidelity a nearly intractable search problem." - Eric Jang

"The problem with Go and chess is that the other player is always trying to do some shit." - Eric Jang

"Why is AlphaGo an elegant RL algorithm? The major reason is that you never have to initialize at a zero percent success rate and solve the exploration problem of how to get to a non-zero success rate." - Eric Jang

"You end up with this much more flexible, high-level, almost grad-student-like ability to just grind a performance metric." - Eric Jang

"You don't necessarily want to jump into the science of studying your man-made artifact before your man-made artifact is interesting enough to be studied." - Eric Jang