PhD Thesis Defense: Dhruv Agarwal, Epistemological LLMs for Search and Discovery Under Evolving Beliefs
Content
Speaker:
Abstract:
Large language models are increasingly used to search for solutions, hypotheses, and designs whenever an evaluation metric is available. This paradigm underlies drug design, program synthesis, and—the canonical application studied in this thesis—automated scientific discovery. Repeatedly sampling solutions from a fixed proposal distribution is sufficient when the task is well represented during training, but remains inadequate when working with unfamiliar domains, proprietary data, previously unseen objectives, expensive verification, or large candidate spaces. The central argument of this thesis is that effective search in such regimes requires evolving beliefs—an LLM system must continuously ingest verification evidence, update its beliefs, and reason under uncertainty so that each round of generation is more informed than the last. We call such systems epistemological LLMs.
We study two strategies for capturing an agent's accumulating experience during search. The first does this by placing prior observations in-context, allowing accumulated evidence to update the assessment of new solutions. We first contribute a position on data-driven discovery as a useful setting in which LLM search can be programmatically tested. We then introduce AutoDiscovery, which uses Monte Carlo tree search to explore hypotheses, guided by Bayesian surprise in the LLM's elicited beliefs. Across 21 datasets and under fixed budgets, it produces up to 29% more discoveries deemed usefully surprising by the LLM compared to alternative search policies. In a subsequent expert-guided application to invasive lobular breast cancer, an AutoDiscovery-generated finding was replicated by medical researchers in an independent patient cohort and confirmed through tumor-tissue analysis. We then examine a limitation of AutoDiscovery's static belief model, whereby it rewards findings that are no longer surprising once earlier discoveries are taken into account. We find that evidence-informed beliefs identify 37.5% of static surprisals as spurious, but with changes that reduce redundant hypothesis proposals, we recover accumulated non-stationary surprisal by 30.6%.
Reasoning in-context, however, only refers to solutions as text, which carries no notion of what lies between two candidate solutions; search, therefore, advances in discrete steps, limited to whatever the model happens to verbalize. Our second strategy explores two methods for more controlled exploration by capturing accumulated feedback in explicit numerical states that guide subsequent proposals. MiGrATe adapts the model's weights through mixed-policy reinforcement learning for single-instance search, while BOPRO instead represents solutions in a continuous embedding space, updates a Bayesian surrogate as evaluations arrive, and uses its latent proposals to prompt the LLM with nearby observations. Both outperform inference-only search. In a word-search task, MiGrATe and BOPRO recover 71% and approximately 85% of targets, respectively, while in-context baselines plateau near 45%. In molecule optimization, MiGrATe achieves a mean Vina docking score below −10 kcal/mol, which none of the prompting baselines attain, while BOPRO completes search on all 58 protein targets in the wall-clock time in which greedy prompting completes only 12.
BOPRO surfaces two challenges in the interface between continuous-space exploration and discrete sequence generation with an LLM. First, external embeddings may fail to preserve the local structure of the search objective, as seen in program search; second, BOPRO’s approximate decoding procedure can fail to accurately realize the latent point proposed by Bayesian optimization. These motivate BOReFT, which learns a compact space of hidden-state interventions in a frozen LLM and uses that space as the domain for Bayesian optimization. Each proposed point steers generation directly, without the need for an inverse map from an external embedding space or prompt-based decoding of the proposal. We theoretically analyze how semantic coverage and interpolation in the learned space constrain the best attainable score, and show that decoding in BOReFT admits a standard stochastic-bandit observation model for adaptive search. Empirically, BOReFT finds more hidden targets in word search than strong LLM baselines and achieves higher property scores on two of three molecular objectives. Consequently, it provides a principled new bridge between discrete proposal spaces of LLM-based search and continuous black-box optimization.
The methods described in this thesis show how accumulated experience can guide search through evolving in-context beliefs, adapted model weights, and optimization over continuous representations. Our contributions move LLM-based search beyond stateless generation towards systems that continually search, verify, and adapt their beliefs to guide future discovery.
Advisor:
Andrew McCallum