PhD Dissertation Proposal: Dhruv Agarwal, Epistemological LLMs for Search and Discovery under Evolving Beliefs
Content
Spaker:
Abstract:
Large language models (LLMs) are increasingly used not merely in settings where a single correct answer is required but also to search for novel solutions, hypotheses, and designs based on access to an evaluation metric. This paradigm underlies drug design, program synthesis, engineering, and—the canonical application studied in this thesis—automated scientific discovery. Repeated sampling from a fixed model is effective when the search task is well represented in pretraining, but it fails for under-represented tasks: unseen domains, proprietary data, and novel objectives, or when verification is expensive and search spaces are vast. The central argument of this thesis is that effective search in these regimes requires evolving beliefs—an LLM system must continuously ingest verification evidence, update its beliefs, and reason under uncertainty so that each round of generation is more informed than the last. We call such systems epistemological LLMs.
We study two distinct strategies for capturing an agent's accumulated experience. The first, non-parametric belief updating, keeps the model fixed and reasons in-context. Grounding this empirically, we contribute a position on data-driven discovery with a multi-agent proof of concept (DataVoyager) and the first benchmark that formalizes and evaluates the task (DiscoveryBench), on which state-of-the-art agents recover only a quarter of the gold hypotheses. Next, we extend this paradigm to open-ended discovery and formalize it as maximizing Bayesian surprise with an LLM's beliefs over hypotheses, realizing it as a Monte Carlo Tree Search over a data-driven verification pipeline (AutoDiscovery) that finds up to 29% more surprising, verifiable discoveries than strong search alternatives. Holding beliefs fixed as new discoveries accumulate, however, rewards search for rediscovering what it already knows and wastes its budget; we therefore develop evidence-informed beliefs that are continually updated in-context, identifying 37.5% of static surprisals as spurious, and propose improvements to search, which raise genuine discovery by 30.6%. We then study settings in which the evidence is too large to optimize over directly, where search collapses into locally exploitative strategies. Using heterogeneous data lakes as a canonical example of this setting, we present a method (Baikal) that first explicitly structures the search space into semantic evidence regions and then uses Bayesian region selection to find promising solutions, improving quality of findings by 28–36% over strong retrieval and agentic baselines.
Reasoning in-context, however, can only refer to solutions as text, which carries no notion of what lies between two candidates; search, therefore, advances in discrete jumps, limited to whatever the model happens to verbalize. Our second strategy, parametric belief updating, thus explores more controlled exploration by folding accumulated evidence into an explicit parametric state fitted at test time. This requires a continuous representation in which known solutions are points in a latent space and interpolations between them become candidate proposals that enable search to proceed at a finer granularity. The parametric state is either the model's own weights, which we show can be adapted through a mixed-policy reinforcement learning method for single-instance search (MiGrATe), or a surrogate over its representations, which we search over via latent-space Bayesian optimization that decodes proposals from a textual embedding space by prompting (BOPRO). Both outperform inference-only search: on a word-search task where prompt-based methods plateau near 45%, they recover 71% and 85% of targets, respectively; in molecule design, MiGrATe reaches binding affinities below -10 kcal/mol that no prompt-based method attains, and BOPRO optimizes all 58 protein targets within the budget in which greedy prompting completes only 12.
Finally, we propose two ongoing directions within each strategy. In the first, we observe that belief updates currently report only whether a hypothesis became more or less credible, giving the human expert no ability to observe or intervene in what mechanism drove the change; we therefore propose AutoDiscovery-Net, which represents hypotheses and their causes as a Bayesian network so that belief updates propagate through named mechanisms, building interpretable and interactive world models. In the second, we identify that decoding a surrogate's proposal using nearby solutions results in the evaluated candidate being different from the one requested, negating the targeted exploration that Bayesian optimization is designed to perform; we therefore propose BOReFT, which learns a smooth subspace of the model's internal representations and realizes proposals by directly intervening in its residual stream. These directions, together with our described contributions, move beyond stateless LLM generation towards systems that iteratively search, verify, and revise coherent beliefs, continually adapting with new evidence to guide future discovery.
Advisor:
Andrew McCallum