PhD Thesis Defense: Saaduddin Mahmud, Aligning Agentic Systems: Improving Specification and Scalability
Content
Speaker:
Abstract:
Agentic systems consist of objective-driven entities, or agents, that perceive, reason, and act within an environment. Over the past decade and a half, fueled by advancements in deep learning and subsequent breakthroughs in computer vision and natural language processing, modern agents have become increasingly capable. As a result, we are witnessing the widespread adoption of these systems in autonomous driving, conversational assistance, software engineering, and automated research. As they become more integrated into daily human life, ensuring the safety of agentic systems has become critical.
One foundational way to frame safety in agentic systems is through the alignment problem, which involves ensuring that an agent’s objective aligns with that of the human it serves. Without proper alignment, an agent may exhibit undesirable behavior while pursuing objectives that diverge from those of the human. This dissertation conceptualizes alignment as two closely coupled processes: inferring the intended objective from direct or indirect signals, and optimizing agent behavior to achieve that objective. Failures in either stage can cause agents to act unpredictably or inefficiently, resulting in degraded utility as well as severe safety hazards.
In the alignment literature, communicating task objectives through signals such as demonstrations, interventions, and preferences has been studied extensively. However, these signals often underspecify the intended objective, leading to ambiguous objective inference in novel scenarios. Even when an objective is inferred correctly, scaling behavior optimization techniques for agents, especially those leveraging large language models (LLMs), remain challenging due to computational requirements. This dissertation addresses both the specification and scalability challenges.
First, to mitigate the underspecification problem, this dissertation introduces approaches that enrich sparse feedback signals by capturing the reasoning behind them. Specifically, we access human reasoning either by modeling predictable patterns in how humans give feedback or by eliciting it directly in natural language through an active learning process. We then design novel loss functions to integrate this reasoning with the original sparse signals to infer the human objective. We find that in many practical scenarios, combining classical sparse signals with readily accessible reasoning can substantially improve objective inference.
Second, this dissertation examines a widely encountered problem in agent behavior optimization: identifying the best agent configuration for a given task objective from a pool of candidates. We formulate this search as a contextual fixed-budget best-arm identification (BAI) problem and develop an algorithmic solver with finite-budget guarantees for best agent identification. Our experiments show that the proposed solver identifies better-performing agent configurations within a fixed evaluation budget.
Advisor:
Shlomo Zilberstein