Bagdasarian, Zilberstein Receive $500,000 to Advance Multi-Agent AI Safety
Content
Manning College of Information and Computer Sciences (CICS) Assistant Professor Eugene Bagdasarian and Professor Shlomo Zilberstein have been awarded a $500,000 grant from Schmidt Sciences to advance research in the safety of AI systems. Their project, titled "Unlocking Multi-Agent Systems Safety" aims to create a framework that allows AI agents to assess the trustworthiness of other agents and dynamically adjust their interactions, thereby safeguarding against adversarial attacks.
The researchers propose the development of innovative "trust modules" that enable AI agents – software entities that perceive their environment through sensors or data inputs, make autonomous decisions based on those perceptions, and take actions to achieve specific goals – within multi-agent systems (MAS) to distinguish between trustworthy and potentially malicious counterparts. This system leverages contextual privacy principles to help agents navigate safely between collaborative and competitive interactions, forming dynamic "islands of trust."
“As AI Agents will operate in the world with a mix of trusted and untrusted actors, it is important for these agents to efficiently alternate between collaborative and competitive modes depending on the context,” said Bagdasarian. “We, therefore, propose for agents to establish 'islands of trust' to isolate untrusted agents from influencing crucial decision-making."
The grant will support the team's efforts to address the inherent risks posed by MAS, particularly in scenarios involving sensitive personal data and differing user objectives. The project will specifically tackle challenges related to malicious influences, such as deceptive behaviors and collusion among agents, drawing from Bagdasarian's expertise in AI security and Zilberstein's extensive research in multi-agent decision-making and resource-bounded reasoning.
“Rapid advances in AI, particularly in planning and large language models, have brought us closer than ever to realizing the vision of multi-agent systems: creating automated assistants that collaborate seamlessly with one another and with humans. However, addressing the safety and security challenges that arise in such systems is critical to their successful deployment,” said Zilberstein.
Bagdasarian, who joined the CICS faculty in 2024 after completing his PhD at Cornell Tech, specializes in security and privacy within AI-based systems, including critical work on backdoor attacks in federated learning. Zilberstein, a member of the CICS faculty since 1993, is internationally recognized in artificial intelligence and automated decision-making, currently serving as General Chair of the 2025 International Joint Conference on Artificial Intelligence, and is a Fellow of both AAAI and the ACM.
The two-year project includes developing and evaluating the proposed framework through simulated and realistic environments. The researchers aim to make their findings open-source and compatible with modern AI agent platforms, enhancing safety and trust in practical applications.
