NLP Seminar: David Chan, Teaching Agents That Use Language to Use Language

Content

Image

Speaker

David M. Chan (University of California, Berkeley)

Abstract

Language is not just something we generate, but something humans use: to inform, adapt, and act in a shared world. Indeed, today’s conversational AI agents speak fluently, but often treat language as prediction rather than interaction, producing responses that sound correct while failing to recognize when requests are ungrounded, impossible, or misunderstood. My research asks what it would take for multimodal agents to take the next step and use language to take actions in grounded contexts. In this talk, I argue that effective language use requires pragmatics: agents must account for who they are communicating with, what situation they’re in, and how a conversation unfolds over time. From visual understanding, to automatic speech recognition, to hallucination detection, I will demonstrate that incorporating pragmatic reasoning substantially improves agent behavior. Together these results point to a necessary shift in how we build multimodal conversational systems: from simple language models to pragmatic conversational agents that can see, listen, act, and speak in context.

Bio

David M. Chan, Ph.D., is a postdoctoral scholar at the University of California, Berkeley, specializing in multimodal conversational AI. His research focuses on developing scalable AI systems that move beyond language prediction toward grounded language use, integrating vision, audio, and language to enable pragmatic interaction, improve AI–human collaboration, and reduce hallucinations in generative models. Beyond academia, he has worked with leading organizations including Amazon, Google, and NASA, to build and deploy safe, efficient, and accessible machine learning systems. He is also the developer and maintainer of TSNE-CUDA, an open-source tool for high-dimensional data visualization, used by over 40,000 researchers in fields ranging from biomedical technology to industrial manufacturing. David holds a Ph.D. and M.Sc. in Computer Science from UC Berkeley, where he was a graduate fellow with the Center for Technology, Society & Policy (CTSP), and dual B.Sc. degrees in Computer Science and Mathematics from the University of Denver.