Abstract: Commonsense knowledge is critical to achieving artificial general intelligence. This shared common background knowledge is implicit in all human communication, facilitating efficient information exchange and understanding. However, commonsense research is hampered by its immense quantity of knowledge because an explicit categorization is impossible. Furthermore, a plumber could repair a sink in a kitchen or a bathroom, indicating that common sense reveals a probable assumption rather than a definitive answer. To align with these properties of commonsense fundamentally, we want to model and evaluate such knowledge human-like using probabilistic abstractions and principles.
Traditional combinatorial probabilistic models, such as probabilistic graphical models, have limitations when modeling large-scale probability distributions containing thousands or even millions of commonsensical events. On the other hand, embedding-based representation learning, including large language models, has the advantage of generalizing to large combinations of events from a large corpus, but their inner workings are opaque and suffer from producing inconsistent probabilities under different styles of queries.
In this thesis, I combine the benefits of both approaches by modeling commonsense with probabilistic box embeddings, which represent joint probability distributions on a latent space of geometric embeddings. By using box embeddings, we can now handle complex queries with intersections, unions, and negations in a way similar to Venn diagram reasoning. In addition, existing evaluations do not reflect the probabilistic nature of commonsense knowledge. High accuracy in multiple-choice evaluation is misleading since the answer spaces are artificially constrained. To fill in the gap, I propose a method of sampling commonsense distributions from human annotators as well as a novel method of generative evaluation. I employ these approaches to create two new commonsense datasets: ProtoQA and Commonsense Frame Completion. The combination of modeling and evaluation methods based on probabilistic principles sheds light on how commonsense knowledge can be incorporated into artificial intelligence models in the future.
Advisor: Andrew McCallum