UMass Amherst, Rice Team Wins Grand Prize in National Automated Scoring Challenge

Doctoral students Nigel Fernandez and Aritra Ghosh, Assistant Professor Andrew Lan, and Benoit Choffin, a former visiting doctoral student of the Manning College of Information and Computer Sciences (CICS) at UMass Amherst recently won a grand prize in the Automated Scoring Challenge organized by the U.S. Department of Education's National Center for Education Statistics (NCES) in collaboration with Naiming Liu, Zichao Wang, and Richard Baraniuk of Rice University.

The Automated Scoring Challenge represents a key component in modernization efforts to incorporate data science and machine learning into operational activities at NCES, and is the first in a series of challenges that use data collected from the National Assessment of Educational Progress (NAEP). The challenge invited AI models for automated scoring of open-ended student responses to reading comprehension questions.

"The winning approaches represent current best practices in natural language processing and demonstrate evidence of similar reliability to human scoring with certain types of items," says Peggy G. Carr, NCES commissioner. "These results suggest a promising path for NAEP to use automated scoring in the near future."

The team's submission, "Automated Scoring for Reading Comprehension via In-context BERT Tuning," was designed to address issues with current automated approaches to scoring open-ended student responses. Most current approaches rely on training separate models for every question. Apart from being poorly scalable, this approach reduces performance for contexts such as reading comprehension, where shared information can be leveraged across questions.

The team introduced a novel approach to automated scoring based on multi-task and meta-learning ideas. Their approach fine-tunes a single, shared language model, using the popular Bidirectional Encoder Representations from Transformers (BERT) model, with a carefully designed input format that captures the context of each individual question, and incorporates recent advances in natural language processing including in-context learning. This approach was able to outperform existing language model-based methods, and significantly outperform other methods.

"The challenge was an intense and exciting experience, requiring rapid prototyping of multiple ideas," says Fernandez. "It took effective collaboration from the entire team to pull it off and I'm glad we were able to win a grand prize."

Nigel Fernandez, advised by Professor Andrew Lan, is pursuing his doctorate in computer science at UMass Amherst with an emphasis on natural language processing, machine learning for education, and program synthesis. Previously, he was a research fellow at the Max Planck Institute for Software Systems in Germany, and completed an AI residency at Naver in Korea. His research publications include deep learning models for fake news detection, and his recent work on automatically synthesizing tasks to teach programming was published in the Conference on Neural Information Processing Systems (NeurIPS). Nigel received his bachelor's and master's degrees from the International Institute of Information Technology Bangalore (IIIT-Bangalore) in India, where he was awarded the university gold medal for outstanding academic and professional contributions.

Aritra Ghosh is a final year doctoral candidate at UMass Amherst, also advised by Professor Lan, whose work focuses on deep learning, reinforcement learning, and meta-learning with applications in recommender systems and in the educational domain. He is a recipient of the Duolingo English Test's 2021 Doctoral Award. He has won the NeurIPS Education challenge and received the Best Student Paper Award in IEEE Big Data 2020. His research publications include learning item selection policies for computerized adaptive testing and sketching (published in AAAI, IJCAI), deep learning models for sequential prediction (published in KDD, IEEE Big Data), and robust loss functions for learning under label noise (published in AAAI, CVPR, WACV). Previously, Aritra was a software engineer at Microsoft, where he worked on improving selection and relevance algorithms for BingAds.

Andrew Lan is an assistant professor at CICS. His research focuses on the development of artificial intelligence (AI) methods to enable scalable and effective personalized learning in education, including areas such as learner modeling, learning content understanding and generation, and human-in-the-loop AI. He received his master's and doctoral degrees in electrical and computer engineering from Rice University and his bachelor's degree in physics and mathematics from the Hong Kong University of Science and Technology.