Lan, Students Assist Middle School Math Learning with Novel LLM Methodologies
A team of doctoral students at the Manning College of Information and Computer Sciences, University of Massachusetts Amherst (CICS), led by Assistant Professor Andrew Lan, is contributing key research to an effort at the Learning Engineering Virtual Institute (LEVI) to double the rate of middle school math learning within five years, with a focus on students from low-income backgrounds. The work dovetails with Lan's 5-year NSF CAREER award granted in 2023 for research on using AI to develop personalized tutoring and feedback for digital learning platforms.
Lan, along with CICS doctoral students Alex Scarlatos, Nigel Fernandez, Hunter McNichols, Jaewook Lee, and Wanyong Feng, is acting as a support hub for seven LEVI project teams based in industry and the academy that are developing digital tools to dramatically increase math learning outcomes for middle school students. Supporting projects including AI-powered avatars that increase engagement and a chatbot that helps Ghanian math students, Lan's team is providing to LEVI project teams the ability to apply novel large language model (LLM) methodologies that go beyond the prompting and fine-tuning methods commonly used in real-world educational applications.
According to Lan, the issue with using large language models like ChatGPT when teaching math is that they currently struggle to identify the source of the error in an incorrect answer—and struggle even more when multiple reasoning steps are required to arrive at the student’s answer.
“There are some things ChatGPT just cannot do, such as understanding what is wrong with a student’s reasoning process beyond the most obvious errors,” he explains. “It’s one thing to wrap a chatbot around your language model, but it takes something very different from existing techniques to understand and engage with student mistakes.”
Understanding students’ mistakes and providing them with helpful feedback is crucial to improving learning outcomes from intelligent tutoring systems and online learning platforms.
As part of their work with LEVI, Lan’s team is helping Eedi, a U.K. company that specializes in online quizzes that identify student misconceptions about how to solve math problems and provides insights on where students are struggling to teachers. To help provide useful feedback to students on their mistakes—including explanations and encouragement—at the scale required for an online learning platform, the team worked with data scientists from eedi to develop novel LLM methods to not only generate feedback but evaluate the quality of that feedback using educational goals.
Another LEVI project team being helped by Lan’s group, Carnegie Learning, is working with a system that processes open-ended answers from students rather than multiple-choice selections. Being able to study the “long tail” of wrong answers given by students provides insights that would be unavailable without this real-world dataset, and—like the work with Eedi—provides Lan’s team with a rare opportunity to study and work with high-quality training data in math education.
“There are many different wrong turns that students can take while learning math,” says Lan. “We hope to train our models to understand what human errors are and help students who aren’t picking up certain concepts—building their confidence and helping to ensure their progress through important and challenging material.”
Andrew Lan has been teaching at CICS as an assistant professor since 2019, after working as a postdoctoral researcher at Princeton University. In addition to his NSF CAREER award, Lan has won two grand prizes in the National Assessment of Educational Progress automated scoring challenges, for 2022 and 2023. He received his master’s and doctoral degrees from Rice University in 2014 and 2016.