PhD Dissertation Proposal: Hunter McNichols, Potentials and Pitfalls of AI-Mediated Learning: Scaling Learning and Understanding Student Behavior with Natural Language Processing
Content
Speaker:
Abstract:
Teaching in higher education has shifted with the advent of ChatGPT and other consumer chatbots. Students now have persistent access to “AI Tutors,” which are convenient and can facilitate learning. However, student conversations with consumer chatbots are opaque to the instructor, unaligned to the curriculum, and may be detrimental to learning. Realizing the potential of AI in education requires NLP methods that serve two complementary goals: building scalable, pedagogically-grounded AI systems for assessment and learning support, and empirically understanding how students currently use AI tools so that future systems can be better designed. This thesis advances both directions of research through empirical evaluation of AI systems in a variety of educational tasks and a grounded year-long classroom case study analysis of ChatGPT usage in a university-level Computer Science course.
First, we study scalable educational assessment using LLMs through two works that showcase the rapid progression of natural language processing technologies in providing teaching aides for mathematics curricula. To provide timely, accurate feedback to students working through algebra requires identifying and interpreting errors in their responses, a task traditionally handled by rigid, rule-based systems with limited generalizability, or by data-driven methods that require strict constraints. We introduce a flexible LLM-based approach to algebra error classification that handles syntactically invalid student responses and outperforms both rule-based and prior data-driven methods. Building on the development of fluent autoregressive language models, we also investigate how LLMs can support teachers in creating learning materials. Specifically, we introduce a method for automated distractor generation for math multiple-choice questions (MCQs). We explore a range of LLM-based approaches to this task, from in-context learning to chain-of-thought prompting to fine-tuning, and conduct extensive quantitative and human evaluation on a real-world math MCQ dataset. We find that while LLMs can produce mathematically valid distractors, they struggle to anticipate the specific errors and misconceptions that real students exhibit, establishing a clear challenge for future work.
Second, we study how students interact with AI-powered tools in authentic educational settings in Computer Science. We introduce StudyChat, a publicly available dataset capturing real-world student dialogues with an LLM-powered tutoring chatbot deployed across a year-long, university-level artificial intelligence course. We conduct large-scale discourse analysis on this dataset using automated LLM-based dialogue act classification, revealing both productive and misaligned patterns in how students use AI tutoring tools and identifying behavioral correlates of learning outcomes. This work provides empirical grounding for understanding the opportunities and risks of AI-mediated learning environments and motivates the need for pedagogically-aware AI systems.
Finally, we investigate how AI can be used to provide personalized feedback on student writing. A well-established approach to teaching across disciplines is to provide high-quality examples of work, yet a canonical example may be too distant from a student's current writing to be helpful for the student. While AI provides a promising direction for automating student rewriting feedback, we find that AI systems rewrite student writing too heavily, completely changing the content of the student work and challenging learning. To address this, we propose a modular approach to counterfactual student writing generation that produces improved versions of student work while maintaining similarity to their original writing. Central to our approach is β-Hindsight control, a novel controlled decoding algorithm that uses latent Gumbel noise as a tunable similarity mechanism during generation. Ongoing experiments with this system on student writing datasets scored on multiple rubric criteria show promise that our method produces counterfactuals that are both rubric-consistent and meaningfully close to each student's own writing, enabling a new class of personalized, example-based feedback at scale.
Advisor:
Andrew Lan