PhD Dissertation Proposal: Nischal Kumar, Scaling Open-Ended Learning: LLMs for Educational Content Generation and Assessment of 21st Century Skills
Content
Speaker:
Abstract:
As large language models (LLMs) become integrated into everyday learning, 21st century skills such as reading comprehension, creative writing, and critical thinking are increasingly important. However, these skills remain far less studied than well-defined domains such as mathematics and programming, and are difficult to teach and assess at scale. Learning activities that develop them are inherently open-ended: a passage admits many valid questions, a story can be told in many styles, and an argument can be judged along many dimensions. Criteria for assessing such skills also evolve over time. In this thesis, we study how LLMs can generate and assess educational content for open-ended learning of 21st century skills in alignment with human judgment.
First, we study reading comprehension question generation, where the task is to generate a question given a passage and a target answer. Since many questions can be valid, a trained model does not know which one educators would prefer. We propose two methods, data augmentation and overgenerate-and-rank. In data augmentation, we enrich the training data with diverse questions for the same passage and answer. In overgenerate-and-rank, we generate a pool of candidate questions and select the best one. We find that our methods outperform the prior state of the art on a dataset of expert-written questions about children’s stories. The gains are largest on implicit questions that require inference rather than recall of explicitly stated facts from the story text.
Second, we study personalized story generation, where LLM-generated stories must reflect an individual author’s writing style. LLMs used as personalized writing assistants can support writers facing writer’s block and second language learners by improving engagement, agency, and writing quality. We introduce Mythos, a dataset of 3.6k stories from 112 authors across five story-writing sources. We design a pipeline in which we first infer implicit characteristics of an author’s writing style from their prior stories and organize them into an Author Writing Sheet. We then use the Sheet to guide an LLM through persona descriptions and story rules to generate stories in the author’s writing style. We conduct automated and human evaluations and find that our generated stories capture authors’ writing styles more closely than baseline methods. We also find that the creativity and language use of stories are easier to personalize than their plot.
Third, we study the assessment of critical thinking in student argumentative essays. Critical thinking is increasingly important amid AI-generated content and misinformation, yet difficult to define and measure. We develop a subskill-level rubric from an established skills progression and have human experts annotate student essays using it. We benchmark automated scoring approaches ranging from prompting LLMs to fine-tuning open-source models. We find that fine-tuned open-source models agree most closely with human experts. However, performance drops on subskills whose score levels are hard to distinguish and on subskills unseen during training. The drop on unseen subskills is important because educators often revise assessment criteria or introduce new ones as instructional goals evolve. Hence, as proposed work, we will study how automated scoring can remain reliable as these criteria change.
Overall, we explore how LLMs can support open-ended learning of 21st century skills, from generating questions and personalized stories to scoring critical thinking as assessment criteria evolve. Together, these contributions advance educational AI systems that align with human judgment.
Advisor:
Andrew Lan