Abstract:
Text generation is an important emerging AI technology that has seen significant research advances in recent years. Due to its closeness to how humans communicate, mastering text generation technology can unlock several important applications such as intelligent chat-bots, creative writing assistance, or newer applications like task-agnostic few-shot learning. However, several key challenges need to be addressed before their real-world deployment is practical. This thesis is aimed at identifying and making progress towards fixing fundamental limitations with current text generation systems. We focus on three limitations of current systems --- (1) poor controllability of text generators; (2) factual inconsistencies between the input and generated text; (3) inherent difficulty in evaluating generated text.
First, we focus on making text generation more controllable, by modifying aspects like style / formality while preserving semantics. We introduce STRAP and DiffUR, two state-of-the-art methods for style transfer of text using diverse paraphrase generation. To encourage future research in this area we also introduce the Corpus of Diverse Styles, a new challenge benchmark with 15M sentences across 11 diverse styles.
Shifting focus to the other two issues, we first present a study in long-form question answering which empirically measures the extent of these issues. We build a retrieval-augmented model for this task which achieves state-of-the-art results on the ELI5 benchmark. However, closer analysis reveals poor consistency between the retrieved input documents and model generation, and difficulty in human and automatic evaluation of long generations. To improve input / output consistency of generated text we introduce RankGen, a 1.2B parameter encoder model trained with large-scale contrastive learning. RankGen significantly outperforms competing open-ended text generation methods in terms of automatic and human evaluation, generating text more faithful to the input.
As proposed work, I hope to make progress on (i) tackling the third limitation on evaluation with a human-in-the-loop evaluation protocol for long-form QA and summarization; (ii) extending our work on controllable text generation beyond a sentence level; (iii) using RankGen for retrieval augmented generation and retrieval tasks more broadly.
Advisor: Mohit Iyyer