AI Seminar: Lipstick on a Pig: Using Language Models as Few-Shot Learners

Today's Natural Language Processing (NLP) strategies are heavily reliant on pre-trained language models due to their ability to deliver semantically rich representations. While these models provide impressive few-shot natural language understanding and reasoning capabilities, simply using them as "fill in the blank" prompts may not be a one-size-fits-all solution. Despite the allure of direct application of these models - an allure that grows with model and dataset sizes - the objectives of language modeling and few-shot learning are not perfectly aligned. Unpacking this disparity is crucial.

In this talk, I will describe some of our work in characterizing the differences between language modeling and few-shot learning. I will show how language modeling comes with crucial shortcomings for few-shot adaptation and describe a simple approach to address them. Then, focusing on numerical reasoning, I will show that the reasoning ability of the language models depends strongly on simple statistics of the pretraining corpus, performing much more accurately for more common terms. These results suggest language modeling may not be sufficient to learn robust reasoners and that we need to take the pretraining data into account when interpreting few-shot evaluation results. While language models hold substantial promise, making these 'pigs' presentable with 'lipstick' may require a more comprehensive approach than currently anticipated.