NLP and Text-as-Data Speaker Series

The NYU NLP and Text-as-Data Speaker Series takes place on Thursdays from 4:00pm to 5:30 p.m. at the Center for Data Science, 60 Fifth Avenue (7th floor common area). Expanding from its original emphasis on text data applied to social science applications, the Series incorporates the growing interest in Natural Language Processing from a variety of disciplines, especially Computer Science and Linguistics. The series provides an opportunity for attendees to see cutting-edge NLP and other text-as-data work from the fields of social science, computer science and other related disciplines.  The seminar is organized by Professors Arthur Spirling, Sam Bowman, He He, and Tal Linzen, and is open to anyone who is affiliated with NYU and wishes to attend.

See below for our schedule of speakers and sign up for our mailing list for details about live-streaming of talks and updates to the schedule.

Note: All talks are currently scheduled to be in-person but this could change with updated guidance from NYU. The schedule listed below is tentative. Please return to this page or join the mailing list for updates in case of unforeseen circumstances. Live talk attendance is limited, but recordings will be posted on this page and may be accessed by anyone.

Spring 2023

  • March 30
    • Sasha (Alexander) Rush
      • Title: “Pretraining without Attention”
      • Abstract: Transformers are essential to pretraining. As we approach 5 years of BERT, the connection between attention as architecture and transfer learning remains key to this central thread in NLP. Other architectures such as CNNs and RNNs have been used to replicate pretraining results, but these either fail to reach the same accuracy or require supplemental attention layers. This work revisits the seminal BERT result and considers pretraining without attention. We consider replacing self-attention layers with recently developed approach for long-range sequence modeling and transformer architecture variants. Specifically, inspired by recent papers like the structured space space sequence model (S4), we use simple routing layers based on state-space models (SSM) and a bidirectional model architecture based on multiplicative gating. We discuss the results of the proposed Bidirectional Gated SSM (BiGS) and present a range of analysis into its properties. Results show that architecture does seem to have a notable impact on downstream performance and a different inductive bias that is worth exploring further.
      • Bio: Alexander “Sasha” Rush is an Associate Professor at Cornell Tech. His work is at the intersection of natural language processing and generative modeling with applications in text generation, efficient inference, and controllability. He has written several popular open-source software projects supporting NLP research and data science, and works part-time as a researcher at Hugging Face. He is the secretary of ICLR and developed software used to run virtual conferences during COVID. His work has received paper and demo awards at major NLP, visualization, and hardware conferences, an NSF Career Award, and a Sloan Fellowship. He tweets and blogs, mostly about coding and ML, at @srush_nlp.
  • April 6
    • Ellie Pavlick
      • Title: “Mechanisms for Compositionality in Neural Networks”
      • Abstract: In this talk, Pavlick will discuss new work which attempts to describe the “algorithms” that neural networks (NNs) implement implicitly, as a result of their training. Pavlick will focus specifically on NNs ability to encode abstract, compositional functions which consist of interpretable subroutines and which operate in a content-independent manner. Pavlick will discuss findings on models trained from scratch on language and vision tasks, as well as large language models (LLMs) in an in-context-learning setting.
      • Bio: Ellie Pavlick is an Assistant Professor of Computer Science at Brown University. She received her PhD from University of Pennsylvania in 2017, where her focus was on paraphrasing and lexical semantics. Ellie’s research is on cognitively-inspired approaches to language acquisition, focusing on grounded language learning and on the emergence of structure (or lack thereof) in neural language models. Ellie leads the language understanding and representation (LUNAR) lab, which collaborates with Brown’s Robotics and Visual Computing labs and with the Department of Cognitive, Linguistic, and Psychological Sciences.
      • Recording: Ellie Pavlick Lecture Recording
  • April 13
    • Mohit Iyyer
      • Title: On using large language models to translate literary works, and detecting when they’ve been used
      • Abstract: In this presentation, Mohit will talk about two projects that are only tangentially related (hence the awkward title!). First, he’ll share my lab’s experiences on working with large language models to translate literary texts (e.g., novels). Mohit and his team’s research in this direction includes (1) efforts to develop human and automatic evaluations of the quality of AI-generated literary translations, which allow us to develop a better understanding of the types of errors made by these systems; and (2) work on building a publicly-accessible platform for readers of AI-generated translation, and extensions to collaborative human-AI translation. In the second part of the talk, Mohit will focus on detecting when a piece of text has been generated by a language model. Mohit and his team evaluate several existing AI-generated text detectors (e.g., watermarking, DetectGPT, etc.) and discover that they are vulnerable to paraphrase attacks: simply passing text generated by ChatGPT through an external paraphrasing model is enough to fool current detectors. Mohit and his team propose a retrieval-based detection algorithm that proves more robust against paraphrasing attacks, but also has its own limitations.
      • Bio: Mohit Iyyer is an assistant professor in computer science at the University of Massachusetts Amherst. His research focuses broadly on designing machine learning models for discourse-level language generation (e.g., for story generation and machine translation), and his group also works on tasks involving creative language understanding (e.g., modeling fictional narratives and characters). He is the recipient of best paper awards at NAACL (2016, 2018) and a best demo award at NeurIPS 2015, and he received the 2022 Samsung AI Researcher of the Year award. He received his PhD in computer science from the University of Maryland, College Park in 2017, advised by Jordan Boyd-Graber and Hal Daumé III, and spent the following year as a researcher at the Allen Institute for Artificial Intelligence.
      • Recording: Mohit Iyyer Lecture Recording
  • April 20
    • Asli Celikyilmaz
      • Title: Exploring Machine Thinking
      • Abstract: As language models continue to evolve and become more sophisticated, it will be important to continually evaluate and improve their ability to reason effectively in context to ensure their reliability and accuracy. In this talk, I will present some of our recent work to help models learn such skills, including exploring machine thinking in in-context learning, assessing the correctness of reasoning steps, prompt chaining for compositionally and refinement, and meta-finetuning to assess reasoning abilities. By examining these works, we’ll gain insights into some of the current systems and discover opportunities for future research.
      • Bio: Asli Celikyilmaz is a Research Manager at Fundamentals AI Research (FAIR) Labs at Meta AI in Seattle. Formerly, she was Senior Principal Researcher at Microsoft Research (MSR) in Redmond, Washington. She is also an Affiliate Associate Member at the University of Washington. She has received a Ph.D. Degree in Information Science from University of Toronto, Canada, and later continued her Postdoc study at the Computer Science Department of the University of California, Berkeley. Her research interests include machine learning and natural language processing, with a focus on areas such as machine teaching, long-form reasoning and evaluation, in-context learning, language generation and evalution, to name a few. She is serving as the Editor-in-Chief of the Transactions of the ACL (TACL) and Associte Editor  on Open Journal of Signal Processing (OJSP). She has received several “best of” awards including Semantic Computing in 2009, and CVPR in 2019.
      • Slides: Asli Celikyilmaz Lectures Slides
  • April 27
    • Jacob Andreas
      • Title: Language Models as World Models
      • Abstract: The extent to which language modeling induces representations of the world outside text—and the broader question of whether it is possible to learn about meaning from text alone—have remained a subject of ongoing debate across NLP and cognitive sciences. I’ll present two studies from my lab showing that transformer language models encode structured and manipulable models of situations in their hidden representations. I’ll begin by presenting evidence from semantic probing indicating that LM representations of entity mentions encode information about entities’ dynamic state, and that these state representations are causally implicated downstream language generation. Despite this, even today’s largest LMs are prone to glaring semantic errors: they hallucinate facts, contradict input text, or even their own previous outputs. Building on our understanding of how LMs build models of entities and events, I’ll present a representation editing model called REMEDI that can correct these errors directly in an LM’s representation space, in some cases making it possible to generate output that cannot be produced with a corresponding textual prompt, and to detect incorrect or incoherent output before it is generated.
      • Bio: Jacob Andreas is the X Consortium Assistant Professor at MIT. His research aims to build intelligent systems that can communicate effectively using language and learn from human guidance. Jacob earned his Ph.D. from UC Berkeley, his M.Phil. from Cambridge (where he studied as a Churchill scholar) and his B.S. from Columbia. As a researcher at Microsoft Semantic Machines, he founded the language generation team and helped develop core pieces of the technology that powers conversational interaction in Microsoft Outlook. He has been named a Samsung AI Researcher of the Year and National Academy of Sciences Kavli Fellow, and has received the NSF CAREER award, MIT’s Junior Bose and Kolokotrones teaching awards, and paper awards at NAACL and ICML.
      • Recording: Jacob Andreas Lecture Recording (captions coming soon)
      • Slides: Jacob Andreas Lecture Slides

Subscribe to our mailing list

* indicates required



 

Request to attend

HAVE A VALID NYU ID?

You may attend the Speaker Series without requesting approval!

DON’T HAVE A VALID NYU ID?

Please see above about livestreaming/poststreaming the talks! We are not able to approve external requests at this time.

Previous speakers

Fall 2022

Spring 2022

Fall 2021

Spring 2021

Fall 2020

Spring 2020

* No meetings from 20 Feb to 9 April *

  • 16 Apr: Cancelled
  • 23 Apr: Cancelled
  • 30 Apr: Cancelled
  • 7 May: Cancelled

Fall 2019

Spring 2019

Fall 2018

Spring 2018

Fall 2017

Summer 2017

  • 26-Jul Yejin Choi (UW) — From Naive Physics to Connotation: Learning about the World from Language

Spring 2017

Fall 2016

Spring 2016

Fall 2015