NLP 280: Seminar in Natural Language Processing

NLP 280 is a seminar course that features talks from industry experts in the natural language processing (NLP) and artificial intelligence (AI) areas.

The speaker schedule may change without notice, due to changes in speaker availability.

Titles, abstracts, and speaker bios will be made available as the talk date approaches.

Some seminar slots do not have a speaker. Instead, the seminar time will be used for discussion.

The seminar meets weekly on Friday at 2:40 PM.

  • 10/2/20: NLP MS Orientation Part II
  • Date
    10/9/20
    Speaker
    Zornitsa Kozareva (Google)
    Title
    Fast and Small On-device Neural Networks for Natural Language Understanding & Conversational AI
    Abstract
    Deep neural networks reach state-of-the-art performance for a wide range of Natural Language Understanding, Computer Vision and Speech applications. Yet, one of the biggest challenges is to run these complex networks on devices with tiny memory footprint and low computational capacity such as mobile phones, smart watches and Internet of Things.

    In this talk, I will introduce novel on-device Self-Governing Neural Networks (SGNNs), which dynamically learn compact projection vectors with local sensitive hashing. The key advantage of SGNNs over existing work is that they surmount the need for pre-trained word embeddings and complex networks with huge parameters. I will showcase results from extensive evaluations on a wide range of natural language tasks such as dialog act classification, user intent prediction, customer feedback and for various languages such as English, Japanese, Spanish and French. Our findings show that SGNNs are effective at capturing low-dimensional semantic text representations, while maintaining high accuracy and improving over state-of-the-art results. This work laid the foundation for the next generation efficient neural models for on-device Natural Language Understanding.

    Next, I will introduce the next generation neural networks SGNN++ that further improve efficiency by using structured and context-dependent partition projections. Through a series of ablation studies, I will show the impact of the partitioned projections and structured information leading to 10% quality improvements. I will also highlight the impact of the model size on accuracy and introduce quantization-aware training for SGNN++ to further reduce the model size, while preserving the same quality. Finally, I will discuss fast inference for mobile phones and touch upon our latest developments in on-device NLP called ProSeqo and ProFormer for long sequences.

    Bio
    Dr. Zornitsa Kozareva is a Manager at Google, leading and managing Apps AI group and ML/NLP efforts in Google. Prior to that, Dr. Kozareva was a Manager of Amazon’s AWS Deep Learning group that built and launched the Natural Language Processing and Dialog services Amazon Comprehend and Amazon Lex. Dr. Kozareva was a Senior Manager at Yahoo! leading the Query Processing group that powered Mobile Search and Advertisement. From 2009 to 2014, Dr. Kozareva wore an academic hat as a Research Professor at the University of Southern California CS Department with affiliation to Information Sciences Institute, where she spearheaded research funded by DARPA and IARPA on learning to read, interpreting metaphors and building knowledge bases from the Web. Dr. Kozareva regularly serves as Senior Area Chair and PC of top tier NLP and ML conferences such as ACL, NAACL, EMNLP, WSDM, AAAI. She is the Co-Chair for ICML 2019 On-device ML and EMNLP 2020 & NAACL 2019 Structured prediction workshops. Dr. Kozareva has organized four SemEval scientific challenges and has published over 80 research papers. Her work has been featured in the press such as Forbes, VentureBeat, GizBot, NextBigWhat. Dr. Kozareva was invited speaker for 2019 National Academy of Engineering (NAE) German-American Frontiers of Engineering symposium. Dr. Kozareva is a recipient of the John Atanasoff Award given by the President of the Republic of Bulgaria in 2016 for her contributions and impact in science, education and industry; the Yahoo! Labs Excellence Award in 2014 and the RANLP Young Scientist Award in 2011.
  • Date
    10/16/20
    Speaker
    Ananth Sankar (LinkedIn)
    Title
    Deep neural networks for search and recommendation systems at LinkedIn
    Abstract
    Deep neural networks, like convolutional neural networks (CNN), recurrent neural networks (RNN), and attention-based encoder-decoder networks have made a big impact in several natural language processing (NLP) applications, such as sentence classification, part of speech tagging, and machine translation. In recent years, transfer learning methods using models like BERT and its variants have improved the state of the art in NLP through contextual word embeddings, and sentence embeddings.

    In this talk, I will give a high-level overview of transfer learning in NLP using deep neural networks, and give examples of its successful use in search and recommendation systems at LinkedIn.

    Bio
    Ananth Sankar is a Principal Staff Engineer in the Artificial Intelligence group at LinkedIn, where he works on multimedia content understanding and natural language processing. During his career, he has also made many R&D contributions in the area of speech recognition. He has taught courses at Stanford and UCLA, given several invited talks, co-authored more than 50 refereed publications, and has 10 accepted patents.
  • Date
    10/23
    Speaker
    Ciprian Chelba (Google)
    Title
    Language Modeling in the Era of Abundant Data
    Abstract
    https://research.google/pubs/pub46011/

    The talk will present an overview of established and more recent methods for building language models in the context of abundant training data. Modeling at scale, practical applications, remaining open problems will be highlighted.

    Bio

    Ciprian Chelba is Staff Research Scientist at Google Research; prior he worked as a Research Scientist on the speech recognition team in Microsoft Research. Ciprian received his PhD in Electrical Engineering from the Center of Language and Speech Processing in 2000 under the advising of Prof. Frederick Jelinek.
    Ciprian's research interests are in statistical language modeling for automatic speech recognition and machine translation, with a wider interest in computational linguistics, information theory and statistics, and machine learning.
    Besides being an active contributor to the academic research community (see Google Scholar: https://scholar.google.com/citations?user=Rtg5ZY8AAAAJ&hl=en), Ciprian is also interested in real-life applications of the above technologies. Ciprian is proud to have been part of the team that launched the Google voice search prototype in 2007. He is listed as co-inventor on more than 50 US patents, many of them filed internationally.

  • 10/30: Class discussion.
  • 11/6: Daniel Preotiuc-Pietro (Bloomberg)
  • 11/13: Bing Liu (FaceBook)
  • 11/20: Yunyao Li (IBM)
  • 11/27: No speaker. Thanksgiving Holiday
  • 12/4: Xiao Bai (Verizon Media)
  • 12/11: Class discussion.