NLP 280: Seminar in Natural Language Processing

NLP 280 is a seminar course that features talks from industry experts in the natural language processing (NLP) and artificial intelligence (AI) areas.

The speaker schedule may change without notice, due to changes in speaker availability.

Titles, abstracts, and speaker bios will be made available as the talk date approaches.

Some seminar slots do not have a speaker. Instead, the seminar time will be used for discussion.

The seminar meets weekly on Friday at 2:40 PM.

  • 10/2/20: NLP MS Orientation Part II
  • Date
    10/9/20
    Speaker
    Zornitsa Kozareva (Google)
    Title
    Fast and Small On-device Neural Networks for Natural Language Understanding & Conversational AI
    Abstract
    Deep neural networks reach state-of-the-art performance for a wide range of Natural Language Understanding, Computer Vision and Speech applications. Yet, one of the biggest challenges is to run these complex networks on devices with tiny memory footprint and low computational capacity such as mobile phones, smart watches and Internet of Things.

    In this talk, I will introduce novel on-device Self-Governing Neural Networks (SGNNs), which dynamically learn compact projection vectors with local sensitive hashing. The key advantage of SGNNs over existing work is that they surmount the need for pre-trained word embeddings and complex networks with huge parameters. I will showcase results from extensive evaluations on a wide range of natural language tasks such as dialog act classification, user intent prediction, customer feedback and for various languages such as English, Japanese, Spanish and French. Our findings show that SGNNs are effective at capturing low-dimensional semantic text representations, while maintaining high accuracy and improving over state-of-the-art results. This work laid the foundation for the next generation efficient neural models for on-device Natural Language Understanding.

    Next, I will introduce the next generation neural networks SGNN++ that further improve efficiency by using structured and context-dependent partition projections. Through a series of ablation studies, I will show the impact of the partitioned projections and structured information leading to 10% quality improvements. I will also highlight the impact of the model size on accuracy and introduce quantization-aware training for SGNN++ to further reduce the model size, while preserving the same quality. Finally, I will discuss fast inference for mobile phones and touch upon our latest developments in on-device NLP called ProSeqo and ProFormer for long sequences.

    Bio
    Dr. Zornitsa Kozareva is a Manager at Google, leading and managing Apps AI group and ML/NLP efforts in Google. Prior to that, Dr. Kozareva was a Manager of Amazon’s AWS Deep Learning group that built and launched the Natural Language Processing and Dialog services Amazon Comprehend and Amazon Lex. Dr. Kozareva was a Senior Manager at Yahoo! leading the Query Processing group that powered Mobile Search and Advertisement. From 2009 to 2014, Dr. Kozareva wore an academic hat as a Research Professor at the University of Southern California CS Department with affiliation to Information Sciences Institute, where she spearheaded research funded by DARPA and IARPA on learning to read, interpreting metaphors and building knowledge bases from the Web. Dr. Kozareva regularly serves as Senior Area Chair and PC of top tier NLP and ML conferences such as ACL, NAACL, EMNLP, WSDM, AAAI. She is the Co-Chair for ICML 2019 On-device ML and EMNLP 2020 & NAACL 2019 Structured prediction workshops. Dr. Kozareva has organized four SemEval scientific challenges and has published over 80 research papers. Her work has been featured in the press such as Forbes, VentureBeat, GizBot, NextBigWhat. Dr. Kozareva was invited speaker for 2019 National Academy of Engineering (NAE) German-American Frontiers of Engineering symposium. Dr. Kozareva is a recipient of the John Atanasoff Award given by the President of the Republic of Bulgaria in 2016 for her contributions and impact in science, education and industry; the Yahoo! Labs Excellence Award in 2014 and the RANLP Young Scientist Award in 2011.
  • Date
    10/16/20
    Speaker
    Ananth Sankar (LinkedIn)
    Title
    Deep neural networks for search and recommendation systems at LinkedIn
    Abstract
    Deep neural networks, like convolutional neural networks (CNN), recurrent neural networks (RNN), and attention-based encoder-decoder networks have made a big impact in several natural language processing (NLP) applications, such as sentence classification, part of speech tagging, and machine translation. In recent years, transfer learning methods using models like BERT and its variants have improved the state of the art in NLP through contextual word embeddings, and sentence embeddings.

    In this talk, I will give a high-level overview of transfer learning in NLP using deep neural networks, and give examples of its successful use in search and recommendation systems at LinkedIn.

    Bio
    Ananth Sankar is a Principal Staff Engineer in the Artificial Intelligence group at LinkedIn, where he works on multimedia content understanding and natural language processing. During his career, he has also made many R&D contributions in the area of speech recognition. He has taught courses at Stanford and UCLA, given several invited talks, co-authored more than 50 refereed publications, and has 10 accepted patents.
  • Date
    10/23
    Speaker
    Ciprian Chelba (Google)
    Title
    Language Modeling in the Era of Abundant Data
    Abstract
    https://research.google/pubs/pub46011/

    The talk will present an overview of established and more recent methods for building language models in the context of abundant training data. Modeling at scale, practical applications, remaining open problems will be highlighted.

    Bio

    Ciprian Chelba is Staff Research Scientist at Google Research; prior he worked as a Research Scientist on the speech recognition team in Microsoft Research. Ciprian received his PhD in Electrical Engineering from the Center of Language and Speech Processing in 2000 under the advising of Prof. Frederick Jelinek.
    Ciprian's research interests are in statistical language modeling for automatic speech recognition and machine translation, with a wider interest in computational linguistics, information theory and statistics, and machine learning.
    Besides being an active contributor to the academic research community (see Google Scholar: https://scholar.google.com/citations?user=Rtg5ZY8AAAAJ&hl=en), Ciprian is also interested in real-life applications of the above technologies. Ciprian is proud to have been part of the team that launched the Google voice search prototype in 2007. He is listed as co-inventor on more than 50 US patents, many of them filed internationally.

  • 10/30: Class discussion.
  • Date
    11/6
    Speaker
    Daniel Preotiuc-Pietro (Bloomberg)
    Title
    Applied Named Entity Recognition @ Bloomberg
    Abstract
    Bloomberg deals with a wealth of heterogeneous data, including (but not limited to): reports from financial analysts, earnings releases, company filings, news stories, web scrapes, social media posts, ticker symbols, and pricing information of a wide variety of securities. With 80% of financial data in unstructured formats, the ability to quickly identify named entities automatically is essential to understanding this content.

    I will briefly introduce Bloomberg's products that use this technology and present the unique challenges related to Bloomberg's data and business. I will then present three of our recent publications that aim to tackle the following challenges: heterogeneity of content, temporal data drift, and ensuring high precision in tagging.

    Bio
    Daniel Preotiuc-Pietro is a Senior Research Scientist and team lead at Bloomberg LP, where he works on analyzing and building models for real-world large scale news mining and information extraction. His research interests are focused on understanding the social and temporal aspects of text, especially from social media, with applications in domains such as Social Psychology, Law, Political Science and Journalism. Several of his research studies were featured in popular press including the Washington Post, BBC, New Scientist, Scientific American or FiveThirtyEight. He is a co-organizer of the Natural Legal Language Processing workshop series. Prior to joining Bloomberg LP, Daniel was a postdoctoral researcher at the University of Pennsylvania with the interdisciplinary World Well Being Project and obtained his PhD in Natural Language Processing and Machine Learning at the University of Sheffield, UK.
  • Date
    11/13
    Speaker
    Bing Liu (FaceBook)
    Title
    Advancing Spoken Dialog System with Deep Neural Networks
    Abstract
    Spoken dialog system is a prominent component in today’s virtual personal assistant, which enables people to perform everyday tasks by interacting with devices via voice interfaces. Recent advances in deep learning enabled new research directions for end-to-end neural network based dialog modeling. Such data-driven learning systems address many limitations of the conventional dialog systems but also introduce new challenges. In this talk, we will discuss recent advancement in spoken dialog systems with deep and reinforcement learning. We will further discuss how we can address the challenges on learning efficiency and scalability by combining large-scale offline training and online interactive learning with human-in-the-loop.
    Bio
    Bing Liu is a Research Scientist at Facebook working on conversational AI. His area of work focuses on machine learning for spoken language processing, natural language understanding, and dialog systems. He builds large scale conversational AI systems that learn continuously from user interactions with little human supervision via deep and reinforcement learning. Before joining Facebook, he was at Google Research working on end-to-end learning of neural dialog systems. Bing received his Ph.D. from Carnegie Mellon University where he worked on deep learning and reinforcement learning for task-oriented dialog systems.
  • Date
    11/20
    Speaker
    Yunyao Li and Kun Qian (IBM)
    Title
    Building Domain-Specific Knowledge Bases with Human in the loop
    Abstract
    The ability to build large-scale domain-specific knowledge bases that capture and extend the implicit knowledge of human experts is the foundation for many AI systems. In this talk, we describe the development of human-in-the-loop tools to capture the implicit knowledge in the mind of human experts to build such knowledge bases. We use an ontology-driven approach for the creation, representation and consumption of such domain-specific knowledge bases. This approach relies on several well-known building blocks: natural language processing, entity resolution, data transformation and fusion. We will present several human-in-the-loop work that target domain experts (rather than programmers) to extract the domain knowledge from the human expert and map it into the "right" models or algorithms. We will also share successful use cases in several domains, such as Compliance, Finance, and Healthcare: by using these tools we can match the level of accuracy achieved by manual effort, but at a significantly lower cost and much higher scale and automation. In particular, we will dive into the entity resolution component and talk about how we use human-in-the-loop techniques to create high-quality deep learning entity normalization and entity resolution models with low human effort.
    Bio:
    Yunyao Li is a Principal Research Staff Member and Senior Research Manager at IBM Almaden Research Center where she manages the Scalable Knowledge Intelligence (SKI) department. She is a member of the inaugural New Voices program of the American National Academies. She is also a Master Inventor and a member of IBM Academy of Technology. Her expertise is in the interdisciplinary areas of natural language processing, databases, human-computer interaction, and information retrieval. She is a founding member of SystemT, a state-of-the-art NLP system currently powering 20+ IBM products, and numerous research projects. She received her PhD and master degrees from the University of Michigan, Ann Arbor and undergraduate degrees from Tsinghua University, Beijing, China.

    Kun Qian is a Research Staff Member at IBM Almaden Research Center. He is a member of the Scalable Knowledge Intelligence (SKI) team. He joined IBM in 2017 after getting his PhD from UC Santa Cruz. His research interests have been focused on data exchange and data integration, and more recently, on human-in-the-loop machine learning. He is interested in learning high-quality models in low-resource scenarios using active learning, weak supervision, and specialized regularization techniques. His work has been published in top AI and database conferences such as AAAI, ACL, EMNLP, COLING, CIKM, VLDB, ICDE, PODS, ISWC, etc.

  • 11/27: No speaker. Thanksgiving Holiday
  • Date
    12/4
    Speaker
    Xiao Bai (Verizon Media)
    Title
    Neural keyphrase generation with syntactic guidance
    Abstract
    Generating a set of keyphrases that summarize the core ideas discussed in a document has a significant impact on many neural language processing and information retrieval applications, such as sentiment analysis, question answering, document retrieval, document categorization and contextual advertising. In recent years, deep neural sequence-to-sequence framework has demonstrated promising results in automatic keyphrase generation. In this talk, I will discuss challenges related to this task and introduce our recently developed neural keyphrase generation model. I will also report the results of our model on real-world datasets from both the scientific domain and the web domain. Finally, I will briefly discuss the application of the keyphrase generation model in contextual advertising.
    Bio
    XIAO BAI is a Principle Research Scientist at Yahoo Research. She received her PhD in Computer Science from INRIA, France. Her research is primarily focused on information retrieval, natural language understanding, and their applications in online advertising. Her contributions to various domains of research have been published in top venues where she regularly serves as PC member, such as SIGIR, CIKM, WWW and AAAI.
  • 12/11: Class discussion.