7 Applications of Deep Learning for Natural Language Processing

The field of natural language processing is shifting from statistical methods to neural network methods.

There are still many challenging problems to solve in natural language. Nevertheless, deep learning methods are achieving state-of-the-art results on some specific language problems.

It is not just the performance of deep learning models on benchmark problems that is most interesting; it is the fact that a single model can learn word meaning and perform language tasks, obviating the need for a pipeline of specialized and hand-crafted methods.

In this post, you will discover 7 interesting natural language processing tasks where deep learning methods are achieving some headway.

Let’s get started.

7 Applications of Deep Learning for Natural Language Processing
Photo by Tim Gorman, some rights reserved.

Overview

In this post, we will look at the following 7 natural language processing problems.

  1. Text Classification
  2. Language Modeling
  3. Speech Recognition
  4. Caption Generation
  5. Machine Translation
  6. Document Summarization
  7. Question Answering

I have tried to focus on the types of end-user problems that you may be interested in, as opposed to more academic or linguistic sub-problems where deep learning does well such as part-of-speech tagging, chunking, named entity recognition, and so on.

Each example provides a description of the problem, an example, and references to papers that demonstrate the methods and results. Most references are drawn from Goldberg’s excellent 2015 primer on deep learning for NLP researchers.

Do you have a favorite NLP application for deep learning that is not listed?
Let me know in the comments below.

1. Text Classification

Given an example of text, predict a predefined class label.

The goal of text categorization is to classify the topic or theme of a document.

— Page 575, Foundations of Statistical Natural Language Processing, 1999.

A popular classification example is sentiment analysis where class labels represent the emotional tone of the source text such as “positive” or “negative“.

Below are a 3 more examples:

  • Spam filtering, classifying email text as spam or not.
  • Language identification, classifying the language of the source text.
  • Genre classification, classifying the genre of a fictional story.

Further, the problem may be framed in a way that requires multiple classes assigned to a text, so-called multi-label classification. Such as predicting multiple hashtags for a source tweet.

For more on the general topic, see:

Below are 3 examples of deep learning papers for text classification:

  • Sentiment analysis of rotten tomato movie reviews.
  • Sentiment analysis of amazon product reviews, IMDB movie reviews and topic categorization of news articles.
  • Sentiment analysis of movie reviews, classifying sentences as being subjective or objective, classifying question types, sentiment of product reviews and more.

2. Language Modeling

Language modeling is really a subtask of more interesting natural language problems, specifically those that condition the language model on some other input.

… the problem is to predict the next word given the previous words. The task is fundamental to speech or optical character recognition, and is also used for spelling correction, handwriting recognition, and statistical machine translation.

— Page 191, Foundations of Statistical Natural Language Processing, 1999.

In addition to the academic interest in language modeling, it is a key component of many deep learning natural language processing architectures.

A language model learns the probabilistic relationship between words such that new sequences of words can be generated that are statistically consistent with the source text.

Alone, language models can be used for text or speech generation; for example:

  • Generating new article headlines.
  • Generating new sentences, paragraphs, or documents.
  • Generating suggested continuation of a sentence.

For more in language modeling, see:

Below is an example of deep learning for language modeling (only):

  • Language model of English texts, books and news articles.

3. Speech Recognition

Speech recognition is the problem of understanding what was said.

The task of speech recognition is to map an acoustic signal containing a spoken natural language utterance into the corresponding sequence of words intended by the speaker.

— Page 458, Deep Learning, 2016.

Given an utterance of text as audio data, the model must produce human readable text.

Given the automatic nature of the process, the problem may also be called Automatic Speech Recognition (ASR).

A language model is used to create the text output that is conditioned on the audio data.

Some examples include:

  • Transcribing a speech.
  • Creating text captions for a movie or TV show.
  • Issuing commands to the radio while driving.

For more on speech recognition, see:

Below are 3 examples of deep learning for speech recognition.

  • English speech to text.
  • English speech to text.
  • English speech to text.

4. Caption Generation

Caption generation is the problem of describing the contents of an image.

Given a digital image, such as a photo, generate a textual description of the contents of the image.

A language model is used to create the caption that is conditioned on the image.

Some examples include:

  • Describing the contents of a scene.
  • Creating a caption for a photograph.
  • Describing a video.

This is not just an application for the hearing impaired, but also in generating human readable text for image and video data that can be searched, such as on the web.

Below are 3 examples of deep learning for caption generation:

  • Generating captions for photographs.
  • Generating captions for photographs.
  • Generating captions for videos.

5. Machine Translation

Machine translation is the problem of converting a source text in one language to another language.

Machine translation, the automatic translation of text or speech from one language to another, is one [of] the most important applications of NLP.

— Page 463, Foundations of Statistical Natural Language Processing, 1999.

Given that deep neural networks are used, the field is referred to as neural machine translation.

In a machine translation task, the input already consists of a sequence of symbols in some language, and the computer program must convert this int a sequence of symbols in another language. This is commonly applied to natural languages, such as translating from English to French. Deep learning has recently begun to have an important impact on this kind of task.

— Page 98, Deep Learning, 2016.

A language model is used to output the destination text in the second language, conditioned on the source text.

Some examples include:

  • Translating a text document from French to English.
  • Translating Spanish audio to German text.
  • Translating English text to Italian audio.

For more on neural machine translation, see:

Below are 3 examples of deep learning for machine translation:

  • Translation of text from English to French.
  • Translation of text from English to French.
  • Translation of text from English to French.

6. Document Summarization

Document summarization is the task where a short description of a text document is created.

As above, a language model is used to output the summary conditioned on the full document.

Some examples of document summarization include:

  • Creating a heading for a document.
  • Creating an abstract of a document.

For more on the topic, see:

Below are 3 examples of deep learning for document summarization:

  • Summarization of sentences in news articles.
  • Summarization of sentences in news articles.
  • Summarization of sentences in news articles.

7. Question Answering

Question answering is the problem where given a subject, such as a document of text, answer a specific question about the subject.

… question answering systems which try to answer a user query that is formulated in the form of a question by return the appropriate none phrase such as a location, a person, or a date. For example, the question Why killed President Kennedy? might be answered with the noun phrase Oswald

— Page 377, Foundations of Statistical Natural Language Processing, 1999.

Some examples include:

For more information on question answering, see:

  • Answering questions about Wikipedia articles.
  • Answering questions about news articles.
  • Answering questions about medical records.

Below are 3 examples of deep learning for question answering:

  • Answering questions about news articles.
  • Answering general knowledge questions over freebase articles.
  • Answers factoid questions given specific documents.

Further Reading

This section provides more resources on deep learning applications for NLP if you are looking go deeper.

Summary

In this post, you discovered 7 applications of deep learning to natural language processing tasks.

Was your favorite example of deep learning for NLP missed?
Let me know in the comments.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.