Learning Representations for Text-level Discourse Parsing

PDTB-style examples

He added [that "having just one firm do this isn't going to mean a hill of beans]_arg1.

But [if this prompts others to consider the same thing, then it may become much more important]_arg2."

— PDTB-style, id: 14904, type: explicit, sense: Comparison.Concession

In addition, Black & Decker had said it would sell two other undisclosed Emhart operations if it received the right price. [Bostic is one of the previously unnamed units, and the first of the five to be sold.]_arg1

[The company is still negotiating the sales of the other four units and expects to announce agreements by the end of the year]_arg1. [The five units generated sales of about $1.3 billion in 1988, almost half of Emhart's $2.3 billion revenue]_arg2. Bostic posted 1988 sales of $255 million.

— PDTB-style, id: 12886, type: entrel, sense: EntRel

PDTB-style discourse parsing

• Penn Discourse Treebank adopts the predicate-argument view and independence of discourse relations

  • 2159 articles from the Wall Street Journal
  • 4 discourse sense classes, 16 types, 23 subtypes

• also called shallow discourse parsing

  • discourse relations are not connected to each another to form a connected structure (tree or graph)
  • adjacent/non-adjacent units in same/different sentences

• primary goals

  • locate explicit or implicit discourse connective
  • locate text spans for argument 1 and 2
  • predict sense that characterizes the nature of the relation

Sequence processing

Text documents of different lengths are usually treated as a sequence of words:

• transition-based processing mechanisms
• recurrent neural networks (RNNs)
  • applying the same set of weights over the sequence (temporal dimension) or structure (tree-based)

Word embeddings

Represent text as numeric vectors of fixed size:

• word embeddings: SGNS (word2vec), GloVe, ...
• feature/phrase/document embeddings
• character-level convolutional networks

Unsupervised pre-training helps develop natural abstractions.

Sharing word embedding in multi-task learning improves their performance in the absence of hand-engineered features.

Key ideas

• unified end-to-end architecture
  • backpropagation as the one learning algorithm for all discourse parsing subtasks and related NLP tasks
• automatic learning of representations
  • in hidden layers of deep learning architectures (bidirectional deep RNN/LSTM)
• shared intermediate representations
  • partially stacked on top of each other to benefit from each others representations
• guided layer-wise multi-task learning
  • jointly learning all discourse parsing subtasks and related NLP tasks including unsupervised pre-training

Guided layer-wise multi-task learning

Complication or useful?

Experiments with single-task learning with bidirectional deep RNN for discourse sense tagging:

Single-task results

• long training time for randomly initialized weights
  • lower tasks improve initialization
• overfitting training data
  • more tasks improve generalization

Future experiments

• various discourse parsing subtasks
• various related NLP tasks (chunking, POS, NER, SRL, ...)
• different representation structures
• different activation, optimization, architectures
• long short-term memory (LSTM)
• neural Turing machines (NTM)