This repository maintains dataset, metrics, and models for TACL 2023 paper MACSum: Controllable Summarization with Mixed Attributes.

Overview

MACSum is the first human-annotated summarization dataset for controlling mixed attributes. It contains source texts from two domains, news articles and dialogues, with human-annotated summaries controlled by five designed attributes (Length, Extractiveness, Specificity, Topic, and Speaker).

Here is an example of MACSum. For the same input source text, the system needs to generate different reference summaries (green boxes) for different mixed control attributes (orange boxes).

Dataset

Statistics

Statistics of MACSum consist of two parts: MAC-Doc from CNNDM and MAC-Dial from QMSum. Source Len., Ref. Len. are tokens in source and reference. Topic, Speaker are averaged number of topics/speakers.

Json Files

One can access the train/valid/test set of MACSum via dataset folder which includes macidal for dialogue domain and macdoc for document domain. The format of one instance in json data is as follows:

{
    "metadata": {          // source text information, one can use metadata to locate the original text
      "dataset": "cnndm",  // origin dataset of source text
      "split": "test",     // split of origin dataset
      "file_id": 339       // id of this split
    },
    "source": [ ... ]      // source text split by sentences/turns in macdoc/macdial dataset
    "references": [        // multiple samples containing the same source with different control_attributes
      {
        "control_attribute": {       // the same as control attributes in paper
          "length": "short",
          "extractiveness": "normal",
          "specificity": "normal",
          "topic": "",
          "Speaker": ""             // only macdial has this field
        },
        "summary": "The 2014 Khumbu Icefall on Everest caused 16 Nepali deaths last year, resulting in at the end of the Nepal season. Mountaineers return to Everest after the deadly 2014 season. To avoid the deadly area, changes were made to the routes. Climbing permits have increased this year.",
        "title": "Nepal Opens Mount Everest Side After 2014 Tragedy",  // title of the summary
        "relevant_turns": [   // two formats: 1) [start, end] (inclusive), or 2) a single number for that turn
          [
            0,
            49
          ],
          55
        ]
      },
      ...
    ]
}

Metrics

We propose two automatic metrics for evaluating the degree of control of generated summary: Control Error Rate (CER) & Control Correlation (CC)

One can find the implementation in metric folder:

• control_correlation_dial/doc.py computes the CC
• control_error_rate_dial/doc.py computes the CER
• {attr}_only.py compute the control attribute metric for single {attr}, corresponding to the