metadata
library_name: transformers
license: mit
base_model: facebook/bart-large-cnn
tags:
- generated_from_trainer
metrics:
- rouge
model-index:
- name: bart-large-cnn-finetuned
results:
- task:
type: summarization
name: Summarization
dataset:
name: billsum
type: billsum
config: 3.0.0
split: train
metrics:
- name: ROUGE-1
type: rouge
value: 51.9605
verified: true
- name: ROUGE-2
type: rouge
value: 20.8149
verified: true
- name: ROUGE-L
type: rouge
value: 36.2784
verified: true
- name: ROUGE-LSUM
type: rouge
value: 47.1043
verified: true
- name: loss
type: loss
value: 1.1553
verified: true
- name: gen_len
type: gen_len
value: 63.9903
verified: true
pipeline_tag: summarization
bart-large-finetuned-billsum
This model is a fine-tuned version of facebook/bart-large-cnn on a FiscalNote/Billsum dataset. It achieves the following results on the evaluation set:
- Loss: 1.1553
- Rouge1: 51.9605
- Rouge2: 36.2784
- Rougel: 44.1511
- Rougelsum: 47.1043
- Gen Len: 63.9903
Model description
More information needed
Intended uses & limitations
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1000
- num_epochs: 3
- mixed_precision_training: Native AMP
Training results
| Train Loss | Step | Val Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Gen Len |
|---|---|---|---|---|---|---|---|
| 1.4735 | 1000 | 1.3306 | 50.6543 | 33.9684 | 42.2550 | 45.4452 | 63.9983 |
| 1.3146 | 2000 | 1.2376 | 51.0888 | 34.9554 | 42.9847 | 45.8933 | 63.9903 |
| 1.1542 | 3000 | 1.1874 | 51.5755 | 35.6875 | 43.6806 | 46.5762 | 63.9800 |
| 1.0917 | 4000 | 1.1714 | 51.8612 | 36.1809 | 44.0608 | 47.0279 | 63.9870 |
| 1.0380 | 5000 | 1.1553 | 51.9605 | 36.2784 | 44.1511 | 47.1043 | 63.9903 |
from transformers import pipeline
summarizer = pipeline("summarization", model="luluw/bart-large-cnn-finetuned")
text = """
The paper "Attention is All You Need" revolutionized the field of natural language processing (NLP) by introducing the Transformer architecture, which relies solely on attention mechanisms to model long-range dependencies in sequential data. Prior to this, models like recurrent neural networks (RNNs) and convolutional neural networks (CNNs) were the primary tools for sequence modeling, but they suffered from limitations such as difficulty in parallelization and the vanishing gradient problem. The Transformer, however, breaks free from these constraints by using a self-attention mechanism, which allows it to attend to different parts of a sequence simultaneously, leading to more efficient training and better performance on tasks such as machine translation, text summarization, and language modeling.
The core innovation of the Transformer model lies in its multi-head self-attention mechanism. Unlike RNNs that process sequences step-by-step, the Transformer processes the entire sequence at once by applying self-attention to every word or token. This allows each token to weigh the relevance of other tokens in the sequence, giving the model a global understanding of context. Multi-head attention refers to applying multiple attention layers in parallel, enabling the model to focus on different parts of the input sequence simultaneously. This enhances the model's ability to capture various relationships and nuances in the data.
The Transformer consists of an encoder-decoder structure. The encoder takes in the input sequence, computes self-attention to understand relationships between tokens, and generates a context-aware representation. The decoder, which also incorporates self-attention, generates the output sequence one token at a time by attending to both the previously generated tokens and the encoder's output. This architecture, coupled with position-wise feed-forward networks and layer normalization, makes the Transformer highly scalable and efficient.
Another significant contribution of the paper is the introduction of positional encoding. Since the Transformer lacks the inherent sequential nature of RNNs, it cannot infer the order of tokens from the architecture itself. To overcome this, the authors introduced positional encodings, which are added to the input embeddings to provide the model with information about the relative position of tokens. These encodings allow the model to maintain a sense of order in the data without explicitly processing tokens sequentially.
The original Transformer model proposed in Attention is All You Need had six layers each in both the encoder and decoder. Each layer consists of multi-head attention and feed-forward layers, with residual connections and normalization. The model was trained using the Adam optimizer and applied to machine translation tasks, where it demonstrated state-of-the-art performance, surpassing previous models like LSTMs and GRUs.
One of the key benefits of the Transformer is its ability to parallelize training, as it does not rely on sequential data processing like RNNs. This parallelism allows it to leverage modern GPU architectures effectively, leading to faster training times and the ability to scale to much larger datasets. Furthermore, Transformers handle long-range dependencies better than previous models because self-attention allows every token to interact with every other token in the sequence, regardless of their distance from each other.
"""
print(summarizer(text, max_new_tokens=128)[0]['generated_text'])
>> Attention is All You Need is a paper that revolutionized the field of natural language processing (NLP) by introducing the Transformer architecture, which relies solely on attention mechanisms to model long-range dependencies in sequential data. The Transformer consists of an encoder-decoder structure: the encoder takes in the input sequence, computes self-attention to understand relationships between tokens, and generates a context-aware representation; and the decoder generates the output sequence one token at a time by attending to both the previously generated tokens and encoder output.
Framework versions
- Transformers 4.44.2
- Pytorch 2.2.1+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1