---
library_name: transformers
tags: []
---

# Model Card for ChatSum

<!-- Provide a quick summary of what the model is/does. -->

This page containes the model for the ChatSum application developed for the Machine Learning Operations course taught by Professor Barbon Junior Sylvio at the University of Trieste for the
Artificial Intelligence & Data Science masters.

## Model Details

The model is based on _google/pegasus-cnn_dailymail_. Our work consisted in  fine tuning the pegasus model on the _samsum dataset_. The objective was to
improve the existing model performance for the task of summarizing "chat-like" behaviour.

### Model Description

<!-- Provide a longer summary of what this model is. -->

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

- **Developed by:** [Erion Islamay](https://github.com/Erionis), [Cortinovis Nicola](https://github.com/NicolaCortinovis), [Paladino Annalisa](https://github.com/annalisapaladino), [Pernice Luca](https://github.com/lucapernice);
- **Language(s) (NLP):** English
- **Finetuned from model [optional]:** google/pegasus-cnn_dailymail

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

This model should be used to create concise summaries of dialogues, chats or correspondaces between two or more parties.


### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

We suggest passing the chats to summarize in the following format to obtain best results:

[Person_A: Some text
Person_B: Some other text
Person_A: Response
Person_A: Other text]

## How to Get Started with the Model

```python
from transformers import pipeline

summarizer = pipeline("summarization", model="Nicovis/ConvSum")


DIALOGUE = """
Linda: I'm going to have my room painted
Linda: Can't decide what colors would best express my personality
Linda: I'm considering advices, do you have one for me maybe? :)
Brian: Fitting your personality... hmm
Brian: I consider you an enthusiastic and lively person and the color for it is yellow
Brian: On the other hand you can be calm and steady and they say those qualities are expressed in greyness
Brian: So yellow & grey, how do you like the idea? :D
Linda: Wow, I'm impressed!
Linda: You've just paid me a compliment and come up with interesting colors!
Brian: Well, those are actually facts :)
Brian: Are you going to make use of the colors?
Linda: Actually, I was planning to ask a few friends and then sum all the ideas up
Linda: But now I think I don't need to do any further research
Linda: Asking you for help was a bull's-eye! :D
Brian: Perfection! :D
Brian: I'll come and check the results of your work soon, it'll be a nice chance for us to talk :)
Linda: Sure, feel invited!
"""

print(summarizer(DIALOGUE, max_length=64, min_length=30, do_sample=False))
>>> [{'summary_text': "Linda is going to have her room painted. Brian recommends yellow and grey colors for her personality. Linda is going to make use of the colors. Brian will come and check the results of Linda's work soon."}]

```


## Training Details

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

The training on the _SAMSum dataset_ was done using Seq2SeqTrainer with DataCollatorForSeq2Seq using PegasusFastTokenizer for both. The training arguments for the model were:

```python
Seq2SeqTrainingArguments(
  num_train_epochs= 5, 
  warmup_steps= 500,
  per_device_train_batch_size= 16,
  per_device_eval_batch_size= 16,
  weight_decay= 0.1,
  logging_steps= 10,
  evaluation_strategy= steps,
  eval_steps= 300, 
  save_steps= 1e6,
  gradient_accumulation_steps= 16
)
```

## Evaluation

Model was evaluated on cross entropy loss & ROUGE(1-2-L) scores during evaluation obtaining:

| Metric | Average at the end of training |
|--------|--------------------------------|
| ROUGE-1| 0.472 |
| ROUGE-2| 0.241 |
| ROUGE-L| 0.376 |
| Cross entropy loss | 1.4 |


### Testing Metrics

The results were tested on the _SAMSum dataset_ testing set on the following metrics: ROUGE(1-2-L) and BERTScore

## Results

We obtained the following results

| Metric | Average at the of testing |
|--------|--------------------------------|
| ROUGE-1| 0.496 |
| ROUGE-2| 0.250 |
| ROUGE-L| 0.400 |
| BERTScore | 0.686 |