Model Card for ChatSum

This page containes the model for the ChatSum application developed for the Machine Learning Operations course taught by Professor Barbon Junior Sylvio at the University of Trieste for the Artificial Intelligence & Data Science masters.

Model Details

The model is based on google/pegasus-cnn_dailymail. Our work consisted in fine tuning the pegasus model on the samsum dataset. The objective was to improve the existing model performance for the task of summarizing "chat-like" behaviour.

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

Developed by: Erion Islamay, Cortinovis Nicola, Paladino Annalisa, Pernice Luca;
Language(s) (NLP): English
Finetuned from model [optional]: google/pegasus-cnn_dailymail

Uses

This model should be used to create concise summaries of dialogues, chats or correspondaces between two or more parties.

Recommendations

We suggest passing the chats to summarize in the following format to obtain best results:

[Person_A: Some text Person_B: Some other text Person_A: Response Person_A: Other text]

How to Get Started with the Model

from transformers import pipeline

summarizer = pipeline("summarization", model="Nicovis/ConvSum")


DIALOGUE = """
Linda: I'm going to have my room painted
Linda: Can't decide what colors would best express my personality
Linda: I'm considering advices, do you have one for me maybe? :)
Brian: Fitting your personality... hmm
Brian: I consider you an enthusiastic and lively person and the color for it is yellow
Brian: On the other hand you can be calm and steady and they say those qualities are expressed in greyness
Brian: So yellow & grey, how do you like the idea? :D
Linda: Wow, I'm impressed!
Linda: You've just paid me a compliment and come up with interesting colors!
Brian: Well, those are actually facts :)
Brian: Are you going to make use of the colors?
Linda: Actually, I was planning to ask a few friends and then sum all the ideas up
Linda: But now I think I don't need to do any further research
Linda: Asking you for help was a bull's-eye! :D
Brian: Perfection! :D
Brian: I'll come and check the results of your work soon, it'll be a nice chance for us to talk :)
Linda: Sure, feel invited!
"""

print(summarizer(DIALOGUE, max_length=64, min_length=30, do_sample=False))
>>> [{'summary_text': "Linda is going to have her room painted. Brian recommends yellow and grey colors for her personality. Linda is going to make use of the colors. Brian will come and check the results of Linda's work soon."}]

Training Details

The training on the SAMSum dataset was done using Seq2SeqTrainer with DataCollatorForSeq2Seq using PegasusFastTokenizer for both. The training arguments for the model were:

Seq2SeqTrainingArguments(
  num_train_epochs= 5, 
  warmup_steps= 500,
  per_device_train_batch_size= 16,
  per_device_eval_batch_size= 16,
  weight_decay= 0.1,
  logging_steps= 10,
  evaluation_strategy= steps,
  eval_steps= 300, 
  save_steps= 1e6,
  gradient_accumulation_steps= 16
)

Evaluation

Model was evaluated on cross entropy loss & ROUGE(1-2-L) scores during evaluation obtaining:

Metric	Average at the end of training
ROUGE-1	0.472
ROUGE-2	0.241
ROUGE-L	0.376
Cross entropy loss	1.4

Testing Metrics

The results were tested on the SAMSum dataset testing set on the following metrics: ROUGE(1-2-L) and BERTScore

Results

We obtained the following results

Metric	Average at the of testing
ROUGE-1	0.496
ROUGE-2	0.250
ROUGE-L	0.400
BERTScore	0.686