Report for mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 2 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_allagree, split train).
👉Robustness issues (2)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.301 | Transform to uppercase | 301/1000 tested samples (30.1%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 30.1% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to uppercase(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 118 | Finnish lifting equipment maker Konecranes Oyj said on July 30 , 2008 that its net profit rose to 71.2 mln euro ( $ 111.1 mln ) for the first half of 2008 from 57.1 mln euro ( $ 89.1 mln ) for the same period of 2007 . | FINNISH LIFTING EQUIPMENT MAKER KONECRANES OYJ SAID ON JULY 30 , 2008 THAT ITS NET PROFIT ROSE TO 71.2 MLN EURO ( $ 111.1 MLN ) FOR THE FIRST HALF OF 2008 FROM 57.1 MLN EURO ( $ 89.1 MLN ) FOR THE SAME PERIOD OF 2007 . | positive (p = 1.00) | neutral (p = 1.00) |
| 875 | Net sales will , however , increase from 2005 . | NET SALES WILL , HOWEVER , INCREASE FROM 2005 . | positive (p = 1.00) | neutral (p = 1.00) |
| 823 | In the end of 2006 , the number of outlets will rise to 60-70 . | IN THE END OF 2006 , THE NUMBER OF OUTLETS WILL RISE TO 60-70 . | positive (p = 1.00) | neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.051 | Add typos | 51/1000 tested samples (5.1%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 5.1% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 178 | Passenger volumes rose by 8.4 % in the accounting period . | assengdr volumss rse by 8.4 %% in the accounting period . | positive (p = 1.00) | neutral (p = 1.00) |
| 2129 | Consolidated pretax profit decreased by 69.2 % to EUR 41.0 mn from EUR 133.1 mn in 2007 . | Consolidated pretax profit decreaswed by U69.2 % to EUR 41.0 mn from EUR 133.1 mn in 2007 . | negative (p = 1.00) | positive (p = 1.00) |
| 216 | Elisa said mobile subscriptions grew 7 percent during 2007 , mainly because of customers adopting so-called third generation mobile technology . | Elisa said mobile subscriptins greq 7 percnt during 2007 , mainly vbecause of customers adopting so-called thjrd generation mohbile technology . | positive (p = 1.00) | neutral (p = 1.00) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!