Report for soleimanian/financial-roberta-large-sentiment
#41
by
inoki-giskard
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 4 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_75agree, split train).
👉Robustness issues (3)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.098 | Transform to uppercase | 98/1000 tested samples (9.8%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 9.8% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to uppercase(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 2605 | One of the largest projects of the magazine division of SanomaWSOY - Sanoma Magazines International in 2006 became launch of the Russian magazine Gloriya . | ONE OF THE LARGEST PROJECTS OF THE MAGAZINE DIVISION OF SANOMAWSOY - SANOMA MAGAZINES INTERNATIONAL IN 2006 BECAME LAUNCH OF THE RUSSIAN MAGAZINE GLORIYA . | positive (p = 1.00) | neutral (p = 1.00) |
| 332 | Finnish Talvivaara Mining Co HEL : TLV1V said Thursday it had picked BofA Merrill Lynch and JPMorgan NYSE : JPM as joint bookrunners of its planned issue of convertible notes worth up to EUR250m USD332m . | FINNISH TALVIVAARA MINING CO HEL : TLV1V SAID THURSDAY IT HAD PICKED BOFA MERRILL LYNCH AND JPMORGAN NYSE : JPM AS JOINT BOOKRUNNERS OF ITS PLANNED ISSUE OF CONVERTIBLE NOTES WORTH UP TO EUR250M USD332M . | positive (p = 1.00) | neutral (p = 1.00) |
| 643 | The company said it observed a current stabilisation in prices and there is potential for higher prices for deliveries in the first quarter of 2011 . | THE COMPANY SAID IT OBSERVED A CURRENT STABILISATION IN PRICES AND THERE IS POTENTIAL FOR HIGHER PRICES FOR DELIVERIES IN THE FIRST QUARTER OF 2011 . | positive (p = 1.00) | negative (p = 0.99) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.083 | Add typos | 83/1000 tested samples (8.3%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 8.3% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 900 | Stockholm-based Nordea Bank ( STO : NDA ) said yesterday it had hired Casper von Koskull to lead its corporate merchant banking and capital markets unit , effective 1 September . | Atockholm-bassd Jordea Bank ( STO : NDA ) said yesteeday it had hiyted Casper von Koskull to lead its corporate merchant bankinfg and capital markets unit , effective 1 Septwmber . | positive (p = 0.53) | neutral (p = 1.00) |
| 973 | The trucks feature an Eco Drive system - a fuel measuring tool which stores data particular to individual drivers . | The trucks feature xa Eco Drive sydtem - a fuel measuring tool which stkres dta particupar to individual drivers . | neutral (p = 1.00) | positive (p = 0.82) |
| 474 | In the second quarter of 2010 , Raute 's net loss narrowed to EUR 123,000 from EUR 1.5 million in the same period of 2009 . | In the second qyarter of 2010 , Raute 's net loss narfowed to EUR 23,000 from EUR 1.5 million in the same period of 2009 . | positive (p = 1.00) | negative (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.054 | Transform to title case | 54/1000 tested samples (5.4%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 5.4% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to title case(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1413 | Scanfil expects net sales in 2008 to remain at the 2007 level . | Scanfil Expects Net Sales In 2008 To Remain At The 2007 Level . | neutral (p = 0.84) | negative (p = 0.97) |
| 2046 | Marathon now has a 4.6 percent stake in PLX , it said , according to Bloomberg . | Marathon Now Has A 4.6 Percent Stake In Plx , It Said , According To Bloomberg . | positive (p = 0.89) | neutral (p = 0.95) |
| 52 | STOCK EXCHANGE ANNOUNCEMENT 20 July 2006 1 ( 1 ) BASWARE SHARE SUBSCRIPTIONS WITH WARRANTS AND INCREASE IN SHARE CAPITAL A total of 119 850 shares have been subscribed with BasWare Warrant Program . | Stock Exchange Announcement 20 July 2006 1 ( 1 ) Basware Share Subscriptions With Warrants And Increase In Share Capital A Total Of 119 850 Shares Have Been Subscribed With Basware Warrant Program . | neutral (p = 1.00) | positive (p = 0.89) |
👉Performance issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "said" |
Balanced Accuracy = 0.856 | — | -5.20% than global |
🔍✨Examples
For records in the dataset where `text` contains "said", the Balanced Accuracy is 5.2% lower than the global Balanced Accuracy.| text | label | Predicted label |
|
|---|---|---|---|
| 332 | Finnish Talvivaara Mining Co HEL : TLV1V said Thursday it had picked BofA Merrill Lynch and JPMorgan NYSE : JPM as joint bookrunners of its planned issue of convertible notes worth up to EUR250m USD332m . | neutral | positive (p = 1.00) |
| 359 | `` Of course , we are in talks with all those coming up with new projects that involve material handling , '' Konecranes President and CEO Pekka Lundmark said , when asked whether it was talking to Reliance Industries for supplying equipment to its upcoming refinery in Jamnagar . | neutral | positive (p = 0.86) |
| 368 | Finnish food company Raisio Oyj HEL : RAIVV said on Friday it has wrapped up the divestment of its margarine operations to US sector player Bunge Ltd NYSE : BG for EUR80m USD119 .2 m . | neutral | positive (p = 0.69) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!