license: apache-2.0
datasets:
- Babelscape/multinerd
language:
- en
metrics:
- f1
- precision
- recall
pipeline_tag: token-classification
tags:
- ner
- named-entity-recognition
- token-classification
model-index:
- name: robert-base on MultiNERD by Jayant Yadav
results:
- task:
type: named-entity-recognition-ner
name: Named Entity Recognition
dataset:
type: Babelscape/multinerd
name: MultiNERD (English)
split: test
revision: 2814b78e7af4b5a1f1886fe7ad49632de4d9dd25
config: Babelscape/multinerd
args:
split: train[:50%]
metrics:
- type: f1
value: 0.943
name: F1
- type: precision
value: 0.939
name: Precision
- type: recall
value: 0.947
name: Recall
config: seqeval
paper: https://aclanthology.org/2022.findings-naacl.60.pdf
base_model: roberta-base
library_name: transformers
Model Card for Model ID
roBERTa-base model was fine-tuned on 50% training English only split of MultiNERD dataset and later evaluated on full test split of the same.
The finetuning script can be fetched from fintuning.py.
Various other model were tested on the same selection of dataset and the best checkpoint was uploaded. The detailed configuration summary can be found in Appendix section of report.
Model Details
Model Description
Head over to github repo for all the scripts used to finetune and evalute token-classification model. The model is ready to use on Kaggle too!
- Developed by: Jayant Yadav
Uses
Token-classification of the following entities are possible:
| Class | Description | Examples |
|---|---|---|
| PER (person) | People | Ray Charles, Jessica Alba, Leonardo DiCaprio, Roger Federer, Anna Massey. |
| ORG (organization) | Associations, companies, agencies, institutions, nationalities and religious or political groups | University of Edinburgh, San Francisco Giants, Google, Democratic Party. |
| LOC (location) | Physical locations (e.g. mountains, bodies of water), geopolitical entities (e.g. cities, states), and facilities (e.g. bridges, buildings, airports). | Rome, Lake Paiku, Chrysler Building, Mount Rushmore, Mississippi River. |
| ANIM (animal) | Breeds of dogs, cats and other animals, including their scientific names. | Maine Coon, African Wild Dog, Great White Shark, New Zealand Bellbird. |
| BIO (biological) | Genus of fungus, bacteria and protoctists, families of viruses, and other biological entities. | Herpes Simplex Virus, Escherichia Coli, Salmonella, Bacillus Anthracis. |
| CEL (celestial) | Planets, stars, asteroids, comets, nebulae, galaxies and other astronomical objects. | Sun, Neptune, Asteroid 187 Lamberta, Proxima Centauri, V838 Monocerotis. |
| DIS (disease) | Physical, mental, infectious, non-infectious, deficiency, inherited, degenerative, social and self-inflicted diseases. | Alzheimer’s Disease, Cystic Fibrosis, Dilated Cardiomyopathy, Arthritis. |
| EVE (event) | Sport events, battles, wars and other events. | American Civil War, 2003 Wimbledon Championships, Cannes Film Festival. |
| FOOD (food) | Foods and drinks. | Carbonara, Sangiovese, Cheddar Beer Fondue, Pizza Margherita. |
| INST (instrument) | Technological instruments, mechanical instruments, musical instruments, and other tools. | Spitzer Space Telescope, Commodore 64, Skype, Apple Watch, Fender Stratocaster. |
| MEDIA (media) | Titles of films, books, magazines, songs and albums, fictional characters and languages. | Forbes, American Psycho, Kiss Me Once, Twin Peaks, Disney Adventures. |
| PLANT (plant) | Types of trees, flowers, and other plants, including their scientific names. | Salix, Quercus Petraea, Douglas Fir, Forsythia, Artemisia Maritima. |
| MYTH (mythological) | Mythological and religious entities. | Apollo, Persephone, Aphrodite, Saint Peter, Pope Gregory I, Hercules. |
| TIME (time) | Specific and well-defined time intervals, such as eras, historical periods, centuries, years and important days. No months and days of the week. | Renaissance, Middle Ages, Christmas, Great Depression, 17th Century, 2012. |
| VEHI (vehicle) | Cars, motorcycles and other vehicles. | Ferrari Testarossa, Suzuki Jimny, Honda CR-X, Boeing 747, Fairey Fulmar. |
Bias, Risks, and Limitations
Only trained on English split of MultiNERD dataset. Therefore will not perform well on other languages.
How to Get Started with the Model
Use the code below to get started with the model:
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline
tokenizer = AutoTokenizer.from_pretrained("jayant-yadav/roberta-base-multinerd")
model = AutoModelForTokenClassification.from_pretrained("jayant-yadav/roberta-base-multinerd")
nlp = pipeline("ner", model=model, tokenizer=tokenizer)
example = "My name is Wolfgang and I live in Berlin"
ner_results = nlp(example)
print(ner_results)
Training Details
Training Data
50% of train split of MultiNERD dataset was used to finetune the model.
Training Procedure
Preprocessing
English dataset was filterd out : train_dataset = train_dataset.filter(lambda x: x['lang'] == 'en')
Training Hyperparameters
The following hyperparameters were used during training:
learning_rate: 5e-05
train_batch_size: 32
eval_batch_size: 32
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1
Evaluation
Evaluation was perfored on 50% of evaluation split of MultiNERD dataset.
Testing Data & Metrics
Testing Data
Tested on Full test split of MultiNERD dataset.
Metrics
Model versions and checkpoint were evaluated using F1, Precision and Recall.
For this seqeval metric was used: metric = load_metric("seqeval").
Results
| Entity | Precision | Recall | F1 score | Support |
|---|---|---|---|---|
| ANIM | 0.71 | 0.77 | 0.739 | 1604 |
| BIO | 0.5 | 0.125 | 0.2 | 8 |
| CEL | 0.738 | 0.756 | 0.746 | 41 |
| DIS | 0.737 | 0.772 | 0.754 | 759 |
| EVE | 0.952 | 0.968 | 0.960 | 352 |
| FOOD | 0.679 | 0.545 | 0.605 | 566 |
| INST | 0.75 | 0.75 | 0.75 | 12 |
| LOC | 0.994 | 0.991 | 0.993 | 12024 |
| MEDIA | 0.940 | 0.969 | 0.954 | 458 |
| ORG | 0.977 | 0.981 | 0.979 | 3309 |
| PER | 0.992 | 0.995 | 0.993 | 5265 |
| PLANT | 0.617 | 0.730 | 0.669 | 894 |
| MYTH | 0.647 | 0.687 | 0.666 | 32 |
| TIME | 0.825 | 0.820 | 0.822 | 289 |
| VEHI | 0.812 | 0.812 | 0.812 | 32 |
| Overall | 0.939 | 0.947 | 0.943 |
Technical Specifications
Model Architecture and Objective
Follows the same as RoBERTa-BASE
Compute Infrastructure
Hardware
Kaggle - GPU T4x2
Google Colab - GPU T4x1
Software
pandas==1.5.3
numpy==1.23.5
seqeval==1.2.2
datasets==2.15.0
huggingface_hub==0.19.4
transformers[torch]==4.35.2
evaluate==0.4.1
matplotlib==3.7.1
collections
torch==2.0.0