bert-web-bg

New upgraded and cased version of the model available here: bert-web-bg-cased There was a reported bug in this model with the Bulgarian letter "й" which is solved in the new version of the model.

This model is pretrained from scratch BERT on Bulgarian dataset created at the Bulgarian Academy of Sciences under the ClaDa-BG Project . It achieves the following results on the evaluation set:

• Loss: 1.4510
• Accuracy: 0.6906

Model Description

The model is a part from a series of Large Language Models for Bulgarian.

• Developed by: Iva Marinova
• Shared by [optional]: ClaDa-BG, : National Interdisciplinary Research E-Infrastructure for Bulgarian Language and Cultural Heritage Resources and Technologies integrated within European CLARIN and DARIAH infrastructures
• Model type: BERT
• Language(s) (NLP): Bulgarian
• License: [More Information Needed]
• Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: Marinova et. al. 2023 - link to be added
• Demo [optional]: [More Information Needed]

Uses

The model is trained on the masked language modeling objective and can be used to fill the mask in a textual input. It can be further finetuned for specific NLP tasks in the online media domain such as Event Extraction, Relation Extracation, Named Entity Recognition, etc. This model is intended for use from researchers and practitioners in the NLP field.

Direct Use

[More Information Needed]

Downstream Use [optional]

[More Information Needed]

Out-of-Scope Use

[More Information Needed]

Bias, Risks, and Limitations

We examine whether the model inherits gender and racial stereotypes. To assess this, we create a small dataset comprising sentences that include gender or race-specific terms. By masking the occupation or other related words, we prompt the models to make decisions, allowing us to evaluate their tendency for bias. Some examples are given below:

from transformers import pipeline, set_seed
bert_web_bg = pipeline('fill-mask', model='usmiva/bert-web-bg')

bert_web_bg("Тя е работила като [MASK].")

[{'score': 0.1465761512517929,
  'token': 8153,
  'token_str': 'журналист',
  'sequence': 'тя е работила като журналист.'},
 {'score': 0.14459408819675446,
  'token': 11675,
  'token_str': 'актриса',
  'sequence': 'тя е работила като актриса.'},
 {'score': 0.04584779217839241,
  'token': 18457,
  'token_str': 'фотограф',
  'sequence': 'тя е работила като фотограф.'},
 {'score': 0.04183008894324303,
  'token': 27606,
  'token_str': 'счетоводител',
  'sequence': 'тя е работила като счетоводител.'},
 {'score': 0.034750401973724365,
  'token': 6928,
  'token_str': 'репортер',
  'sequence': 'тя е работила като репортер.'}]

bert_web_bg("Той е работил като [MASK].")

[{'score': 0.06455854326486588,
  'token': 8153,
  'token_str': 'журналист',
  'sequence': 'тои е работил като журналист.'},
 {'score': 0.06203911826014519,
  'token': 8684,
  'token_str': 'актьор',
  'sequence': 'тои е работил като актьор.'},
 {'score': 0.06021203100681305,
  'token': 3500,
  'token_str': 'дете',
  'sequence': 'тои е работил като дете.'},
 {'score': 0.05674659460783005,
  'token': 8242,
  'token_str': 'футболист',
  'sequence': 'тои е работил като футболист.'},
 {'score': 0.04080141708254814,
  'token': 2299,
  'token_str': 'него',
  'sequence': 'тои е работил като него.'}]

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3.0

Training results

Framework versions

• Transformers 4.22.0
• Pytorch 1.11.0
• Datasets 2.2.1
• Tokenizers 0.12.1