| language: | |
| - sl | |
| - en | |
| - multilingual | |
| tags: | |
| - generated_from_trainer | |
| licence: cc-by-sa-4.0 | |
| # SloBERTa-SlEng | |
| SloBERTa-SlEng is a masked language model, based on the [SloBERTa](https://huggingface.co./EMBEDDIA/sloberta) Slovene model. | |
| SloBERTa-SlEng replaces the tokenizer, vocabulary and the embeddings layer of the SloBERTa model. | |
| The tokenizer and vocabulary used are bilingual, Slovene-English, based on conversational, non-standard, and slang language the model was trained on. | |
| They are the same as in the [SlEng-bert](https://huggingface.co./cjvt/sleng-bert) model. | |
| The new embedding weights were initialized from the SloBERTa embeddings. | |
| The new SloBERTa-SlEng model is SloBERTa model, which was further pre-trained for two epochs on the conversational English and Slovene corpora, | |
| the same as the [SlEng-bert](https://huggingface.co./cjvt/sleng-bert) model. | |
| ## Training corpora | |
| The model was trained on English and Slovene tweets, Slovene corpora [MaCoCu](http://hdl.handle.net/11356/1517) and [Frenk](http://hdl.handle.net/11356/1201), | |
| and a small subset of English [Oscar](https://huggingface.co./datasets/oscar) corpus. We tried to keep the sizes of English and Slovene corpora as equal as possible. | |
| Training corpora had in total about 2.7 billion words. | |
| ### Framework versions | |
| - Transformers 4.22.0.dev0 | |
| - Pytorch 1.13.0a0+d321be6 | |
| - Datasets 2.4.0 | |
| - Tokenizers 0.12.1 | |