Update README.md

README.md

@@ -1,3 +1,204 @@
 ---
+language:
+- pt
+tags:
+- albertina-pt*
+- albertina-ptpt
+- albertina-ptbr
+- albertina-ptpt-base
+- albertina-ptbr-base
+- albertina-1b5-portuguese-ptpt
+- albertina-1b5-portuguese-ptbr
+- fill-mask
+- bert
+- deberta
+- portuguese
+- encoder
+- foundation model
 license: mit
+datasets:
+- PORTULAN/glue-ptpt
+widget:
+- text: >-
+    A culinária portuguesa é rica em sabores e [MASK], tornando-se um dos
+    maiores tesouros do país.
 ---
+---
+<img align="left" width="40" height="40" src="https://github.githubassets.com/images/icons/emoji/unicode/1f917.png">
+<p style="text-align: center;">&nbsp;&nbsp;&nbsp;&nbsp;This is the model card for Albertina 1.5 billion PT-PT 
+  You may be interested in some of the other models in the <a href="https://huggingface.co/PORTULAN">Albertina (encoders) and Gervásio (decoders) families</a>.
+</p>
+
+---
+
+# Albertina 1.5B PT-PT
+
+
+**Albertina 1.5B PT-PT** is a foundation, large language model for European **Portuguese** from **Portugal**.
+
+It is an **encoder** of the BERT family, based on the neural architecture Transformer and 
+developed over the DeBERTa model, with most competitive performance for this language. 
+It has different versions that were trained for different variants of Portuguese (PT), 
+namely the European variant from Portugal (**PT-PT**) and the American variant from Brazil (**PT-BR**), 
+and it is distributed free of charge and under a most permissible license.
+
+**Albertina 1.5B PT-PT** is the version for European **Portuguese** from **Portugal**, 
+and to the best of our knowledge, this is an encoder specifically for this language and variant 
+that,  at the time of its initial distribution, sets a new state of the art for it, and is made publicly available 
+and distributed for reuse.
+
+It is an **encoder** of the BERT family, based on the neural architecture Transformer and 
+developed over the DeBERTa model, with most competitive performance for this language. 
+It is distributed free of charge and under a most permissible license.
+
+
+**Albertina 1.5B PT-PT base** is developed by a joint team from the University of Lisbon and the University of Porto, Portugal. 
+For further details, check the respective [publication](https://arxiv.org/abs/?):
+
+``` latex
+@misc{albertina-pt,
+      title={Fostering the Ecosystem of Open Neural Encoders for Portuguese with Albertina PT-* family}, 
+      author={Rodrigo Santos and João Rodrigues and Luís Gomes and João Silva and António Branco and Henrique Lopes Cardoso
+              and Tomás Freitas Osório and Bernardo Leite},
+      year={2024},
+      eprint={?},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+Please use the above cannonical reference when using or citing this model.
+
+<br>
+
+
+# Model Description
+
+**This model card is for Albertina 1.5B PT-PT**, with 1.5 billion parameters, 48 layers and a hidden size of 1536.
+
+Albertina-PT-PT base is distributed under an [MIT license](https://huggingface.co/PORTULAN/albertina-ptpt/blob/main/LICENSE).
+
+DeBERTa is distributed under an [MIT license](https://github.com/microsoft/DeBERTa/blob/master/LICENSE).
+
+
+<br>
+
+# Training Data
+
+[**Albertina 1.5B PT-PT**](https://huggingface.co/PORTULAN/albertina-ptpt-base) was trained over a 4 billion token data set that resulted from gathering some openly available corpora of European Portuguese from the following sources:
+
+- [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX): the CulturaX is a multilingual corpus, freely available for research and AI development, created by combining and extensively cleaning two other large datasets, mC4 and OSCAR. It is the result of a selection performed over the [Common Crawl](https://commoncrawl.org/) data set, crawled from the Web, that retains only pages whose metadata indicates permission to be crawled, that performs deduplication, and that removes some boilerplate, among other filters. Given that it does not discriminate between the Portuguese variants, we performed extra filtering by retaining only documents whose meta-data indicate the Internet country code top-level domain of Portugal.
+- [DCEP](https://joint-research-centre.ec.europa.eu/language-technology-resources/dcep-digital-corpus-european-parliament_en): the Digital Corpus of the European Parliament is a multilingual corpus including documents in all official EU languages published on the European Parliament&#39;s official website. We retained its European Portuguese portion.
+- [Europarl](https://www.statmt.org/europarl/): the European Parliament Proceedings Parallel Corpus is extracted from the proceedings of the European Parliament from 1996 to 2011. We retained its European Portuguese portion.
+- [ParlamentoPT](https://huggingface.co/datasets/PORTULAN/parlamento-pt): the ParlamentoPT is a data set we obtained by gathering the publicly available documents with the transcription of the debates in the Portuguese Parliament.
+
+
+## Preprocessing
+
+We filtered the PT-PT corpora using the [BLOOM pre-processing](https://github.com/bigscience-workshop/data-preparation) pipeline.
+We skipped the default filtering of stopwords since it would disrupt the syntactic structure, and also the filtering for language identification given the corpus was pre-selected as Portuguese.
+
+
+## Training
+
+As codebase, we resorted to the [DeBERTa V2 xxlarge](https://huggingface.co/microsoft/deberta-v2-xxlarge), for English.
+
+To train **Albertina 1.5B PT-PT**, the data set was tokenized with the original DeBERTa tokenizer with a 128-token sequence truncation and dynamic padding for 250k steps, 
+a 256-token sequence-truncation for 80k steps and finally a 512-token sequence-truncation for 60k steps.
+These steps correspond to the equivalent setup of 48 hours on a2-megagpu-16gb Google Cloud A2 node for the 128-token input sequences, 24 hours of computation for the 256-token 
+input sequences and 24 hours of computation for the 512-token input sequences.
+We opted for a learning rate of 1e-5 with linear decay and 10k warm-up steps. 
+
+<br>
+
+# Evaluation
+
+The base model version was evaluated on downstream tasks, namely the translations into PT-PT of the English data sets used for a few of the tasks in the widely-used [GLUE benchmark](https://huggingface.co/datasets/glue).
+
+
+## GLUE tasks translated
+
+
+We resorted to [GLUE-PT](https://huggingface.co/datasets/PORTULAN/glue-ptpt), a **PT-PT version of the GLUE** benchmark. 
+We automatically translated the same four tasks from GLUE using [DeepL Translate](https://www.deepl.com/), which specifically provides translation from English to PT-PT as an option.
+
+| Model                     | RTE (Accuracy) | WNLI (Accuracy)| MRPC (F1) | STS-B (Pearson) |
+|---------------------------|----------------|----------------|-----------|-----------------|
+| **Albertina 1.5B PT-PT**  | **0.?**        | 0.             | **0.    **| **0.    **      | 
+| **Albertina PT-PT(900M)** |  0.8339        | 0.4225         | 0.9171    | 0.8801          |
+| **Albertina 100M PT-PT**  |  0.5848        | 0.5634         | 0.8793    | 0.8624          |
+
+<br>
+
+# How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='PORTULAN/albertina-1b5-portuguese-ptpt-encoder')
+>>> unmasker("A culinária portuguesa é rica em sabores e [MASK], tornando-se um dos maiores tesouros do país.")
+
+[{'score': 0.8332648277282715, 'token': 14690, 'token_str': ' costumes', 'sequence': 'A culinária portuguesa é rica em sabores e costumes, tornando-se um dos maiores tesouros do país.'},
+{'score': 0.07860890030860901, 'token': 29829, 'token_str': ' cores', 'sequence': 'A culinária portuguesa é rica em sabores e cores, tornando-se um dos maiores tesouros do país.'},
+{'score': 0.03278181701898575, 'token': 35277, 'token_str': ' arte', 'sequence': 'A culinária portuguesa é rica em sabores e arte, tornando-se um dos maiores tesouros do país.'},
+{'score': 0.009515956044197083, 'token': 9240, 'token_str': ' cor', 'sequence': 'A culinária portuguesa é rica em sabores e cor, tornando-se um dos maiores tesouros do país.'},
+{'score': 0.009381960146129131, 'token': 33455, 'token_str': ' nuances', 'sequence': 'A culinária portuguesa é rica em sabores e nuances, tornando-se um dos maiores tesouros do país.'}]
+
+
+
+```
+
+The model can be used by fine-tuning it for a specific task:
+
+```python
+>>> from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
+>>> from datasets import load_dataset
+
+>>> model = AutoModelForSequenceClassification.from_pretrained("PORTULAN/albertina-ptpt-base", num_labels=2)
+>>> tokenizer = AutoTokenizer.from_pretrained("PORTULAN/albertina-ptpt-base")
+>>> dataset = load_dataset("PORTULAN/glue-ptpt", "rte")
+
+>>> def tokenize_function(examples):
+...     return tokenizer(examples["sentence1"], examples["sentence2"], padding="max_length", truncation=True)
+
+>>> tokenized_datasets = dataset.map(tokenize_function, batched=True)
+
+>>> training_args = TrainingArguments(output_dir="albertina-ptpt-rte", evaluation_strategy="epoch")
+>>> trainer = Trainer(
+...     model=model,
+...     args=training_args,
+...     train_dataset=tokenized_datasets["train"],
+...     eval_dataset=tokenized_datasets["validation"],
+... )
+
+>>> trainer.train()
+
+```
+
+<br>
+
+# Citation
+
+When using or citing this model, kindly cite the following [publication](https://arxiv.org/abs/?):
+
+``` latex
+@misc{albertina-pt,
+      title={Fostering the Ecosystem of Open Neural Encoders for Portuguese with Albertina PT-* family}, 
+      author={Rodrigo Santos and João Rodrigues and Luís Gomes and João Silva and António Branco and Henrique Lopes Cardoso
+              and Tomás Freitas Osório and Bernardo Leite},
+      year={2024},
+      eprint={?},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+<br>
+
+# Acknowledgments
+
+The research reported here was partially supported by: PORTULAN CLARIN—Research Infrastructure for the Science and Technology of Language,
+funded by Lisboa 2020, Alentejo 2020 and FCT—Fundação para a Ciência e Tecnologia under the
+grant PINFRA/22117/2016; research project ALBERTINA - Foundation Encoder Model for Portuguese and AI, funded by FCT—Fundação para a Ciência e Tecnologia under the
+grant CPCA-IAC/AV/478394/2022; innovation project ACCELERAT.AI - Multilingual Intelligent Contact Centers, funded by IAPMEI, I.P. - Agência para a Competitividade e Inovação under the grant C625734525-00462629, of Plano de Recuperação e Resiliência, call RE-C05-i01.01 – Agendas/Alianças Mobilizadoras para a Reindustrialização; and LIACC - Laboratory for AI and Computer Science, funded by FCT—Fundação para a Ciência e Tecnologia under the grant FCT/UID/CEC/0027/2020.