Hugging Face's logo

PORTULAN
/
albertina-1b5-portuguese-ptpt-encoder

Fill-Mask
Transformers
PyTorch
Portuguese
deberta-v2
albertina-pt*
albertina-100m-portuguese-ptpt
albertina-100m-portuguese-ptbr
albertina-900m-portuguese-ptpt
albertina-900m-portuguese-ptbr
albertina-1b5-portuguese-ptpt
albertina-1b5-portuguese-ptbr
bert
deberta
portuguese
encoder
foundation model
Inference Endpoints
Model card Files Community
jarodrigues commited on
Commit
c8274c0
•
1 Parent(s): 60ee937

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +30 -30
README.md CHANGED
@@ -25,16 +25,16 @@ widget:
25
  ---
26
  ---
27
  <img align="left" width="40" height="40" src="https://github.githubassets.com/images/icons/emoji/unicode/1f917.png">
28
- <p style="text-align: center;">&nbsp;&nbsp;&nbsp;&nbsp;This is the model card for Albertina 1.5B PT-PT
29
  You may be interested in some of the other models in the <a href="https://huggingface.co/PORTULAN">Albertina (encoders) and Gervásio (decoders) families</a>.
30
  </p>
31
 
32
  ---
33
 
34
- # Albertina 1.5B PT-PT
35
 
36
 
37
- **Albertina 1.5B PT-PT** is a foundation, large language model for European **Portuguese** from **Portugal**.
38
 
39
  It is an **encoder** of the BERT family, based on the neural architecture Transformer and
40
  developed over the DeBERTa model, with most competitive performance for this language.
@@ -44,16 +44,16 @@ and it is distributed free of charge and under a most permissible license.
44
 
45
  | Albertina's Family of Models |
46
  |----------------------------------------------------------------------------------------------------------|
47
- | [**Albertina 1.5B PT-PT**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptpt-encoder) |
48
- | [**Albertina 1.5B PT-BR**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptbr-encoder) |
49
- | [**Albertina 1.5B PT-PT 256**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptpt-encoder-256)|
50
- | [**Albertina 1.5B PT-BR 256**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptbr-encoder-256)|
51
- | [**Albertina 900M PT-PT**](https://huggingface.co/PORTULAN/albertina-900m-portuguese-ptpt-encoder) |
52
- | [**Albertina 900M PT-BR**](https://huggingface.co/PORTULAN/albertina-900m-portuguese-ptbr-encoder) |
53
- | [**Albertina 100M PT-PT**](https://huggingface.co/PORTULAN/albertina-100m-portuguese-ptpt-encoder) |
54
- | [**Albertina 100M PT-BR**](https://huggingface.co/PORTULAN/albertina-100m-portuguese-ptbr-encoder) |
55
-
56
- **Albertina 1.5B PT-PT** is the version for European **Portuguese** from **Portugal**,
57
  and to the best of our knowledge, this is an encoder specifically for this language and variant
58
  that, at the time of its initial distribution, sets a new state of the art for it, and is made publicly available
59
  and distributed for reuse.
@@ -63,7 +63,7 @@ developed over the DeBERTa model, with most competitive performance for this lan
63
  It is distributed free of charge and under a most permissible license.
64
 
65
 
66
- **Albertina 1.5B PT-PT** is developed by a joint team from the University of Lisbon and the University of Porto, Portugal.
67
  For further details, check the respective [publication](https://arxiv.org/abs/?):
68
 
69
  ``` latex
@@ -85,9 +85,9 @@ Please use the above cannonical reference when using or citing this model.
85
 
86
  # Model Description
87
 
88
- **This model card is for Albertina 1.5B PT-PT**, with 1.5 billion parameters, 48 layers and a hidden size of 1536.
89
 
90
- Albertina 1.5B PT-PT is distributed under an [MIT license](https://huggingface.co/PORTULAN/albertina-ptpt/blob/main/LICENSE).
91
 
92
  DeBERTa is distributed under an [MIT license](https://github.com/microsoft/DeBERTa/blob/master/LICENSE).
93
 
@@ -96,7 +96,7 @@ DeBERTa is distributed under an [MIT license](https://github.com/microsoft/DeBER
96
 
97
  # Training Data
98
 
99
- [**Albertina 1.5B PT-PT**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptpt-encoder) was trained over a 4 billion token data set that resulted from gathering some openly available corpora of European Portuguese from the following sources:
100
 
101
  - [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX): the CulturaX is a multilingual corpus, freely available for research and AI development, created by combining and extensively cleaning two other large datasets, mC4 and OSCAR. It is the result of a selection performed over the [Common Crawl](https://commoncrawl.org/) data set, crawled from the Web, that retains only pages whose metadata indicates permission to be crawled, that performs deduplication, and that removes some boilerplate, among other filters. Given that it does not discriminate between the Portuguese variants, we performed extra filtering by retaining only documents whose meta-data indicate the Internet country code top-level domain of Portugal.
102
  - [DCEP](https://joint-research-centre.ec.europa.eu/language-technology-resources/dcep-digital-corpus-european-parliament_en): the Digital Corpus of the European Parliament is a multilingual corpus including documents in all official EU languages published on the European Parliament&#39;s official website. We retained its European Portuguese portion.
@@ -106,7 +106,7 @@ DeBERTa is distributed under an [MIT license](https://github.com/microsoft/DeBER
106
 
107
  ## Preprocessing
108
 
109
- We filtered the PT-PT corpora using the [BLOOM pre-processing](https://github.com/bigscience-workshop/data-preparation) pipeline.
110
  We skipped the default filtering of stopwords since it would disrupt the syntactic structure, and also the filtering for language identification given the corpus was pre-selected as Portuguese.
111
 
112
 
@@ -114,8 +114,8 @@ We skipped the default filtering of stopwords since it would disrupt the syntact
114
 
115
  As codebase, we resorted to the [DeBERTa V2 xxlarge](https://huggingface.co/microsoft/deberta-v2-xxlarge), for English.
116
 
117
- To train **Albertina 1.5B PT-PT**, the data set was tokenized with the original DeBERTa tokenizer with a 128-token sequence truncation and dynamic padding for 250k steps,
118
- a 256-token sequence-truncation for 80k steps ([**Albertina 1.5B PT-PT 256**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptpt-encoder-256)) and finally a 512-token sequence-truncation for 60k steps.
119
  These steps correspond to the equivalent setup of 48 hours on a2-megagpu-16gb Google Cloud A2 node for the 128-token input sequences, 24 hours of computation for the 256-token
120
  input sequences and 24 hours of computation for the 512-token input sequences.
121
  We opted for a learning rate of 1e-5 with linear decay and 10k warm-up steps.
@@ -125,15 +125,15 @@ We opted for a learning rate of 1e-5 with linear decay and 10k warm-up steps.
125
  # Evaluation
126
 
127
 
128
- We resorted to [HyperGlue-PT](?), a **PT-PT version of the GLUE and SUPERGLUE** benchmark.
129
  We automatically translated the tasks from GLUE and SUPERGLUE using [DeepL Translate](https://www.deepl.com/), which specifically provides translation from English to PT-PT as an option.
130
 
131
  | Model | RTE (Accuracy) | WNLI (Accuracy)| MRPC (F1) | STS-B (Pearson) | COPA (Accuracy) | CB (F1) | MultiRC (F1) | BoolQ (Accuracy) |
132
  |-------------------------------|----------------|----------------|-----------|-----------------|-----------------|------------|--------------|------------------|
133
- | **Albertina 1.5B PT-PT** | **0.8809** | 0.4742 | 0.8457 | **0.9034** | **0.8433** | **0.7840** | **0.7688** | **0.8602** |
134
- | **Albertina 1.5B PT-PT 256** | 0.8809 | 0.5493 | 0.8752 | 0.8795 | 0.8400 | 0.5832 | 0.6791 | 0.8496 |
135
- | **Albertina 900M PT-PT** | 0.8339 | 0.4225 | **0.9171**| 0.8801 | 0.7033 | 0.6018 | 0.6728 | 0.8224 |
136
- | **Albertina 100M PT-PT** | 0.6919 | 0.4742 | 0.8047 | 0.8590 | n.a. | 0.4529 | 0.6481 | 0.7578 |
137
  ||||||||||
138
  | **DeBERTa 1.5B EN** | 0.8147 | 0.4554 | 0.8696 | 0.8557 | 0.5167 | 0.4901 | 0.6687 | 0.8347 |
139
  | **DeBERTa 100M EN** | 0.6029 | **0.5634** | 0.7802 | 0.8320 | n.a. | 0.4698 | 0.6368 | 0.6829 |
@@ -141,15 +141,15 @@ We automatically translated the tasks from GLUE and SUPERGLUE using [DeepL Trans
141
 
142
 
143
 
144
- **para modelo PT-BR**
145
 
146
  | Model | RTE (Accuracy) | WNLI (Accuracy)| MRPC (F1) | STS-B (Pearson) | COPA (Accuracy) | CB (F1) | MultiRC (F1) | BoolQ (Accuracy) |
147
  |-------------------------------|----------------|----------------|-----------|-----------------|-----------------|------------|--------------|------------------|
148
- | **Albertina 1.5B PT-BR** | **0.8676** | 0.4742 | 0.8622 | **0.9007** | 0.7767 | 0.6372 | **0.7667** | **0.8654** |
149
- | **Albertina 1.5B PT-BR 256** | 0.8123 | 0.4225 | 0.8638 | 0.8968 | **0.8533** | **0.6884** | 0.6799 | 0.8509 |
150
- | **Albertina 900M PT-BR** | 0.7545 | 0.4601 | **0.9071**| 0.8910 | 0.7767 | 0.5799 | 0.6731 | 0.8385 |
151
  | **BERTimbau (335M)** | 0.6446 | **0.5634** | 0.8873 | 0.8842 | 0.6933 | 0.5438 | 0.6787 | 0.7783 |
152
- | **Albertina 100M PT-BR** | 0.6582 | **0.5634** | 0.8149 | 0.8489 | n.a. | 0.4771 | 0.6469 | 0.7537 |
153
  ||||||||||
154
  | **DeBERTa 1.5B EN** | 0.7112 | **0.5634** | 0.8545 | 0.0123 | 0.5700 | 0.4307 | 0.3639 | 0.6217 |
155
  | **DeBERTa 100M EN** | 0.5716 | 0.5587 | 0.8060 | 0.8266 | n.a. | 0.4739 | 0.6391 | 0.6838 |
 
25
  ---
26
  ---
27
  <img align="left" width="40" height="40" src="https://github.githubassets.com/images/icons/emoji/unicode/1f917.png">
28
+ <p style="text-align: center;">&nbsp;&nbsp;&nbsp;&nbsp;This is the model card for Albertina 1.5B PTPT
29
  You may be interested in some of the other models in the <a href="https://huggingface.co/PORTULAN">Albertina (encoders) and Gervásio (decoders) families</a>.
30
  </p>
31
 
32
  ---
33
 
34
+ # Albertina 1.5B PTPT
35
 
36
 
37
+ **Albertina 1.5B PTPT** is a foundation, large language model for European **Portuguese** from **Portugal**.
38
 
39
  It is an **encoder** of the BERT family, based on the neural architecture Transformer and
40
  developed over the DeBERTa model, with most competitive performance for this language.
 
44
 
45
  | Albertina's Family of Models |
46
  |----------------------------------------------------------------------------------------------------------|
47
+ | [**Albertina 1.5B PTPT**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptpt-encoder) |
48
+ | [**Albertina 1.5B PTBR**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptbr-encoder) |
49
+ | [**Albertina 1.5B PTPT 256**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptpt-encoder-256)|
50
+ | [**Albertina 1.5B PTBR 256**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptbr-encoder-256)|
51
+ | [**Albertina 900M PTPT**](https://huggingface.co/PORTULAN/albertina-900m-portuguese-ptpt-encoder) |
52
+ | [**Albertina 900M PTBR**](https://huggingface.co/PORTULAN/albertina-900m-portuguese-ptbr-encoder) |
53
+ | [**Albertina 100M PTPT**](https://huggingface.co/PORTULAN/albertina-100m-portuguese-ptpt-encoder) |
54
+ | [**Albertina 100M PTBR**](https://huggingface.co/PORTULAN/albertina-100m-portuguese-ptbr-encoder) |
55
+
56
+ **Albertina 1.5B PTPT** is the version for European **Portuguese** from **Portugal**,
57
  and to the best of our knowledge, this is an encoder specifically for this language and variant
58
  that, at the time of its initial distribution, sets a new state of the art for it, and is made publicly available
59
  and distributed for reuse.
 
63
  It is distributed free of charge and under a most permissible license.
64
 
65
 
66
+ **Albertina 1.5B PTPT** is developed by a joint team from the University of Lisbon and the University of Porto, Portugal.
67
  For further details, check the respective [publication](https://arxiv.org/abs/?):
68
 
69
  ``` latex
 
85
 
86
  # Model Description
87
 
88
+ **This model card is for Albertina 1.5B PTPT**, with 1.5 billion parameters, 48 layers and a hidden size of 1536.
89
 
90
+ Albertina 1.5B PTPT is distributed under an [MIT license](https://huggingface.co/PORTULAN/albertina-ptpt/blob/main/LICENSE).
91
 
92
  DeBERTa is distributed under an [MIT license](https://github.com/microsoft/DeBERTa/blob/master/LICENSE).
93
 
 
96
 
97
  # Training Data
98
 
99
+ [**Albertina 1.5B PTPT**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptpt-encoder) was trained over a 4 billion token data set that resulted from gathering some openly available corpora of European Portuguese from the following sources:
100
 
101
  - [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX): the CulturaX is a multilingual corpus, freely available for research and AI development, created by combining and extensively cleaning two other large datasets, mC4 and OSCAR. It is the result of a selection performed over the [Common Crawl](https://commoncrawl.org/) data set, crawled from the Web, that retains only pages whose metadata indicates permission to be crawled, that performs deduplication, and that removes some boilerplate, among other filters. Given that it does not discriminate between the Portuguese variants, we performed extra filtering by retaining only documents whose meta-data indicate the Internet country code top-level domain of Portugal.
102
  - [DCEP](https://joint-research-centre.ec.europa.eu/language-technology-resources/dcep-digital-corpus-european-parliament_en): the Digital Corpus of the European Parliament is a multilingual corpus including documents in all official EU languages published on the European Parliament&#39;s official website. We retained its European Portuguese portion.
 
106
 
107
  ## Preprocessing
108
 
109
+ We filtered the PTPT corpora using the [BLOOM pre-processing](https://github.com/bigscience-workshop/data-preparation) pipeline.
110
  We skipped the default filtering of stopwords since it would disrupt the syntactic structure, and also the filtering for language identification given the corpus was pre-selected as Portuguese.
111
 
112
 
 
114
 
115
  As codebase, we resorted to the [DeBERTa V2 xxlarge](https://huggingface.co/microsoft/deberta-v2-xxlarge), for English.
116
 
117
+ To train **Albertina 1.5B PTPT**, the data set was tokenized with the original DeBERTa tokenizer with a 128-token sequence truncation and dynamic padding for 250k steps,
118
+ a 256-token sequence-truncation for 80k steps ([**Albertina 1.5B PTPT 256**](https://huggingface.co/PORTULAN/albertina-1b5-portuguese-ptpt-encoder-256)) and finally a 512-token sequence-truncation for 60k steps.
119
  These steps correspond to the equivalent setup of 48 hours on a2-megagpu-16gb Google Cloud A2 node for the 128-token input sequences, 24 hours of computation for the 256-token
120
  input sequences and 24 hours of computation for the 512-token input sequences.
121
  We opted for a learning rate of 1e-5 with linear decay and 10k warm-up steps.
 
125
  # Evaluation
126
 
127
 
128
+ We resorted to [HyperGlue-PT](?), a **PTPT version of the GLUE and SUPERGLUE** benchmark.
129
  We automatically translated the tasks from GLUE and SUPERGLUE using [DeepL Translate](https://www.deepl.com/), which specifically provides translation from English to PT-PT as an option.
130
 
131
  | Model | RTE (Accuracy) | WNLI (Accuracy)| MRPC (F1) | STS-B (Pearson) | COPA (Accuracy) | CB (F1) | MultiRC (F1) | BoolQ (Accuracy) |
132
  |-------------------------------|----------------|----------------|-----------|-----------------|-----------------|------------|--------------|------------------|
133
+ | **Albertina 1.5B PTPT** | **0.8809** | 0.4742 | 0.8457 | **0.9034** | **0.8433** | **0.7840** | **0.7688** | **0.8602** |
134
+ | **Albertina 1.5B PTPT 256** | 0.8809 | 0.5493 | 0.8752 | 0.8795 | 0.8400 | 0.5832 | 0.6791 | 0.8496 |
135
+ | **Albertina 900M PTPT** | 0.8339 | 0.4225 | **0.9171**| 0.8801 | 0.7033 | 0.6018 | 0.6728 | 0.8224 |
136
+ | **Albertina 100M PTPT** | 0.6919 | 0.4742 | 0.8047 | 0.8590 | n.a. | 0.4529 | 0.6481 | 0.7578 |
137
  ||||||||||
138
  | **DeBERTa 1.5B EN** | 0.8147 | 0.4554 | 0.8696 | 0.8557 | 0.5167 | 0.4901 | 0.6687 | 0.8347 |
139
  | **DeBERTa 100M EN** | 0.6029 | **0.5634** | 0.7802 | 0.8320 | n.a. | 0.4698 | 0.6368 | 0.6829 |
 
141
 
142
 
143
 
144
+ **para modelo PTBR**
145
 
146
  | Model | RTE (Accuracy) | WNLI (Accuracy)| MRPC (F1) | STS-B (Pearson) | COPA (Accuracy) | CB (F1) | MultiRC (F1) | BoolQ (Accuracy) |
147
  |-------------------------------|----------------|----------------|-----------|-----------------|-----------------|------------|--------------|------------------|
148
+ | **Albertina 1.5B PTBR** | **0.8676** | 0.4742 | 0.8622 | **0.9007** | 0.7767 | 0.6372 | **0.7667** | **0.8654** |
149
+ | **Albertina 1.5B PTBR 256** | 0.8123 | 0.4225 | 0.8638 | 0.8968 | **0.8533** | **0.6884** | 0.6799 | 0.8509 |
150
+ | **Albertina 900M PTBR** | 0.7545 | 0.4601 | **0.9071**| 0.8910 | 0.7767 | 0.5799 | 0.6731 | 0.8385 |
151
  | **BERTimbau (335M)** | 0.6446 | **0.5634** | 0.8873 | 0.8842 | 0.6933 | 0.5438 | 0.6787 | 0.7783 |
152
+ | **Albertina 100M PTBR** | 0.6582 | **0.5634** | 0.8149 | 0.8489 | n.a. | 0.4771 | 0.6469 | 0.7537 |
153
  ||||||||||
154
  | **DeBERTa 1.5B EN** | 0.7112 | **0.5634** | 0.8545 | 0.0123 | 0.5700 | 0.4307 | 0.3639 | 0.6217 |
155
  | **DeBERTa 100M EN** | 0.5716 | 0.5587 | 0.8060 | 0.8266 | n.a. | 0.4739 | 0.6391 | 0.6838 |