File size: 7,810 Bytes
16130a4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 |
---
license: cc-by-nc-sa-4.0
datasets:
- somosnlp/es-inclusive-language
language:
- es
---
# es-inclusivo-translator
This model is a fine-tuned version of [projecte-aina/aguila-7b](https://huggingface.co./projecte-aina/aguila-7b) on the dataset [somosnlp/es-inclusive-language](https://huggingface.co./datasets/somosnlp/es-inclusive-language).
Languages are powerful tools to communicate ideas, but their use is not impartial. The selection of words carries inherent biases and reflects subjective perspectives. In some cases, language is wielded to enforce ideologies, marginalize certain groups, or promote specific political agendas.
Spanish is not the exception to that. For instance, when we say “los alumnos” or “los ingenieros”, we are excluding women from those groups. Similarly, expressions such as “los gitanos” o “los musulmanes” perpetuate discrimination against these communities.
In response to these linguistic challenges, this model offers a way to construct inclusive alternatives in accordance with official guidelines on inclusive language from various Spanish speaking countries. Its purpose is to provide grammatically correct and inclusive solutions to situations where our language choices might otherwise be exclusive.
This is a tool that contributes to the fifth of the Sustainable Development Goals: Achieve gender equality and empower all women and girls.
The model works in such a way that, given an input text, it returns the original text rewritten using inclusive language.
It achieves the following results on the evaluation set:
- Loss: 0.6030
## Model description
- **Developed by**: Andrés Martínez Fernández-Salguero ([andresmfs](https://huggingface.co./Andresmfs)), Imanuel Rozenberg (manu_20392), Gaia Quintana Fleitas (gaiaq), Josué Sauca (josue_sauca), Miguel López (wizmik12)
- **Language(s)**: Spanish
- **Fine-tuned from the model**: [projecte-aina/aguila-7b](https://huggingface.co./projecte-aina/aguila-7b)
- **License**: cc-by-nc-sa-4.0
## Social Impact
An inclusive translator holds significant social impact by promoting equity and representation within texts. By rectifying biases ingrained in language and fostering inclusivity, it combats discrimination, amplifies the visibility of marginalized groups, and contributes to the cultivation of a more inclusive and respectful society.
This is a tool that contributes to the fifth of the Sustainable Development Goals: Achieve gender equality and empower all women and girls.
## Intended uses & limitations
More information needed
### How to use
Here is how to use this model:
```python
from transformers import AutoTokenizer
from transformers import AutoModelForCausalLM
import torch
# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained('somosnlp/', trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained('somosnlp/', trust_remote_code=True,
quantization_config=bnb_config,
device_map="auto")
# generation_config
generation_config = model.generation_config
generation_config.max_new_tokens = 100
generation_config.temperature = 0.7
generation_config.top_p = 0.7
generation_config.num_return_sequences = 1
generation_config.pad_token_id = tokenizer.eos_token_id
generation_config.eos_token_id = tokenizer.eos_token_id
# Define inference function
def translate_es_inclusivo(exclusive_text):
# generate input prompt
eval_prompt = f"""Reescribe el siguiente texto utilizando lenguaje inclusivo.\n
Texto: {exclusive_text}\n
Texto en lenguaje inclusivo:"""
# tokenize input
model_input = tokenizer(eval_prompt, return_tensors="pt").to(model.device)
# set max_new_tokens if necessary
if len(model_input['input_ids'][0]) > 80:
model.generation_config.max_new_tokens = len(model_input['input_ids'][0]) + 0.2 * len(model_input['input_ids'][0])
# get length of encoded prompt
prompt_token_len = len(model_input['input_ids'][0])
# generate and decode
with torch.no_grad():
inclusive_text = tokenizer.decode(model.generate(**model_input, generation_config=generation_config)[0][prompt_token_len:],
skip_special_tokens=True)
return inclusive_text
##########
input_text = 'Los alumnos atienden a sus profesores'
print(translate_es_inclusivo(input_text))
```python
As it is a heavy model, you may want to use it in 4-bits:
```python
from transformers import AutoTokenizer
from transformers import AutoModelForCausalLM
from transformers import BitsAndBytesConfig
import torch
## Load model in 4bits
# bnb_configuration
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=False)
# model
model = AutoModelForCausalLM.from_pretrained('somosnlp/', trust_remote_code=True,
quantization_config=bnb_config,
device_map="auto")
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained('somosnlp/', trust_remote_code=True)
# generation_config
generation_config = model.generation_config
generation_config.max_new_tokens = 100
generation_config.temperature = 0.7
generation_config.top_p = 0.7
generation_config.num_return_sequences = 1
generation_config.pad_token_id = tokenizer.eos_token_id
generation_config.eos_token_id = tokenizer.eos_token_id
# Define inference function
def translate_es_inclusivo(exclusive_text):
# generate input prompt
eval_prompt = f"""Reescribe el siguiente texto utilizando lenguaje inclusivo.\n
Texto: {exclusive_text}\n
Texto en lenguaje inclusivo:"""
# tokenize input
model_input = tokenizer(eval_prompt, return_tensors="pt").to(model.device)
# set max_new_tokens if necessary
if len(model_input['input_ids'][0]) > 80:
model.generation_config.max_new_tokens = len(model_input['input_ids'][0]) + 0.2 * len(model_input['input_ids'][0])
# get length of encoded prompt
prompt_token_len = len(model_input['input_ids'][0])
# generate and decode
with torch.no_grad():
inclusive_text = tokenizer.decode(model.generate(**model_input, generation_config=generation_config)[0][prompt_token_len:],
skip_special_tokens=True)
return inclusive_text
##########
input_text = 'Los alumnos atienden a sus profesores'
print(translate_es_inclusivo(input_text))
```python
## Training and evaluation data
Training and evaluation data can be found in [somosnlp/es-inclusive-language](https://huggingface.co./datasets/somosnlp/es-inclusive-language)
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log | 1.0 | 402 | 0.8020 |
| 1.0274 | 2.0 | 804 | 0.7019 |
| 0.6745 | 3.0 | 1206 | 0.6515 |
| 0.5826 | 4.0 | 1608 | 0.6236 |
| 0.5104 | 5.0 | 2010 | 0.6161 |
| 0.5104 | 6.0 | 2412 | 0.6149 |
| 0.4579 | 7.0 | 2814 | 0.6030 |
| 0.4255 | 8.0 | 3216 | 0.6151 |
| 0.3898 | 9.0 | 3618 | 0.6209 |
| 0.3771 | 10.0 | 4020 | 0.6292 |
### Framework versions
- Transformers 4.30.0
- Pytorch 2.2.2+cu121
- Datasets 2.18.0
- Tokenizers 0.13.3 |