--- license: cc-by-nc-sa-4.0 datasets: - somosnlp/es-inclusive-language language: - es --- # es-inclusivo-translator This model is a fine-tuned version of [projecte-aina/aguila-7b](https://huggingface.co./projecte-aina/aguila-7b) on the dataset [somosnlp/es-inclusive-language](https://huggingface.co./datasets/somosnlp/es-inclusive-language). Languages are powerful tools to communicate ideas, but their use is not impartial. The selection of words carries inherent biases and reflects subjective perspectives. In some cases, language is wielded to enforce ideologies, marginalize certain groups, or promote specific political agendas. Spanish is not the exception to that. For instance, when we say “los alumnos” or “los ingenieros”, we are excluding women from those groups. Similarly, expressions such as “los gitanos” o “los musulmanes” perpetuate discrimination against these communities. In response to these linguistic challenges, this model offers a way to construct inclusive alternatives in accordance with official guidelines on inclusive language from various Spanish speaking countries. Its purpose is to provide grammatically correct and inclusive solutions to situations where our language choices might otherwise be exclusive. This is a tool that contributes to the fifth of the Sustainable Development Goals: Achieve gender equality and empower all women and girls. The model works in such a way that, given an input text, it returns the original text rewritten using inclusive language. It achieves the following results on the evaluation set: - Loss: 0.6030 ## Model description - **Developed by**: Andrés Martínez Fernández-Salguero ([andresmfs](https://huggingface.co./Andresmfs)), Imanuel Rozenberg (manu_20392), Gaia Quintana Fleitas (gaiaq), Josué Sauca (josue_sauca), Miguel López (wizmik12) - **Language(s)**: Spanish - **Fine-tuned from the model**: [projecte-aina/aguila-7b](https://huggingface.co./projecte-aina/aguila-7b) - **License**: cc-by-nc-sa-4.0 ## Social Impact An inclusive translator holds significant social impact by promoting equity and representation within texts. By rectifying biases ingrained in language and fostering inclusivity, it combats discrimination, amplifies the visibility of marginalized groups, and contributes to the cultivation of a more inclusive and respectful society. This is a tool that contributes to the fifth of the Sustainable Development Goals: Achieve gender equality and empower all women and girls. ## Intended uses & limitations ### Intended uses The general uses of this model are texts adaptations to inclusive language. It can be used to adapt news, blogposts, emails and official documents among others. ### How to use Here is how to use this model: ```python from transformers import AutoTokenizer from transformers import AutoModelForCausalLM import torch # Load tokenizer and model tokenizer = AutoTokenizer.from_pretrained('somosnlp/', trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained('somosnlp/', trust_remote_code=True, quantization_config=bnb_config, device_map="auto") # generation_config generation_config = model.generation_config generation_config.max_new_tokens = 100 generation_config.temperature = 0.7 generation_config.top_p = 0.7 generation_config.num_return_sequences = 1 generation_config.pad_token_id = tokenizer.eos_token_id generation_config.eos_token_id = tokenizer.eos_token_id # Define inference function def translate_es_inclusivo(exclusive_text): # generate input prompt eval_prompt = f"""Reescribe el siguiente texto utilizando lenguaje inclusivo.\n Texto: {exclusive_text}\n Texto en lenguaje inclusivo:""" # tokenize input model_input = tokenizer(eval_prompt, return_tensors="pt").to(model.device) # set max_new_tokens if necessary if len(model_input['input_ids'][0]) > 80: model.generation_config.max_new_tokens = len(model_input['input_ids'][0]) + 0.2 * len(model_input['input_ids'][0]) # get length of encoded prompt prompt_token_len = len(model_input['input_ids'][0]) # generate and decode with torch.no_grad(): inclusive_text = tokenizer.decode(model.generate(**model_input, generation_config=generation_config)[0][prompt_token_len:], skip_special_tokens=True) return inclusive_text ########## input_text = 'Los alumnos atienden a sus profesores' print(translate_es_inclusivo(input_text)) ``` As it is a heavy model, you may want to use it in 4-bits: ``` python from transformers import AutoTokenizer from transformers import AutoModelForCausalLM from transformers import BitsAndBytesConfig import torch ## Load model in 4bits # bnb_configuration bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=False) # model model = AutoModelForCausalLM.from_pretrained('somosnlp/', trust_remote_code=True, quantization_config=bnb_config, device_map="auto") # Load tokenizer tokenizer = AutoTokenizer.from_pretrained('somosnlp/', trust_remote_code=True) # generation_config generation_config = model.generation_config generation_config.max_new_tokens = 100 generation_config.temperature = 0.7 generation_config.top_p = 0.7 generation_config.num_return_sequences = 1 generation_config.pad_token_id = tokenizer.eos_token_id generation_config.eos_token_id = tokenizer.eos_token_id # Define inference function def translate_es_inclusivo(exclusive_text): # generate input prompt eval_prompt = f"""Reescribe el siguiente texto utilizando lenguaje inclusivo.\n Texto: {exclusive_text}\n Texto en lenguaje inclusivo:""" # tokenize input model_input = tokenizer(eval_prompt, return_tensors="pt").to(model.device) # set max_new_tokens if necessary if len(model_input['input_ids'][0]) > 80: model.generation_config.max_new_tokens = len(model_input['input_ids'][0]) + 0.2 * len(model_input['input_ids'][0]) # get length of encoded prompt prompt_token_len = len(model_input['input_ids'][0]) # generate and decode with torch.no_grad(): inclusive_text = tokenizer.decode(model.generate(**model_input, generation_config=generation_config)[0][prompt_token_len:], skip_special_tokens=True) return inclusive_text ########## input_text = 'Los alumnos atienden a sus profesores' print(translate_es_inclusivo(input_text)) ``` ### Limitations - The model has not been trained on long-complex texts. - It has been trained mostly with sentences where the terms to be modified are at the beginning of the sentence. - It returns only one option. - The model does not detect or modify hate speech ## Training and evaluation data Training and evaluation data can be found in [somosnlp/es-inclusive-language](https://huggingface.co./datasets/somosnlp/es-inclusive-language) ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0001 - train_batch_size: 8 - eval_batch_size: 8 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - num_epochs: 10 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:----:|:---------------:| | No log | 1.0 | 402 | 0.8020 | | 1.0274 | 2.0 | 804 | 0.7019 | | 0.6745 | 3.0 | 1206 | 0.6515 | | 0.5826 | 4.0 | 1608 | 0.6236 | | 0.5104 | 5.0 | 2010 | 0.6161 | | 0.5104 | 6.0 | 2412 | 0.6149 | | 0.4579 | 7.0 | 2814 | 0.6030 | | 0.4255 | 8.0 | 3216 | 0.6151 | | 0.3898 | 9.0 | 3618 | 0.6209 | | 0.3771 | 10.0 | 4020 | 0.6292 | ### Framework versions - Transformers 4.30.0 - Pytorch 2.2.2+cu121 - Datasets 2.18.0 - Tokenizers 0.13.3 ### Hardware - Nvidia T4 medium (8 vCPU, 30Gb RAM, 16Gb VRAM)