patrickvonplaten commited on
Commit
0b5509c
1 Parent(s): d9c2864

upload model

Browse files
README.md ADDED
@@ -0,0 +1,184 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: es
3
+ datasets:
4
+ - common_voice
5
+ metrics:
6
+ - wer
7
+ - cer
8
+ tags:
9
+ - audio
10
+ - automatic-speech-recognition
11
+ - speech
12
+ - xlsr-fine-tuning-week
13
+ license: apache-2.0
14
+ model-index:
15
+ - name: XLSR Wav2Vec2 Spanish by Jonatas Grosman
16
+ results:
17
+ - task:
18
+ name: Speech Recognition
19
+ type: automatic-speech-recognition
20
+ dataset:
21
+ name: Common Voice es
22
+ type: common_voice
23
+ args: es
24
+ metrics:
25
+ - name: Test WER
26
+ type: wer
27
+ value: 8.81
28
+ - name: Test CER
29
+ type: cer
30
+ value: 2.70
31
+ ---
32
+
33
+ # Wav2Vec2-Large-XLSR-53-Spanish
34
+
35
+ Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Spanish using the [Common Voice](https://huggingface.co/datasets/common_voice).
36
+ When using this model, make sure that your speech input is sampled at 16kHz.
37
+
38
+ This model has been fine-tuned thanks to the GPU credits generously given by the [OVHcloud](https://www.ovhcloud.com/en/public-cloud/ai-training/) :)
39
+
40
+ The script used for training can be found here: https://github.com/jonatasgrosman/wav2vec2-sprint
41
+
42
+ ## Usage
43
+
44
+ The model can be used directly (without a language model) as follows...
45
+
46
+ Using the [ASRecognition](https://github.com/jonatasgrosman/asrecognition) library:
47
+
48
+ ```python
49
+ from asrecognition import ASREngine
50
+
51
+ asr = ASREngine("es", model_path="jonatasgrosman/wav2vec2-large-xlsr-53-spanish")
52
+
53
+ audio_paths = ["/path/to/file.mp3", "/path/to/another_file.wav"]
54
+ transcriptions = asr.transcribe(audio_paths)
55
+ ```
56
+
57
+ Writing your own inference script:
58
+
59
+ ```python
60
+ import torch
61
+ import librosa
62
+ from datasets import load_dataset
63
+ from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
64
+
65
+ LANG_ID = "es"
66
+ MODEL_ID = "jonatasgrosman/wav2vec2-large-xlsr-53-spanish"
67
+ SAMPLES = 10
68
+
69
+ test_dataset = load_dataset("common_voice", LANG_ID, split=f"test[:{SAMPLES}]")
70
+
71
+ processor = Wav2Vec2Processor.from_pretrained(MODEL_ID)
72
+ model = Wav2Vec2ForCTC.from_pretrained(MODEL_ID)
73
+
74
+ # Preprocessing the datasets.
75
+ # We need to read the audio files as arrays
76
+ def speech_file_to_array_fn(batch):
77
+ speech_array, sampling_rate = librosa.load(batch["path"], sr=16_000)
78
+ batch["speech"] = speech_array
79
+ batch["sentence"] = batch["sentence"].upper()
80
+ return batch
81
+
82
+ test_dataset = test_dataset.map(speech_file_to_array_fn)
83
+ inputs = processor(test_dataset["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
84
+
85
+ with torch.no_grad():
86
+ logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
87
+
88
+ predicted_ids = torch.argmax(logits, dim=-1)
89
+ predicted_sentences = processor.batch_decode(predicted_ids)
90
+
91
+ for i, predicted_sentence in enumerate(predicted_sentences):
92
+ print("-" * 100)
93
+ print("Reference:", test_dataset[i]["sentence"])
94
+ print("Prediction:", predicted_sentence)
95
+ ```
96
+
97
+ | Reference | Prediction |
98
+ | ------------- | ------------- |
99
+ | HABITA EN AGUAS POCO PROFUNDAS Y ROCOSAS. | HABITAN AGUAS POCO PROFUNDAS Y ROCOSAS |
100
+ | OPERA PRINCIPALMENTE VUELOS DE CABOTAJE Y REGIONALES DE CARGA. | OPERA PRINCIPALMENTE VUELO DE CARBOTAJES Y REGIONALES DE CARGAN |
101
+ | PARA VISITAR CONTACTAR PRIMERO CON LA DIRECCIÓN. | PARA VISITAR CONTACTAR PRIMERO CON LA DIRECCIÓN |
102
+ | TRES | TRES |
103
+ | REALIZÓ LOS ESTUDIOS PRIMARIOS EN FRANCIA, PARA CONTINUAR LUEGO EN ESPAÑA. | REALIZÓ LOS ESTUDIOS PRIMARIOS EN FRANCIA PARA CONTINUAR LUEGO EN ESPAÑA |
104
+ | EN LOS AÑOS QUE SIGUIERON, ESTE TRABAJO ESPARTA PRODUJO DOCENAS DE BUENOS JUGADORES. | EN LOS AÑOS QUE SIGUIERON ESTE TRABAJO ESPARTA PRODUJO DOCENA DE BUENOS JUGADORES |
105
+ | SE ESTÁ TRATANDO DE RECUPERAR SU CULTIVO EN LAS ISLAS CANARIAS. | SE ESTÓ TRATANDO DE RECUPERAR SU CULTIVO EN LAS ISLAS CANARIAS |
106
+ | SÍ | SÍ |
107
+ | "FUE ""SACADA"" DE LA SERIE EN EL EPISODIO ""LEAD"", EN QUE ALEXANDRA CABOT REGRESÓ." | FUE SACADA DE LA SERIE EN EL EPISODIO LEED EN QUE ALEXANDRA KAOT REGRESÓ |
108
+ | SE UBICAN ESPECÍFICAMENTE EN EL VALLE DE MOKA, EN LA PROVINCIA DE BIOKO SUR. | SE UBICAN ESPECÍFICAMENTE EN EL VALLE DE MOCA EN LA PROVINCIA DE PÍOCOSUR |
109
+
110
+ ## Evaluation
111
+
112
+ The model can be evaluated as follows on the Spanish test data of Common Voice.
113
+
114
+ ```python
115
+ import torch
116
+ import re
117
+ import librosa
118
+ from datasets import load_dataset, load_metric
119
+ from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
120
+
121
+ LANG_ID = "es"
122
+ MODEL_ID = "jonatasgrosman/wav2vec2-large-xlsr-53-spanish"
123
+ DEVICE = "cuda"
124
+
125
+ CHARS_TO_IGNORE = [",", "?", "¿", ".", "!", "¡", ";", ";", ":", '""', "%", '"', "�", "ʿ", "·", "჻", "~", "՞",
126
+ "؟", "،", "।", "॥", "«", "»", "„", "“", "”", "「", "」", "‘", "’", "《", "》", "(", ")", "[", "]",
127
+ "{", "}", "=", "`", "_", "+", "<", ">", "…", "–", "°", "´", "ʾ", "‹", "›", "©", "®", "—", "→", "。",
128
+ "、", "﹂", "﹁", "‧", "~", "﹏", ",", "{", "}", "(", ")", "[", "]", "【", "】", "‥", "〽",
129
+ "『", "』", "〝", "〟", "⟨", "⟩", "〜", ":", "!", "?", "��", "؛", "/", "\\", "º", "−", "^", "ʻ", "ˆ"]
130
+
131
+ test_dataset = load_dataset("common_voice", LANG_ID, split="test")
132
+
133
+ wer = load_metric("wer.py") # https://github.com/jonatasgrosman/wav2vec2-sprint/blob/main/wer.py
134
+ cer = load_metric("cer.py") # https://github.com/jonatasgrosman/wav2vec2-sprint/blob/main/cer.py
135
+
136
+ chars_to_ignore_regex = f"[{re.escape(''.join(CHARS_TO_IGNORE))}]"
137
+
138
+ processor = Wav2Vec2Processor.from_pretrained(MODEL_ID)
139
+ model = Wav2Vec2ForCTC.from_pretrained(MODEL_ID)
140
+ model.to(DEVICE)
141
+
142
+ # Preprocessing the datasets.
143
+ # We need to read the audio files as arrays
144
+ def speech_file_to_array_fn(batch):
145
+ with warnings.catch_warnings():
146
+ warnings.simplefilter("ignore")
147
+ speech_array, sampling_rate = librosa.load(batch["path"], sr=16_000)
148
+ batch["speech"] = speech_array
149
+ batch["sentence"] = re.sub(chars_to_ignore_regex, "", batch["sentence"]).upper()
150
+ return batch
151
+
152
+ test_dataset = test_dataset.map(speech_file_to_array_fn)
153
+
154
+ # Preprocessing the datasets.
155
+ # We need to read the audio files as arrays
156
+ def evaluate(batch):
157
+ inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
158
+
159
+ with torch.no_grad():
160
+ logits = model(inputs.input_values.to(DEVICE), attention_mask=inputs.attention_mask.to(DEVICE)).logits
161
+
162
+ pred_ids = torch.argmax(logits, dim=-1)
163
+ batch["pred_strings"] = processor.batch_decode(pred_ids)
164
+ return batch
165
+
166
+ result = test_dataset.map(evaluate, batched=True, batch_size=8)
167
+
168
+ predictions = [x.upper() for x in result["pred_strings"]]
169
+ references = [x.upper() for x in result["sentence"]]
170
+
171
+ print(f"WER: {wer.compute(predictions=predictions, references=references, chunk_size=1000) * 100}")
172
+ print(f"CER: {cer.compute(predictions=predictions, references=references, chunk_size=1000) * 100}")
173
+ ```
174
+
175
+ **Test Result**:
176
+
177
+ In the table below I report the Word Error Rate (WER) and the Character Error Rate (CER) of the model. I ran the evaluation script described above on other models as well (on 2021-04-22). Note that the table below may show different results from those already reported, this may have been caused due to some specificity of the other evaluation scripts used.
178
+
179
+ | Model | WER | CER |
180
+ | ------------- | ------------- | ------------- |
181
+ | jonatasgrosman/wav2vec2-large-xlsr-53-spanish | **8.81%** | **2.70%** |
182
+ | pcuenq/wav2vec2-large-xlsr-53-es | 10.55% | 3.20% |
183
+ | facebook/wav2vec2-large-xlsr-53-spanish | 16.99% | 5.40% |
184
+ | mrm8488/wav2vec2-large-xlsr-53-spanish | 19.20% | 5.96% |
alphabet.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"labels": ["", "<s>", "</s>", "\u2047", " ", "'", "-", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "\u00e1", "\u00e9", "\u00ed", "\u00f1", "\u00f3", "\u00f6", "\u00fa", "\u00fc"], "is_bpe": false}
config.json ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "facebook/wav2vec2-large-xlsr-53",
3
+ "activation_dropout": 0.05,
4
+ "apply_spec_augment": true,
5
+ "architectures": [
6
+ "Wav2Vec2ForCTC"
7
+ ],
8
+ "attention_dropout": 0.1,
9
+ "bos_token_id": 1,
10
+ "conv_bias": true,
11
+ "conv_dim": [
12
+ 512,
13
+ 512,
14
+ 512,
15
+ 512,
16
+ 512,
17
+ 512,
18
+ 512
19
+ ],
20
+ "conv_kernel": [
21
+ 10,
22
+ 3,
23
+ 3,
24
+ 3,
25
+ 3,
26
+ 2,
27
+ 2
28
+ ],
29
+ "conv_stride": [
30
+ 5,
31
+ 2,
32
+ 2,
33
+ 2,
34
+ 2,
35
+ 2,
36
+ 2
37
+ ],
38
+ "ctc_loss_reduction": "mean",
39
+ "ctc_zero_infinity": true,
40
+ "do_stable_layer_norm": true,
41
+ "eos_token_id": 2,
42
+ "feat_extract_activation": "gelu",
43
+ "feat_extract_dropout": 0.0,
44
+ "feat_extract_norm": "layer",
45
+ "feat_proj_dropout": 0.05,
46
+ "final_dropout": 0.0,
47
+ "gradient_checkpointing": true,
48
+ "hidden_act": "gelu",
49
+ "hidden_dropout": 0.05,
50
+ "hidden_size": 1024,
51
+ "initializer_range": 0.02,
52
+ "intermediate_size": 4096,
53
+ "layer_norm_eps": 1e-05,
54
+ "layerdrop": 0.05,
55
+ "mask_channel_length": 10,
56
+ "mask_channel_min_space": 1,
57
+ "mask_channel_other": 0.0,
58
+ "mask_channel_prob": 0.0,
59
+ "mask_channel_selection": "static",
60
+ "mask_feature_length": 10,
61
+ "mask_feature_prob": 0.0,
62
+ "mask_time_length": 10,
63
+ "mask_time_min_space": 1,
64
+ "mask_time_other": 0.0,
65
+ "mask_time_prob": 0.05,
66
+ "mask_time_selection": "static",
67
+ "model_type": "wav2vec2",
68
+ "num_attention_heads": 16,
69
+ "num_conv_pos_embedding_groups": 16,
70
+ "num_conv_pos_embeddings": 128,
71
+ "num_feat_extract_layers": 7,
72
+ "num_hidden_layers": 24,
73
+ "pad_token_id": 0,
74
+ "transformers_version": "4.7.0.dev0",
75
+ "vocab_size": 41
76
+ }
language_model/attrs.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"alpha": 0.5, "beta": 1.5, "unk_score_offset": -10.0, "score_boundary": true}
language_model/kenLM.arpa ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7faed063138421da63215f92a08664c1bb499cf97b541e50beac1093627b59b7
3
+ size 963830967
language_model/unigrams.txt ADDED
The diff for this file is too large to render. See raw diff
 
preprocessor_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_normalize": true,
3
+ "feature_extractor_type": "Wav2Vec2FeatureExtractor",
4
+ "feature_size": 1,
5
+ "padding_side": "right",
6
+ "padding_value": 0.0,
7
+ "return_attention_mask": true,
8
+ "sampling_rate": 16000
9
+ }
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c05f795e83023b4ff4b23d23c5fb646f9828ae0d967de5e858710dc84ebc5c2b
3
+ size 1262085873
special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>"}
vocab.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"<pad>": 0, "<s>": 1, "</s>": 2, "<unk>": 3, "|": 4, "'": 5, "-": 6, "A": 7, "B": 8, "C": 9, "D": 10, "E": 11, "F": 12, "G": 13, "H": 14, "I": 15, "J": 16, "K": 17, "L": 18, "M": 19, "N": 20, "O": 21, "P": 22, "Q": 23, "R": 24, "S": 25, "T": 26, "U": 27, "V": 28, "W": 29, "X": 30, "Y": 31, "Z": 32, "Á": 33, "É": 34, "Í": 35, "Ñ": 36, "Ó": 37, "Ö": 38, "Ú": 39, "Ü": 40}