Update README.md

README.md

@@ -6,87 +6,190 @@ tags:
 model-index:
 - name: speecht5_tts-wolof
   results: []
+datasets:
+- galsenai/wolof_tts
+language:
+- wo
+pipeline_tag: text-to-speech
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
 # speecht5_tts-wolof
 
-This model is a fine-tuned version of [microsoft/speecht5_tts](https://huggingface.co/microsoft/speecht5_tts) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.3697
+This model is a fine-tuned version of [SpeechT5](https://huggingface.co/microsoft/speecht5_tts) for Text-to-Speech (TTS) on a Wolof dataset. It uses a custom tokenizer designed for Wolof and adjusts the baseline model's configuration to account for the new vocabulary introduced by the custom tokenizer. This version of SpeechT5 provides speech synthesis capabilities specifically tuned for the Wolof language.
 
 ## Model description
 
-More information needed
+This model is based on the `SpeechT5` architecture, which integrates both speech recognition and synthesis into a unified framework. It is fine-tuned for Text-to-Speech (TTS) using a custom-trained tokenizer and an adapted configuration that accounts for the unique vocabulary of the Wolof language. The fine-tuning process was carried out using a dataset containing text in Wolof to help the model synthesize speech that captures the nuances of the language.
+
+---
+
+### Installation Instructions for Users
+
+To install the necessary dependencies, run the following command:
+
+```bash
+!pip install transformers datasets
+```
+
+### Model Loading and Speech Generation Code
+
+```python
+import torch
+from transformers import SpeechT5ForTextToSpeech, SpeechT5Processor
+from transformers import SpeechT5HifiGan
+
+def load_speech_model(checkpoint="bilalfaye/speecht5_tts-wolof", vocoder_checkpoint="microsoft/speecht5_hifigan"):
+    """
+    Load the SpeechT5 model, processor, and vocoder for text-to-speech.
+    
+    Args:
+        checkpoint (str): The model checkpoint for SpeechT5 TTS.
+        vocoder_checkpoint (str): The checkpoint for the HiFi-GAN vocoder.
+    
+    Returns:
+        processor: The processor for the model.
+        model: The loaded SpeechT5 model.
+        vocoder: The loaded HiFi-GAN vocoder.
+        device: The device (CPU or GPU) the model is loaded on.
+    """
+    # Check for GPU availability and set device accordingly
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+    
+    # Load the SpeechT5 processor and model
+    processor = SpeechT5Processor.from_pretrained(checkpoint)
+    model = SpeechT5ForTextToSpeech.from_pretrained(checkpoint).to(device)  # Move model to the correct device
+
+    # Load the HiFi-GAN vocoder
+    vocoder = SpeechT5HifiGan.from_pretrained(vocoder_checkpoint).to(device)  # Move vocoder to the correct device
+
+    return processor, model, vocoder, device
+
+# Example usage
+processor, model, vocoder, device = load_speech_model()
+
+# Verify the device being used
+print(f"Model and vocoder loaded on device: {device}")
+
+from datasets import load_dataset
+# Load speaker embeddings (this dataset contains speaker-specific embeddings)
+embeddings_dataset = load_dataset("Matthijs/cmu-arctic-xvectors", split="validation")
+speaker_embedding = torch.tensor(embeddings_dataset[7306]["xvector"]).unsqueeze(0)
+
+import torch
+from transformers import SpeechT5ForTextToSpeech, SpeechT5Processor, SpeechT5HifiGan
+from IPython.display import Audio, display
+
+def generate_speech_from_text(text, 
+                              speaker_embedding=speaker_embedding,
+                              processor=processor,
+                              model=model,
+                              vocoder=vocoder):            
+    """
+    Generates speech from a given text using SpeechT5 and HiFi-GAN vocoder.
+
+    Args:
+        text (str): The input text to be converted to speech.
+        checkpoint (str): The model checkpoint for SpeechT5 TTS.
+        vocoder_checkpoint (str): The checkpoint for the HiFi-GAN vocoder.
+        speaker_embedding (torch.Tensor): The speaker embedding tensor.
+        processor (SpeechT5Processor): The processor for the model.
+        model (SpeechT5ForTextToSpeech): The loaded SpeechT5 model.
+        vocoder (SpeechT5HifiGan): The loaded HiFi-GAN vocoder.
+
+    Returns:
+        None
+    """
+    # Parameters for text-to-speech generation
+    max_text_positions = model.config.max_text_positions  # Token limit
+    max_length = model.config.max_length * 1.2  # Slightly extended max_length
+    min_length = max_length // 3  # Adjust based on max_length
+    num_beams = 7  # Use beam search for better quality
+    temperature = 0.6  # Reduce temperature for stability
+
+    # Tokenize the input text and move input tensor to the correct device
+    inputs = processor(text=text, return_tensors="pt", padding=True, truncation=True, max_length=max_text_positions)
+    inputs = {key: value.to(model.device) for key, value in inputs.items()}  # Move inputs to device
+
+    # Generate speech
+    speech = model.generate(
+        inputs["input_ids"],
+        speaker_embeddings=speaker_embedding.to(model.device),  # Ensure speaker_embedding is also on the correct device
+        vocoder=vocoder,
+        max_length=int(max_length),
+        min_length=int(min_length),
+        num_beams=num_beams,
+        temperature=temperature,
+        no_repeat_ngram_size=3,
+        repetition_penalty=1.5,
+        eos_token_id=None,
+        use_cache=True
+    )
+
+    # Detach the speech from the computation graph and move it to CPU
+    speech = speech.detach().cpu().numpy()
+
+    # Play the generated speech using IPython Audio
+    display(Audio(speech, rate=16000))
+
+
+# Example usage
+text = "ñu ne ñoom ñooy nattukaay satélite yi"
+generate_speech_from_text(text)
+```
 
 ## Intended uses & limitations
 
-More information needed
+### Intended uses:
+- **Text-to-Speech Generation**: This model can be used to convert Wolof text into natural-sounding speech. It can be integrated into applications that require voice interfaces, virtual assistants, or voice synthesis for Wolof-speaking communities.
+  
+### Limitations:
+- **Limited Scope**: The model has been specifically fine-tuned for Wolof and may not perform well with other languages or accents.
+- **Data Availability**: While the model was fine-tuned on a Wolof dataset, the quality of the generated speech may vary depending on the complexity of the input text and the dataset used for training.
+- **Vocabulary and Tokenizer Constraints**: The tokenizer was specially trained for Wolof, so it may not handle out-of-vocabulary words or unknown characters effectively.
 
 ## Training and evaluation data
 
-More information needed
+The model was fine-tuned on a custom dataset consisting of text in the Wolof language. This dataset was used to adjust the model to generate speech that accurately reflects the phonetic and syntactic properties of Wolof.
 
 ## Training procedure
 
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 1e-05
-- train_batch_size: 8
-- eval_batch_size: 2
-- seed: 42
-- gradient_accumulation_steps: 8
-- total_train_batch_size: 64
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_steps: 500
-- training_steps: 255000
-- mixed_precision_training: Native AMP
+
+- **Learning Rate**: 1e-05
+- **Training Batch Size**: 8
+- **Evaluation Batch Size**: 2
+- **Seed**: 42
+- **Gradient Accumulation Steps**: 8
+- **Total Train Batch Size**: 64
+- **Optimizer**: Adam with betas=(0.9, 0.999) and epsilon=1e-08
+- **Learning Rate Scheduler Type**: Linear
+- **Warmup Steps**: 500
+- **Training Steps**: 255000
+- **Mixed Precision Training**: Native AMP
 
 ### Training results
 
-| Training Loss | Epoch   | Step  | Validation Loss |
-|:-------------:|:-------:|:-----:|:---------------:|
-| 0.5033        | 0.9989  | 562   | 0.4592          |
-| 0.471         | 1.9996  | 1125  | 0.4319          |
-| 0.4603        | 2.9984  | 1687  | 0.4224          |
-| 0.4496        | 3.9991  | 2250  | 0.4135          |
-| 0.4418        | 4.9998  | 2813  | 0.4090          |
-| 0.4296        | 5.9987  | 3375  | 0.4001          |
-| 0.425         | 6.9993  | 3938  | 0.3950          |
-| 0.4197        | 8.0     | 4501  | 0.3927          |
-| 0.4173        | 8.9989  | 5063  | 0.3888          |
-| 0.4133        | 9.9996  | 5626  | 0.3852          |
-| 0.4112        | 10.9984 | 6188  | 0.3832          |
-| 0.4072        | 11.9991 | 6751  | 0.3808          |
-| 0.404         | 12.9998 | 7314  | 0.3788          |
-| 0.4055        | 13.9987 | 7876  | 0.3792          |
-| 0.401         | 14.9993 | 8439  | 0.3759          |
-| 0.3988        | 16.0    | 9002  | 0.3755          |
-| 0.3984        | 16.9989 | 9564  | 0.3761          |
-| 0.3992        | 17.9996 | 10127 | 0.3735          |
-| 0.392         | 18.9984 | 10689 | 0.3731          |
-| 0.393         | 19.9991 | 11252 | 0.3730          |
-| 0.3945        | 20.9998 | 11815 | 0.3713          |
-| 0.3929        | 21.9987 | 12377 | 0.3720          |
-| 0.3919        | 22.9993 | 12940 | 0.3736          |
-| 0.3907        | 24.0    | 13503 | 0.3702          |
-| 0.3893        | 24.9989 | 14065 | 0.3700          |
-| 0.3894        | 25.9996 | 14628 | 0.3707          |
-| 0.39          | 26.9984 | 15190 | 0.3687          |
-| 0.3858        | 27.9991 | 15753 | 0.3712          |
-| 0.3874        | 28.9998 | 16316 | 0.3669          |
-| 0.3887        | 29.9987 | 16878 | 0.3685          |
-| 0.3854        | 30.9993 | 17441 | 0.3670          |
-| 0.3856        | 32.0    | 18004 | 0.3697          |
-
-
-### Framework versions
-
-- Transformers 4.41.2
-- Pytorch 2.4.0+cu121
-- Datasets 3.2.0
-- Tokenizers 0.19.1
+| Epoch | Training Loss | Validation Loss |
+|-------|---------------|-----------------|
+| 26    | 0.3894        | 0.3687          |
+| 27    | 0.3858        | 0.3712          |
+| 28    | 0.3874        | 0.3669          |
+| 29    | 0.3887        | 0.3685          |
+| 30    | 0.3854        | 0.3670          |
+| 32    | 0.3856        | 0.3697          |
+
+The evaluation table only includes the last 5 epochs as requested.
+
+### Framework version
+
+- **Transformers**: 4.41.2
+- **PyTorch**: 2.4.0+cu121
+- **Datasets**: 3.2.0
+- **Tokenizers**: 0.19.1
+
+
+# Author
+
+- **Bilal FAYE**