ai4bharat
/

indic-parler-tts-pretrained

@@ -1,199 +1,282 @@
 ---
 library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
 library_name: transformers
+tags:
+- text-to-speech
+- annotation
+license: apache-2.0
+language:
+- en
+- as
+- bn
+- gu
+- hi
+- kn
+- ks
+- or
+- ml
+- mr
+- ne
+- pa
+- sa
+- sd
+- ta
+- te
+- ur
+- om
+pipeline_tag: text-to-speech
+inference: false
+datasets:
+- ai4b-hf/GLOBE-annotated
 ---
+<img src="https://huggingface.co/datasets/parler-tts/images/resolve/main/thumbnail.png" alt="Parler Logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
+# Indic Parler-TTS
+<a target="_blank" href="https://huggingface.co/spaces/PHBJT/multi_parler_tts">
+  <img src="https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg" alt="Open in HuggingFace"/>
+</a>
+**Indic Parler-TTS** is a multilingual Indic extension of [Parler-TTS Mini](https://huggingface.co/parler-tts/parler-tts-mini-v1.1).
+It is a fine-tuned version, trained on a **8,385 hours** multilingual Indic and English dataset.
+**Indic Parler-TTS Mini** can officially speak in 20 Indic languages, making it comprehensive for regional language technologies, and in English. The **21 languages** supported are: Assamese, Bengali, Bodo, Chhattisgarhi, Dogri, English, Gujarati, Hindi, Kannada, Konkani, Maithili, Malayalam, Manipuri, Marathi, Nepali, Odia, Sanskrit, Santali, Sindhi, Tamil, Telugu, and Urdu.
+Thanks to its **better prompt tokenizer**, it can easily be extended to other languages. This tokenizer has a larger vocabulary and handles byte fallback, which simplifies multilingual training.
+🚨 This work is the result of a collaboration between the **HuggingFace audio team** and the **[AI4Bharat](https://ai4bharat.iitm.ac.in/) team**. 🚨
+## 📖 Quick Index
+* [👨‍💻 Installation](#👨‍💻-installation)
+* [🎲 Using a random voice](#🎲-random-voice)
+* [🌍 Switching languages](#[🌍-switching-languages)
+* [🎯 Using a specific speaker](#🎯-using-a-specific-speaker)
+* [📐Evaluation](#📐-evaluation)
+* [Motivation](#motivation)
+* [Optimizing inference](https://github.com/huggingface/parler-tts/blob/main/INFERENCE.md)
+## 🛠️ Usage
+🚨 Unlike previous versions of Parler-TTS, here we use two tokenizers - one for the prompt and one for the description. 🚨
+### 👨‍💻 Installation
+Using Parler-TTS is as simple as "bonjour". Simply install the library once:
+```sh
+pip install git+https://github.com/huggingface/parler-tts.git
+```
+### 🎲 Random voice
+**Parler-TTS** has been trained to generate speech with features that can be controlled with a simple text prompt, for example:
+```py
+import torch
+from parler_tts import ParlerTTSForConditionalGeneration
+from transformers import AutoTokenizer
+import soundfile as sf
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+model = ParlerTTSForConditionalGeneration.from_pretrained("ai4b-hf/indic-parler-tts-pretrained-v3").to(device)
+tokenizer = AutoTokenizer.from_pretrained("ai4b-hf/indic-parler-tts-pretrained-v3")
+description_tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder._name_or_path)
+prompt = "Hey, how are you doing today?"
+description = "A female speaker with a British accent delivers a slightly expressive and animated speech with a moderate speed and pitch. The recording is of very high quality, with the speaker's voice sounding clear and very close up."
+input_ids = description_tokenizer(description, return_tensors="pt").input_ids.to(device)
+prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
+generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
+audio_arr = generation.cpu().numpy().squeeze()
+sf.write("indic_tts_out.wav", audio_arr, model.config.sampling_rate)
+```
+### 🌍 Switching languages
+The template automatically adapts to the language it detects in the prompt. You don't need to specify the language you want to use. For example, to switch to Hindi, simply use an Hindi prompt:
+```py
+import torch
+from parler_tts import ParlerTTSForConditionalGeneration
+from transformers import AutoTokenizer
+import soundfile as sf
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+model = ParlerTTSForConditionalGeneration.from_pretrained("ai4b-hf/indic-parler-tts-pretrained-v3").to(device)
+tokenizer = AutoTokenizer.from_pretrained("ai4b-hf/indic-parler-tts-pretrained-v3")
+description_tokenizer = AutoTokenizer.from_pretrained(model.config.text_encoder._name_or_path)
+prompt = "अरे, तुम आज कैसे हो?"
+description = "A female speaker delivers a slightly expressive and animated speech with a moderate speed and pitch. The recording is of very high quality, with the speaker's voice sounding clear and very close up."
+input_ids = description_tokenizer(description, return_tensors="pt").input_ids.to(device)
+prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
+generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
+audio_arr = generation.cpu().numpy().squeeze()
+sf.write("indic_tts_out.wav", audio_arr, model.config.sampling_rate)
+```
+### 🎯 Using a specific speaker
+To ensure speaker consistency across generations, this checkpoint was also trained on pre-determined speakers, characterized by name (e.g. Rohit, Karan, Leela, Maya, Sita, ...).
+To take advantage of this, simply adapt your text description to specify which speaker to use: `Divya's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise.`
+```py
+import torch
+from parler_tts import ParlerTTSForConditionalGeneration
+from transformers import AutoTokenizer
+import soundfile as sf
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+model = ParlerTTSForConditionalGeneration.from_pretrained("parler-tts/parler-tts-mini-v1").to(device)
+tokenizer = AutoTokenizer.from_pretrained("parler-tts/parler-tts-mini-v1")
+prompt = "अरे, तुम आज कैसे हो?"
+description = "Divya's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise."
+input_ids = tokenizer(description, return_tensors="pt").input_ids.to(device)
+prompt_input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
+generation = model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids)
+audio_arr = generation.cpu().numpy().squeeze()
+sf.write("parler_tts_out.wav", audio_arr, model.config.sampling_rate)
+```
+Here is the list of speakers:
+| Language      | Male     | Female   |
+|---------------|----------|----------|
+| Assamese      | Amit     | Aditi    |
+| Bengali       | Arjun    | Sita     |
+| Bodo          | Bikram   | Maya     |
+| Chhattisgarhi | Bhanu    | Champa   |
+| Dogri         | Karan    | Leela    |
+| English       | Thoma    | Mary     |
+| Gujarati      | Yash     | Neha     |
+| Hindi         | Rohit    | Divya    |
+| Kannada       | Suresh   | Anu      |
+| Kashmiri      | Rohit    | Shabnam  |
+| Konkani       | Deepak   | Sunita   |
+| Maithili      | Anil     | Divya    |
+| Malayalam     | Harish   | Anjali   |
+| Manipuri      | Ranjit   | Laishram |
+| Marathi       | Sanjay   | Sunita   |
+| Nepali        | Ram      | Amrita   |
+| Odia          | Manas    | Debjani  |
+| Punjabi       | Gurpreet | Divjot   |
+| Sanskrit      | Aryan    | Vasudha  |
+| Santali       | Arjun    | Pushpa   |
+| Sindhi        | Rajesh   | Rekha    |
+| Tamil         | NA       | Jaya     |
+| Telugu        | Prakash  | Lalitha  |
+| Urdu          | Rohit    | Zainab   |
+**Tips**:
+* We've set up an [inference guide](https://github.com/huggingface/parler-tts/blob/main/INFERENCE.md) to make generation faster. Think SDPA, torch.compile, batching and streaming!
+* Include the term "very clear audio" to generate the highest quality audio, and "very noisy audio" for high levels of background noise
+* Punctuation can be used to control the prosody of the generations, e.g. use commas to add small breaks in speech
+* The remaining speech features (gender, speaking rate, pitch and reverberation) can be controlled directly through the prompt
+## 📐 Evaluation
+## Motivation
+Parler-TTS is a reproduction of work from the paper [Natural language guidance of high-fidelity text-to-speech with synthetic annotations](https://www.text-description-to-speech.com) by Dan Lyth and Simon King, from Stability AI and Edinburgh University respectively.
+Parler-TTS was released alongside:
+* [The Parler-TTS repository](https://github.com/huggingface/parler-tts) - you can train and fine-tuned your own version of the model.
+* [The Data-Speech repository](https://github.com/huggingface/dataspeech) - a suite of utility scripts designed to annotate speech datasets.
+* [The Parler-TTS organization](https://huggingface.co/parler-tts) - where you can find the annotated datasets as well as the future checkpoints.
+## Training dataset
+- **Description**:
+  The model was trained on an internal **Indic-Parler-Dataset**, a large-scale multilingual speech corpus designed to train the **Indic Parler-TTS** model. It provides comprehensive coverage of 24 languages, which includes all the 22 official languages of India along with Chattisgarhi and English, making it an invaluable resource for speech technologies focused on the subcontinent.
+- **Key Statistics**:
+| Dataset         | Duration (hrs) | Languages Covered | No. of Utterances | License      |
+|:---------------:|:--------------:|:-----------------:|:-----------------:|:------------:|
+| GLOBE           | 535.0          | 1                 | 581,725           | CC V1        |
+| IndicTTS        | 382.0          | 12                | 220,606           | CC BY 4.0    |
+| IndicVoices     | 2,651.0        | 22                | 1,121,104         | CC BY 4.0    |
+| IndicVoices-R   | 4,067.0        | 22                | 1,749,066         | CC BY 4.0    |
+| LIMMITS         | 568.0          | 7                 | 246,008           | CC BY 4.0    |
+| Rasa            | 288.0          | 9                 | 155,734           | CC BY 4.0    |
+- **Languages Covered**:
+  The dataset supports **22 official languages** of India, along with English and Chhattisgarhi, making it comprehensive for regional language technologies. These languages include Assamese, Bengali, Bodo, Chhattisgarhi, Dogri, English, Gujarati, Hindi, Kannada, Kashmiri, Konkani, Maithili, Malayalam, Manipuri, Marathi, Nepali, Odia, Punjabi, Sanskrit, Santali, Sindhi, Tamil, Telugu, and Urdu.
+- **Language-Wise Data Breakdown**:
+| Language        | Duration (hrs) | No. of Utterances |
+|:---------------:|:--------------:|:-----------------:|
+| Assamese        | 563.87         | 256,102           |
+| Bengali         | 561.18         | 234,663           |
+| Bodo            | 637.79         | 320,584           |
+| Chhattisgarhi   | 80.11          | 38,148            |
+| Dogri           | 263.22         | 109,348           |
+| English         | 765.56         | 711,196           |
+| Gujarati        | 31.68          | 11,845            |
+| Hindi           | 396.75         | 162,343           |
+| Kannada         | 364.26         | 154,994           |
+| Kashmiri        | 216.17         | 93,343            |
+| Konkani         | 205.16         | 91,804            |
+| Maithili        | 473.07         | 197,886           |
+| Malayalam       | 394.33         | 168,281           |
+| Manipuri        | 112.47         | 57,068            |
+| Marathi         | 333.32         | 142,925           |
+| Nepali          | 542.69         | 244,007           |
+| Odia            | 264.16         | 105,469           |
+| Punjabi         | 280.48         | 109,795           |
+| Sanskrit        | 143.12         | 63,908            |
+| Santali         | 298.19         | 148,184           |
+| Sindhi          | 66.62          | 27,578            |
+| Tamil           | 561.92         | 236,293           |
+| Telugu          | 560.76         | 213,858           |
+| Urdu            | 268.46         | 112,932           |
+## Citation
+If you found this repository useful, please consider citing this work and also the original Stability AI paper:
+```
+@misc{lacombe-etal-2024-parler-tts,
+  author = {Yoach Lacombe and Vaibhav Srivastav and Sanchit Gandhi},
+  title = {Parler-TTS},
+  year = {2024},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/huggingface/parler-tts}}
+}
+```
+```
+@misc{lyth2024natural,
+      title={Natural language guidance of high-fidelity text-to-speech with synthetic annotations},
+      author={Dan Lyth and Simon King},
+      year={2024},
+      eprint={2402.01912},
+      archivePrefix={arXiv},
+      primaryClass={cs.SD}
+}
+```
+## License
+This model is permissively licensed under the Apache 2.0 license.