Spaces:
Running
on
Zero
Running
on
Zero
| <h1 align="center"> | |
| <a href="https://applio.org" target="_blank"><img src="https://github.com/IAHispano/Applio/assets/133521603/78e975d8-b07f-47ba-ab23-5a31592f322a" alt="Applio"></a> | |
| </h1> | |
| <p align="center"> | |
| <img alt="Contributors" src="https://img.shields.io/github/contributors/iahispano/applio?style=for-the-badge&color=FFFFFF" /> | |
| <img alt="Release" src="https://img.shields.io/github/release/iahispano/applio?style=for-the-badge&color=FFFFFF" /> | |
| <img alt="Stars" src="https://img.shields.io/github/stars/iahispano/applio?style=for-the-badge&color=FFFFFF" /> | |
| <img alt="Fork" src="https://img.shields.io/github/forks/iahispano/applio?style=for-the-badge&color=FFFFFF" /> | |
| <img alt="Issues" src="https://img.shields.io/github/issues/iahispano/applio?style=for-the-badge&color=FFFFFF" /> | |
| </p> | |
| <p align="center">VITS-based Voice Conversion focused on simplicity, quality, and performance.</p> | |
| <p align="center"> | |
| <a href="https://applio.org" target="_blank">🌐 Website</a> | |
| • | |
| <a href="https://docs.applio.org" target="_blank">📚 Documentation</a> | |
| • | |
| <a href="https://discord.gg/iahispano" target="_blank">☎️ Discord</a> | |
| </p> | |
| <p align="center"> | |
| <a href="https://github.com/IAHispano/Applio-Plugins" target="_blank">🛒 Plugins</a> | |
| • | |
| <a href="https://huggingface.co./IAHispano/Applio/tree/main/Compiled" target="_blank">📦 Compiled</a> | |
| • | |
| <a href="https://applio.org/playground" target="_blank">🎮 Playground</a> | |
| • | |
| <a href="https://colab.research.google.com/github/iahispano/applio/blob/master/assets/Applio.ipynb" target="_blank">🔎 Google Colab (UI)</a> | |
| • | |
| <a href="https://colab.research.google.com/github/iahispano/applio/blob/master/assets/Applio_NoUI.ipynb" target="_blank">🔎 Google Colab (No UI)</a> | |
| </p> | |
| ## Table of Contents | |
| - [Installation](#installation) | |
| - [Windows](#windows) | |
| - [macOS](#macos) | |
| - [Linux](#linux) | |
| - [Makefile](#makefile) | |
| - [Usage](#usage) | |
| - [Windows](#windows-1) | |
| - [macOS](#macos-1) | |
| - [Linux](#linux-1) | |
| - [Makefile](#makefile-1) | |
| - [Technical Information](#technical-information) | |
| - [Repository Enhancements](#repository-enhancements) | |
| - [Commercial Usage](#commercial-usage) | |
| - [References](#references) | |
| - [Contributors](#contributors) | |
| ## Installation | |
| Download the latest version from [GitHub Releases](https://github.com/IAHispano/Applio-RVC-Fork/releases) or use the [Compiled Versions](https://huggingface.co./IAHispano/Applio/tree/main/Compiled). | |
| ### Windows | |
| ```bash | |
| ./run-install.bat | |
| ``` | |
| ### macOS | |
| For macOS, you need to install the requirements in a Python environment version 3.9 to 3.11. Here are the steps: | |
| ```bash | |
| python3 -m venv .venv | |
| source .venv/bin/activate | |
| chmod +x run-install.sh | |
| ./run-install.sh | |
| ``` | |
| ### Linux | |
| Certain Linux-based operating systems may encounter complications with the installer. In such instances, we suggest installing the `requirements.txt` within a Python environment version 3.9 to 3.11. | |
| ```bash | |
| chmod +x run-install.sh | |
| ./run-install.sh | |
| ``` | |
| ### Makefile | |
| For platforms such as [Paperspace](https://www.paperspace.com/): | |
| ```bash | |
| make run-install | |
| ``` | |
| ## Usage | |
| Visit [Applio Documentation](https://docs.applio.org/) for a detailed UI usage explanation. | |
| ### Windows | |
| ```bash | |
| ./run-applio.bat | |
| ``` | |
| ### macOS | |
| ```bash | |
| chmod +x run-applio.sh | |
| ./run-applio.sh | |
| ``` | |
| ### Linux | |
| ```bash | |
| chmod +x run-applio.sh | |
| ./run-applio.sh | |
| ``` | |
| ### Makefile | |
| For platforms such as [Paperspace](https://www.paperspace.com/): | |
| ```bash | |
| make run-applio | |
| ``` | |
| ## Technical Information | |
| Applio uses an enhanced version of the Retrieval-based Voice Conversion (RVC) model, a powerful technique for transforming the voice of an audio signal to sound like another person. This advanced implementation of RVC in Applio enables high-quality voice conversion while maintaining simplicity and performance. | |
| ### 0. Pre-Learning: Key Concepts in Speech Processing and Voice Conversion | |
| This section introduces fundamental concepts in speech processing and voice conversion, paving the way for a deeper understanding of the RVC pipeline: | |
| #### 1. Speech Representation | |
| - **Phoneme:** The smallest unit of sound in a language that distinguishes one word from another. Examples: /k/, /æ/, /t/. | |
| - **Spectrogram:** A visual representation of the frequency content of a sound over time, showing how the intensity of different frequencies changes over the duration of the audio. | |
| - **Mel-Spectrogram:** A type of spectrogram that mimics human auditory perception, emphasizing frequencies that are more important to human hearing. | |
| - **Speaker Embedding:** A vector representation that captures the unique acoustic characteristics of a speaker's voice, encoding information about pitch, tone, timbre, and other vocal qualities. | |
| #### 2. Text-to-Speech (TTS) | |
| - **TTS Model:** A machine learning model that generates artificial speech from written text. | |
| - **Encoder-Decoder Architecture:** A common architecture in TTS models, where an encoder processes the text and pitch information to create a latent representation, and a decoder uses this representation to synthesize the audio signal. | |
| - **Transformer Architecture:** A powerful neural network architecture particularly well-suited for sequence modeling, allowing the model to handle long sequences of text or audio and capture relationships between elements. | |
| #### 3. Voice Conversion | |
| - **Voice Conversion (VC):** The process of transforming the voice of a speaker in an audio signal to sound like another speaker. | |
| - **Speaker Adaptation:** The process of adapting a TTS model to a specific speaker, often by training on a small dataset of the speaker's voice. | |
| - **Retrieval-Based VC (RVC):** A voice conversion approach where speaker embeddings are retrieved from a database and used to guide the TTS model in synthesizing audio with the target speaker's voice. | |
| #### 4. Additional Concepts | |
| - **ContentVec:** A powerful self-supervised learning model for speech representation, excelling at capturing speaker-specific information. | |
| - **FAISS:** A library for efficient similarity search, used to retrieve speaker embeddings that are similar to the extracted ContentVec embedding. | |
| - **Neural Source Filter (NSF):** A module that models audio generation as a filtering process, allowing the model to produce high-quality and realistic audio signals by learning complex relationships between the source signal and the output waveform. | |
| #### 5. Why are these concepts important? | |
| Understanding these concepts is essential for appreciating the mechanics and capabilities of the RVC pipeline: | |
| - **Speech Representation:** Different representations capture different aspects of speech, allowing for effective analysis and manipulation. | |
| - **TTS Models:** The TTS model forms the foundation of RVC, providing the ability to synthesize audio from text and pitch. | |
| - **Voice Conversion:** Voice conversion aims to transfer a speaker's identity to a different audio signal. | |
| - **ContentVec and Speaker Embeddings:** ContentVec provides a powerful way to extract speaker-specific information, which is crucial for accurate voice conversion. | |
| - **FAISS:** This library enables efficient speaker embedding retrieval, facilitating the selection of appropriate target voices. | |
| - **NSF:** The NSF is a critical component of the TTS model, contributing to the generation of realistic and high-quality audio. | |
| ### 1. Model Architecture | |
| The RVC model comprises two main components: | |
| #### A. Encoder-Decoder Network | |
| This network synthesizes audio based on text and pitch information while incorporating speaker characteristics from the ContentVec embedding. | |
| **Encoder:** | |
| - **Input:** Phoneme sequences (text representation) and pitch information (optional). | |
| - **Embeddings:** | |
| - Phonemes are represented as vectors using linear layers, creating a dense representation of the text input. | |
| - Pitch is usually converted to a one-hot encoding or a continuous value and embedded similarly. | |
| - **Transformer Encoder:** Processes the embedded features in a highly parallel manner. | |
| It employs: | |
| - **Self-Attention:** Allows the encoder to attend to different parts of the input sequence to understand the relationships between words and their context. | |
| - **Feedforward Networks (FFN):** Apply non-linear transformations to further refine the features captured by self-attention. | |
| - **Layer Normalization:** Stabilizes training and improves performance by normalizing the outputs of each layer. | |
| - **Dropout:** A regularization technique to prevent overfitting. | |
| - **Output:** Produces a latent representation of the input text and pitch, capturing their relationships and serving as the input for the decoder. | |
| **Decoder:** | |
| - **Input:** The latent representation from the encoder. | |
| - **Transformer Decoder:** Receives the encoder output and utilizes: | |
| - **Self-Attention:** Allows the decoder to attend to different parts of the generated sequence to maintain consistency and coherence in the output audio. | |
| - **Encoder-Decoder Attention:** Enables the decoder to incorporate information from the input text and pitch into the audio generation process. | |
| - **Neural Source Filter (NSF):** A powerful component for generating audio, modeling the generation process as a filter applied to a source signal. It uses: | |
| - **Upsampling:** Increases the resolution of the latent representation to match the desired length of the audio signal. | |
| - **Residual Blocks:** Learn complex and non-linear relationships between input features and the output audio, contributing to realistic and detailed waveforms. | |
| - **Source Module:** Generates the excitation signal (often harmonic) that drives the NSF. It combines sine waves (for voiced sounds) and noise (for unvoiced sounds) to create a natural source signal. | |
| - **Noise Convolution:** Convolves noise with the harmonic signal to introduce additional variation and realism. | |
| - **Final Convolutional Layer:** Converts the filtered output to a single-channel audio waveform. | |
| - **Output:** Synthesized audio signal. | |
| #### B. ContentVec Speaker Embedding Extractor | |
| Extracts speaker-specific information from the input audio. | |
| - **Input:** The preprocessed audio signal. | |
| - **Processing:** The ContentVec model, trained on a massive dataset of speech data, processes the input audio and extracts a speaker embedding vector, capturing the unique acoustic properties of the speaker's voice. | |
| - **Output:** A speaker embedding vector representing the voice of the speaker. | |
| ### 2. Training Stage | |
| The RVC model is trained using a combination of two key losses: | |
| - **Generative Loss:** | |
| - **Mel-Spectrogram:** The Mel-spectrogram is computed for both the target audio and the generated audio. | |
| - **L1 Loss:** Measures the absolute difference between the Mel-spectrograms of the target and generated audio, encouraging the decoder to produce audio with a similar spectral profile. | |
| - **Discriminative Loss:** | |
| - **Multi-Period Discriminator:** Tries to distinguish between real and generated audio at different time scales, using convolution layers to capture long-term dependencies in the audio. | |
| - **Adversarial Training:** The generator tries to fool the discriminator by producing audio that sounds real, while the discriminator is trained to correctly identify generated audio. | |
| - **Optional KL Divergence Loss:** Measures the difference between the distributions of latent variables generated by the encoder and a posterior encoder (which infers the latent representation from the target audio). Encourages the model to learn a more efficient and stable latent representation. | |
| ### 3. Inference Stage | |
| The inference stage utilizes the trained model to convert the voice of an audio input to sound like a target speaker. Here's a breakdown: | |
| **Input:** | |
| - Phoneme sequences (text representation). | |
| - Pitch information (optional). | |
| - Target speaker ID (identifies the desired voice). | |
| **Steps:** | |
| - **ContentVec Embedding Extraction:** | |
| - The ContentVec model processes the input audio and extracts a speaker embedding vector, capturing the voice characteristics of the speaker. | |
| - **Optional Embedding Retrieval:** | |
| - **FAISS Index:** Used to efficiently search for speaker embeddings similar to the extracted ContentVec embedding. It helps guide the voice conversion process toward a specific speaker when multiple speakers are available. | |
| - **Embedding Retrieval:** The FAISS index is queried using the extracted ContentVec embedding, and similar embeddings are retrieved. | |
| - **Embedding Manipulation:** | |
| - **Blending:** The extracted ContentVec embedding can be blended with retrieved embeddings using the index_rate parameter, allowing control over how much the target speaker's voice influences the conversion. | |
| - **Encoder-Decoder Processing:** | |
| - **Encoder:** Encodes the phoneme sequences and pitch into a latent representation, capturing the relationships between them. | |
| - **Decoder:** Synthesizes the audio signal, incorporating the speaker characteristics from the ContentVec embedding (potentially blended with retrieved embeddings). | |
| - **Post-Processing:** | |
| - **Resampling:** Adjusts the sampling rate of the generated audio if needed. | |
| - **RMS Adjustment:** Adjusts the volume (RMS) of the output audio to match the input audio. | |
| ### 4. Key Techniques | |
| - **Transformer Architecture:** The Transformer architecture is a powerful tool for sequence modeling, enabling the encoder and decoder to efficiently process long sequences and capture complex relationships within the data. | |
| - **Neural Source Filter (NSF):** Models audio generation as a filtering process, allowing the model to produce high-quality and realistic audio signals by learning complex relationships between the source signal and the output waveform. | |
| - **Flow-Based Generative Model:** Enables the model to learn complex probability distributions for the audio signal, leading to more realistic and diverse generated speech. | |
| - **Multi-period Discriminator:** Helps improve the quality and realism of the generated audio by evaluating the audio at | |
| different temporal scales and providing feedback to the generator. | |
| - **Relative Positional Encoding:** Helps the model understand the relative positions of elements within the input sequences, improving the model's ability to handle long sequences and maintain context. | |
| ### 5. Future Challenges | |
| Despite the advancements in Retrieval-Based Voice Conversion, several challenges and areas for future research remain: | |
| - **Speaker Generalization:** Improving the ability of models to generalize to unseen speakers with minimal data. | |
| - **Real-time Processing:** Enhancing the efficiency of models to support real-time voice conversion applications. | |
| - **Emotional Expression:** Better capturing and transferring emotional nuances in voice conversion. | |
| - **Noise Robustness:** Improving the robustness of voice conversion models to handle noisy and low-quality input audio. | |
| ## Repository Enhancements | |
| This repository has undergone significant enhancements to improve its functionality and maintainability: | |
| - **Modular Codebase:** Restructured codebase for better organization, readability, and maintenance. | |
| - **Hop Length Implementation:** Improved efficiency and performance, especially on Crepe (formerly Mangio-Crepe), thanks to [@Mangio621](https://github.com/Mangio621/Mangio-RVC-Fork). | |
| - **Translations in 30+ Languages:** Added support for over 30 languages. | |
| - **Cross-Platform Compatibility:** Ensured seamless operation across various platforms. | |
| - **Optimized Requirements:** Fine-tuned project requirements for enhanced performance. | |
| - **Streamlined Installation:** Simplified installation process for a user-friendly setup. | |
| - **Hybrid F0 Estimation:** Introduced a personalized 'hybrid' F0 estimation method utilizing nanmedian. | |
| - **Easy-to-Use UI:** Implemented an intuitive user interface. | |
| - **Plugin System:** Introduced a plugin system for extending functionality. | |
| - **Overtraining Detector:** Implemented a detector to prevent excessive training. | |
| - **Model Search:** Integrated model search feature for easy discovery. | |
| - **Pretrained Models:** Added support for custom pretrained models. | |
| - **Voice Blender:** Developed a feature to combine two trained models to create a new one. | |
| - **Accessibility Improvements:** Enhanced with descriptive tooltips for UI elements. | |
| - **New F0 Extraction Methods:** Introduced methods like FCPE or Hybrid for pitch extraction. | |
| - **Output Format Selection:** Added feature to choose audio file formats. | |
| - **Hashing System:** Assigned unique IDs to models to prevent unauthorized duplication. | |
| - **Model Download System:** Supported downloads from various platforms. | |
| - **TTS Enhancements:** Improved Text-to-Speech functionality. | |
| - **Split Audio:** Implemented audio splitting for faster processing. | |
| - **Discord Presence:** Displayed usage status on Discord. | |
| - **Flask Integration:** Enabled automatic model downloads via Flask. | |
| - **Support Tab:** Added a tab for screen recording to report issues. | |
| These enhancements contribute to a more robust and scalable codebase, making the repository more accessible for contributors and users alike. | |
| ## Commercial Usage | |
| For commercial purposes, please adhere to the guidelines outlined in the [MIT license](./LICENSE) governing this project. Prior to integrating Applio into your application, we kindly request that you contact us at [email protected] to ensure ethical use. | |
| Please note, the use of Applio-generated audio files falls under your own responsibility and must always respect applicable copyrights. We encourage you to consider supporting the continuous development and maintenance of Applio through a donation. | |
| Your cooperation and support are greatly appreciated. Thank you! | |
| ## References | |
| Applio is possible to these projects and those cited in their references. | |
| - [gradio-screen-recorder](https://huggingface.co./spaces/gstaff/gradio-screen-recorder) by gstaff | |
| - [rvc-cli](https://github.com/blaise-tk/rvc-cli) by blaisewf | |
| ### Contributors | |
| <a href="https://github.com/IAHispano/Applio/graphs/contributors" target="_blank"> | |
| <img src="https://contrib.rocks/image?repo=IAHispano/Applio" /> | |
| </a> | |