README.md

---
license: creativeml-openrail-m
---
---
license: cc-by-nc-nd-4.0
---

# AudioLDM 2

AudioLDM 2 is a latent text-to-audio diffusion model capable of generating realistic audio samples given any text input. 
It is available in the 🧨 Diffusers library from v0.21.0 onwards.

# Model Details

AudioLDM 2 was proposed in the paper [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://arxiv.org/abs/2308.05734) by Haohe Liu et al.

AudioLDM takes a text prompt as input and predicts the corresponding audio. It can generate text-conditional sound effects, 
human speech and music.

# Checkpoint Details

This is the original, **base** version of the AudioLDM 2 model, also referred to as **audioldm2-full**. 

There are three official AudioLDM 2 checkpoints. Two of these checkpoints are applicable to the general task of text-to-audio 
generation. The third checkpoint is trained exclusively on text-to-music generation. All checkpoints share the same 
model size for the text encoders and VAE. They differ in the size and depth of the UNet. See table below for details on 
the three official checkpoints:

| Checkpoint                                                      | Task          | UNet Model Size | Total Model Size | Training Data / h |
|-----------------------------------------------------------------|---------------|-----------------|------------------|-------------------|
| [audioldm2](https://huggingface.co./cvssp/audioldm2)             | Text-to-audio | 350M            | 1.1B             | 1150k             |
| [audioldm2-large](https://huggingface.co./cvssp/audioldm2-large) | Text-to-audio | 750M            | 1.5B             | 1150k             |
| [audioldm2-music](https://huggingface.co./cvssp/audioldm2-music) | Text-to-music | 350M            | 1.1B             | 665k              |
| [audioldm2-gigaspeech](https://huggingface.co./anhnct/audioldm2_gigaspeech) | Text-to-speech | 350M            | 1.1B             |10k              |
| [audioldm2-ljspeech](https://huggingface.co./anhnct/audioldm2_ljspeech) | Text-to-speech | 350M            | 1.1B             |              |

## Model Sources

- [**Original Repository**](https://github.com/haoheliu/audioldm2)
- [**🧨 Diffusers Pipeline**](https://huggingface.co./docs/diffusers/api/pipelines/audioldm2)
- [**Paper**](https://arxiv.org/abs/2308.05734)
- [**Demo**](https://huggingface.co./spaces/haoheliu/audioldm2-text2audio-text2music)

# Usage

First, install the required packages:

```
pip install --upgrade diffusers transformers accelerate
```

## Text-to-Speech

For text-to-speech generation, the [AudioLDM2Pipeline](https://huggingface.co./docs/diffusers/api/pipelines/audioldm2) can be 
used to load pre-trained weights and generate text-conditional audio outputs:

```python
import scipy
import torch
from diffusers import AudioLDM2Pipeline

repo_id = "anhnct/audioldm2_ljspeech"
pipe = AudioLDM2Pipeline.from_pretrained(repo_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

# define the prompts
prompt = "An female actor say with angry voice"
transcript = "wish you have a good day, i hope you never forget me"
negative_prompt = "low quality"

# set the seed for generator
generator = torch.Generator("cuda").manual_seed(1)

# run the generation
audio = pipe(
    prompt,
    negative_prompt=negative_prompt,
    transcription=transcript_1,
    num_inference_steps=200,
    audio_length_in_s=8.0,
    num_waveforms_per_prompt=1,
    generator=generator,
    max_new_tokens=512
).audios

# save the best audio sample (index 0) as a .wav file
scipy.io.wavfile.write("techno_2.wav", rate=16000, data=audio[0])
```

# Citation

**BibTeX:**
```
@article{liu2023audioldm2,
  title={"AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining"},
  author={Haohe Liu and Qiao Tian and Yi Yuan and Xubo Liu and Xinhao Mei and Qiuqiang Kong and Yuping Wang and Wenwu Wang and Yuxuan Wang and Mark D. Plumbley},
  journal={arXiv preprint arXiv:2308.05734},
  year={2023}
}
```