File size: 5,387 Bytes
4aae1b5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 |
---
license: apache-2.0
tags:
- text-to-audio
- music
library_name: transformers
---
# Lumina Text-to-Music
We will provide our implementation and pretrained models as open source in this repository recently.
- Generation Model: Flag-DiT
- Text Encoder: [FLAN-T5-Large](https://huggingface.co./google/flan-t5-large)
- VAE: Make an Audio 2, finetuned from [Makee an Audio](https://github.com/Text-to-Audio/Make-An-Audio)
- Decoder: [Vocoder](https://github.com/NVIDIA/BigVGAN)
## 📰 News
- [2024-06-07] 🚀🚀🚀 We release the initial version of `Lumina-T2Music` for text-to-music generation.
## Installation
Before installation, ensure that you have a working ``nvcc``
```bash
# The command should work and show the same version number as in our case. (12.1 in our case).
nvcc --version
```
On some outdated distros (e.g., CentOS 7), you may also want to check that a late enough version of
``gcc`` is available
```bash
# The command should work and show a version of at least 6.0.
# If not, consult distro-specific tutorials to obtain a newer version or build manually.
gcc --version
```
Downloading Lumina-T2X repo from github:
```bash
git clone https://github.com/Alpha-VLLM/Lumina-T2X
```
### 1. Create a conda environment and install PyTorch
Note: You may want to adjust the CUDA version [according to your driver version](https://docs.nvidia.com/deploy/cuda-compatibility/#default-to-minor-version).
```bash
conda create -n Lumina_T2X -y
conda activate Lumina_T2X
conda install python=3.11 pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y
```
### 2. Install dependencies
>[!Warning]
> The environment dependencies for Lumina-T2Music are different from those for Lumina-T2I. Please install the appropriate environment.
Installing `Lumina-T2Music` dependencies:
```bash
cd .. # If you are in the `lumina_music` directory, execute this line.
pip install -e ".[music]"
```
or you can use `requirements.txt` to install the environment.
```bash
cd lumina_music # If you are not in the `lumina_music` folder, run this line.
pip install -r requirements.txt
```
### 3. Install ``flash-attn``
```bash
pip install flash-attn --no-build-isolation
```
### 4. Install [nvidia apex](https://github.com/nvidia/apex) (optional)
>[!Warning]
> While Apex can improve efficiency, it is *not* a must to make Lumina-T2X work.
>
> Note that Lumina-T2X works smoothly with either:
> + Apex not installed at all; OR
> + Apex successfully installed with CUDA and C++ extensions.
>
> However, it will fail when:
> + A Python-only build of Apex is installed.
>
> If the error `No module named 'fused_layer_norm_cuda'` appears, it typically means you are using a Python-only build of Apex. To resolve this, please run `pip uninstall apex`, and Lumina-T2X should then function correctly.
You can clone the repo and install following the official guidelines (note that we expect a full
build, i.e., with CUDA and C++ extensions)
```bash
pip install ninja
git clone https://github.com/NVIDIA/apex
cd apex
# if pip >= 23.1 (ref: https://pip.pypa.io/en/stable/news/#v23-1) which supports multiple `--config-settings` with the same key...
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
# otherwise
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./
```
## Inference
### Preparation
Prepare the pretrained checkpoints.
⭐⭐ (Recommended) you can use `huggingface-cli` downloading our model:
```bash
huggingface-cli download --resume-download Alpha-VLLM/Lumina-T2Music --local-dir /path/to/ckpt
```
or using git for cloning the model you want to use:
```bash
git clone https://huggingface.co./Alpha-VLLM/Lumina-T2Music
```
### Web Demo
To host a local gradio demo for interactive inference, run the following command:
1. updated `AutoencoderKL` ckpt path
you should update `configs/lumina-text2music.yaml` to set `AutoencoderKL` checkpoint path. Please replace `/path/to/ckpt` with the path where your checkpoints are located (<real_path>).
```diff
...
depth: 16
max_len: 1000
first_stage_config:
target: models.autoencoder1d.AutoencoderKL
params:
embed_dim: 20
monitor: val/rec_loss
- ckpt_path: /path/to/ckpt/maa2/maa2.ckpt
+ ckpt_path: <real_path>/maa2/maa2.ckpt
ddconfig:
double_z: true
in_channels: 80
out_ch: 80
...
```
2. setting `Lumina-T2Music` and `Vocoder` checkpoint path and run demo
Please replace `/path/to/ckpt` with the actual downloaded path.
```bash
# `/path/to/ckpt` should be a directory containing `music_generation`, `maa2`, and `bigvnat`.
# default
python -u demo_music.py \
--ckpt "/path/to/ckpt/music_generation" \
--vocoder_ckpt "/path/to/ckpt/bigvnat" \
--config_path "configs/lumina-text2music.yaml" \
--sample_rate 16000
```
## Disclaimer
Any organization or individual is prohibited from using any technology mentioned in this paper to generate someone's speech without his/her consent, including but not limited to government leaders, political figures, and celebrities. If you do not comply with this item, you could be in violation of copyright laws. |