egs/metrics/README.md

# Amphion Evaluation Recipe

## Supported Evaluation Metrics

Until now, Amphion Evaluation has supported the following objective metrics:

- **F0 Modeling**:
  - F0 Pearson Coefficients (FPC)
  - F0 Periodicity Root Mean Square Error (PeriodicityRMSE)
  - F0 Root Mean Square Error (F0RMSE)
  - Voiced/Unvoiced F1 Score (V/UV F1)
- **Energy Modeling**:
  - Energy Root Mean Square Error (EnergyRMSE)
  - Energy Pearson Coefficients (EnergyPC)
- **Intelligibility**:
  - Character Error Rate (CER) based on [Whipser](https://github.com/openai/whisper)
  - Word Error Rate (WER) based on [Whipser](https://github.com/openai/whisper)
- **Spectrogram Distortion**:
  - Frechet Audio Distance (FAD)
  - Mel Cepstral Distortion (MCD)
  - Multi-Resolution STFT Distance (MSTFT)
  - Perceptual Evaluation of Speech Quality (PESQ)
  - Short Time Objective Intelligibility (STOI)
  - Scale Invariant Signal to Distortion Ratio (SISDR)
  - Scale Invariant Signal to Noise Ratio (SISNR)
- **Speaker Similarity**:
  - Cosine similarity based on:
    - [Rawnet3](https://github.com/Jungjee/RawNet)
    - [Resemblyzer](https://github.com/resemble-ai/Resemblyzer)
    - [WavLM](https://huggingface.co./microsoft/wavlm-base-plus-sv)

We provide a recipe to demonstrate how to objectively evaluate your generated audios. There are three steps in total:

1. Pretrained Models Preparation
2. Audio Data Preparation
3. Evaluation

## 1. Pretrained Models Preparation

If you want to calculate `RawNet3` based speaker similarity, you need to download the pretrained model first, as illustrated [here](../../pretrained/README.md).

## 2. Audio Data Preparation

Prepare reference audios and generated audios in two folders, the `ref_dir` contains the reference audio and the `gen_dir` contains the generated audio. Here is an example.

```plaintext
 ┣ {ref_dir}
 ┃ ┣ sample1.wav
 ┃ ┣ sample2.wav
 ┣ {gen_dir}
 ┃ ┣ sample1.wav
 ┃ ┣ sample2.wav
```

You have to make sure that the pairwise **reference audio and generated audio are named the same**, as illustrated above (sample1 to sample1, sample2 to sample2).

## 3. Evaluation

Run the `run.sh` with specified refenrece folder, generated folder, dump folder and metrics.

```bash
cd Amphion
sh egs/metrics/run.sh \
	--reference_folder [Your path to the reference audios] \
	--generated_folder [Your path to the generated audios] \
	--dump_folder [Your path to dump the objective results] \
	--metrics [The metrics you need] \
	--fs [Optional. To calculate all metrics in the specified sampling rate] \
	--similarity_model [Optional. To choose the model for calculating the speaker similarity. Currently "rawnet", "wavlm" and "resemblyzer" are available. Default to "wavlm"] \
	--similarity_mode [Optional. To choose the mode for calculating the speaker similarity. "pairwith" for calculating a series of ground truth / prediction audio pairs to obtain the speaker similarity, and "overall" for computing the average score with all possible pairs between the refernece folder and generated folder. Default to "pairwith"] \
	--intelligibility_mode [Optionoal. To choose the mode for computing CER and WER. "gt_audio" means selecting the recognition content of the reference audio as the target, "gt_content" means using transcription as the target. Default to "gt_audio"] \
	--ltr_path [Optional. Path to the transcription file] \
	--language [Optional. Language for computing CER and WER. Default to "english"]
```

As for the metrics, an example is provided below:

```bash
--metrics "mcd pesq fad"
```

All currently available metrics keywords are listed below:

| Keys                      | Description                                |
| ------------------------- | ------------------------------------------ |
| `fpc`                     | F0 Pearson Coefficients                    |
| `f0_periodicity_rmse`     | F0 Periodicity Root Mean Square Error      |
| `f0rmse`                  | F0 Root Mean Square Error                  |
| `v_uv_f1`                 | Voiced/Unvoiced F1 Score                   |
| `energy_rmse`             | Energy Root Mean Square Error              |
| `energy_pc`               | Energy Pearson Coefficients                |
| `cer`                     | Character Error Rate                       |
| `wer`                     | Word Error Rate                            |
| `similarity`      | Speaker Similarity
| `fad`                     | Frechet Audio Distance                     |
| `mcd`                     | Mel Cepstral Distortion                    |
| `mstft`                   | Multi-Resolution STFT Distance             |
| `pesq`                    | Perceptual Evaluation of Speech Quality    |
| `si_sdr`                  | Scale Invariant Signal to Distortion Ratio |
| `si_snr`                  | Scale Invariant Signal to Noise Ratio      |
| `stoi`                    | Short Time Objective Intelligibility       |

For example, if want to calculate the speaker similarity between the synthesized audio and the reference audio with the same content, run:

```bash
sh egs/metrics/run.sh \
	--reference_folder [Your path to the reference audios] \
	--generated_folder [Your path to the generated audios] \
	--dump_folder [Your path to dump the objective results] \
	--metrics "similarity" \
	--similarity_model [Optional. To choose the model for calculating the speaker similarity. Currently "rawnet", "wavlm" and "resemblyzer" are available. Default to "wavlm"] \
	--similarity_mode "pairwith" \
```

If you don't have the reference audio with the same content, run the following to get the conteng-free similarity score:

```bash
sh egs/metrics/run.sh \
	--reference_folder [Your path to the reference audios] \
	--generated_folder [Your path to the generated audios] \
	--dump_folder [Your path to dump the objective results] \
	--metrics "similarity" \
	--similarity_model [Optional. To choose the model for calculating the speaker similarity. Currently "rawnet", "wavlm" and "resemblyzer" are available. Default to "wavlm"] \
	--similarity_mode "overall" \
```

## Troubleshooting
### FAD (Using Offline Models)
If your system is unable to access huggingface.co from the terminal, you might run into an error like "OSError: Can't load tokenizer for ...". To work around this, follow these steps to use local models:

1. Download the [bert-base-uncased](https://huggingface.co./bert-base-uncased), [roberta-base](https://huggingface.co./roberta-base), and [facebook/bart-base](https://huggingface.co./facebook/bart-base) models from `huggingface.co`. Ensure that the models are complete and uncorrupted. Place these directories within `Amphion/pretrained`. For a detailed file structure reference, see [This README](../../pretrained/README.md#optional-model-dependencies-for-evaluation) under `Amphion/pretrained`.
2. Inside the `Amphion/pretrained` directory, create a bash script with the content outlined below. This script will automatically update the tokenizer paths used by your system:
  ```bash
  #!/bin/bash

  BERT_DIR="bert-base-uncased"
  ROBERTA_DIR="roberta-base"
  BART_DIR="facebook/bart-base"
  PYTHON_SCRIPT="[YOUR ENV PATH]/lib/python3.9/site-packages/laion_clap/training/data.py"

  update_tokenizer_path() {
      local dir_name=$1
      local tokenizer_variable=$2
      local full_path

      if [ -d "$dir_name" ]; then
          full_path=$(realpath "$dir_name")
          if [ -f "$PYTHON_SCRIPT" ]; then
              sed -i "s|${tokenizer_variable}.from_pretrained(\".*\")|${tokenizer_variable}.from_pretrained(\"$full_path\")|" "$PYTHON_SCRIPT"
              echo "Updated ${tokenizer_variable} path to $full_path."
          else
              echo "Error: The specified Python script does not exist."
              exit 1
          fi
      else
          echo "Error: The directory $dir_name does not exist in the current directory."
          exit 1
      fi
  }

  update_tokenizer_path "$BERT_DIR" "BertTokenizer"
  update_tokenizer_path "$ROBERTA_DIR" "RobertaTokenizer"
  update_tokenizer_path "$BART_DIR" "BartTokenizer"

  echo "BERT, BART and RoBERTa Python script paths have been updated."

  ```

3. The script provided is intended to adjust the tokenizer paths in the `data.py` file, found under `/lib/python3.9/site-packages/laion_clap/training/`, within your specific environment. For those utilizing conda, you can determine your environment path by running `conda info --envs`. Then, substitute `[YOUR ENV PATH]` in the script with this path. If your environment is configured differently, you'll need to update the `PYTHON_SCRIPT` variable to correctly point to the `data.py` file.
4. Run the script. If it executes successfully, the tokenizer paths will be updated, allowing them to be loaded locally.

### WavLM-based Speaker Similarity (Using Offline Models)

If your system is unable to access huggingface.co from the terminal and you want to calculate `WavLM` based speaker similarity, you need to download the pretrained model first, as illustrated [here](../../pretrained/README.md).