README.md

---
base_model:
- openai/whisper-large-v2
datasets:
- MERaLiON/MNSC
library_name: transformers
license: other
license_name: meralion-public-license
license_link: https://huggingface.co./MERaLiON/MERaLiON-AudioLLM-Whisper-SEA-LION/blob/main/MERaLiON-Public-Licence-v1.pdf
metrics:
- bleu
- wer
pipeline_tag: automatic-speech-recognition
tags:
- vllm
- LLM-as-a-Judge
- chat
- audio
- safetensors
widget:
- example_title: Sentence-level ASR
  src: librispeech_clean.wav
  output:
    text: "USER Instruction: Please transcribe this speech.\n MODEL Output: <Speaker1>\
      \ When they were outside, Ung simply latched the door and started up the path."
- example_title: Dialogue-level ASR
  src: IMDA_conversation.wav
  output:
    text: "USER Instruction: Please turn this speech into written format.\n MODEL\
      \ Output: <Speaker1> Okay, (um) in that case, (uh) how do I apply? <Speaker2>\
      \ Alright, you can just (um) apply it online and then (um) we'll need some documents\
      \ from you. (um) let's say the bank statement with your address and your name,\
      \ and also the (um) Nsf card that you have to enjoy the promotion. <Speaker1>\
      \ (mmhmm) (mmhmm) (mmhmm) [ah] I see the green one, right?"
- example_title: Spoken Dialogue Summarization
  src: IMDA_conversation.wav
  output:
    text: "USER Instruction: Please briefly summarize this conversation.\n MODEL Output:\
      \ Speaker1 and Speaker2 discussed the application process for a promotion. Speaker2\
      \ explained that the application can be done online and provided a list of required\
      \ documents, including a bank statement with the applicant's address and name,\
      \ and an NSF card. Speaker1 acknowledged the information and confirmed the details\
      \ of the required documents."
---

# MERaLiON

MERaLiON-AudioLLM is a Speech-Text Large Language Model tailored for Singapore’s multilingual and multicultural landscape. Integrating a localised [Whisper-large-v2](https://huggingface.co./openai/whisper-large-v2) speech encoder and [SEA-LION V3](https://huggingface.co./aisingapore/gemma2-9b-cpt-sea-lionv3-instruct) text decoder, MERaLiON-AudioLLM is finetuned on **260,000 hours of speech and audio data**, **6 various tasks**, to address the diverse linguistic nuances of Singapore's local accents and dialects.

MERaLiON stands for **M**ultimodal **E**mpathetic **R**easoning **a**nd **L**earning **i**n **O**ne **N**etwork.

- **Developed by:** I<sup>2</sup>R, A\*STAR, with collaboration with AISG, Singapore
- **Model type:** Multimodal LLM
- **Language(s):** Primarily English (Global and Singapore), with support for input and output in other languages compatible with Whisper and SEA-LION models.
- **License:** [MERaLiON Public License](https://huggingface.co./MERaLiON/MERaLiON-AudioLLM-Whisper-SEA-LION/blob/main/MERaLiON-Public-Licence-v1.pdf)
- **Demo:** [MERaLiON-AudioLLM Web Demo](https://huggingface.co./spaces/MERaLiON/MERaLiON-AudioLLM)

We support model inference using the [Huggingface](#inference) and [vLLM](vllm_plugin_meralion/README.md) frameworks. For more technical details, please refer to our [technical report](https://arxiv.org/abs/2412.09818).

## Acknowledgement
This research is supported by the National Research Foundation, Singapore and Infocomm Media Development Authority, Singapore under its National Large Language Models Funding Initiative.

## Model Description

MERaLiON-AudioLLM is designed to take in an **audio-text pair** as input and generate a **text output**.

The architecture comprises three key components: an **audio encoder** that transforms speech or audio inputs into sequences of vector representations, a **text decoder** that interprets and responds to natural language instructions, and an **adaptor module** that compresses the encoder representations while aligning the encoder’s hidden dimension with the text decoder’s embedding size.

Specifically, we fine-tuned the **MERaLiON-Whisper** encoder from Whisper-large-v2 for the audio encoder and used SEA-LION V3, a localised LLM developed by our partner AI Singapore as the text decoder.

<img src="model_architecture.png" alt="model_architecture" width="400" style="margin-left:'auto' margin-right:'auto' display:'block'"/>

## Capabilities

MERaLiON-AudioLLM is trained to mainly address 6 tasks, namely `Automatic Speech Recognition` (ASR), 
`Speech Translation` (ST), `Spoken Question Answering` (SQA), 
`Spoken Dialogue Summarization` (SDS), `Speech Instruction` (SI), and `Paralinguistics` (PARA). 

We benchmark MERaLiON-AudioLLM with a series of test sets from the [AudioBench benchmark](https://github.com/AudioLLMs/AudioBench) 
against three well-known AudioLLMs: `Qwen2-Audio 7B`, `WavLLM`, `SALMONN`, and a cascaded model. 
As is shown in the following table, MERaLiON-AudioLLM performs better in the Singapore local context, 
as evidenced by evaluation results on Singapore's [Multitask National Speech Corpus](https://huggingface.co./datasets/MERaLiON/Multitask-National-Speech-Corpus-v1) (MNSC) datasets. 

> [!NOTE]
> MNSC is a multitask speech understanding dataset derived and further annotated from [IMDA NSC Corpus](https://www.imda.gov.sg/how-we-can-help/national-speech-corpus).
> It focuses on the knowledge of Singapore's local accent, localised terms, and code-switching.

We assess ASR and ST tasks using Word Error Rate (WER) and BLEU scores, respectively. For other tasks, we employ the LLM-as-a-Judge framework,
which uses a pre-trained large language model to evaluate task performance by generating and scoring responses based on relevance, coherence, and accuracy criteria.
Refer to the [AudioBench paper](https://arxiv.org/abs/2406.16020) for more details.

<div class="table*">
<table>
<thead>
<tr>
  <th style="text-align: center;"><strong>Task</strong></th>
  <th style="text-align: center;"><strong>Dataset</strong></th>
  <th style="text-align: center;"><strong>MERaLiON</strong></th>
  <th style="text-align: center;"><strong>Qwen2-Audio 7B</strong></th>
  <th style="text-align: center;"><strong>WavLLM</strong></th>
  <th style="text-align: center;"><strong>SALMONN-7B</strong></th>
  <th style="text-align: center;"><strong>Cascaded Model</strong></th>
</tr>
</thead>
<tbody>
<tr>
  <td style="text-align: center;" rowspan="11"><strong>Automatic Speech Recognition</strong><br>WER (<span
  class="math inline">↓</span>)</td>
  <td style="text-align: center;">LibriSpeech-Test-Clean</td>
  <td style="text-align: center;">0.03</td>
  <td style="text-align: center;">0.03</td>
  <td style="text-align: center;"><strong><u>0.02</u></strong></td>
  <td style="text-align: center;">0.10</td>
  <td style="text-align: center;">0.03</td>
</tr>
<tr>
  <td style="text-align: center;">LibriSpeech-Test-Other</td>
  <td style="text-align: center;"><strong><u>0.05</u></strong></td>
  <td style="text-align: center;">0.06</td>
  <td style="text-align: center;"><strong><u>0.05</u></strong></td>
  <td style="text-align: center;">0.10</td>
  <td style="text-align: center;"><u>0.05</u></td>
</tr>
<tr>
  <td style="text-align: center;">Common-Voice-15-En-Test</td>
  <td style="text-align: center;"><strong><u>0.10</u></strong></td>
  <td style="text-align: center;">0.11</td>
  <td style="text-align: center;">0.15</td>
  <td style="text-align: center;">0.31</td>
  <td style="text-align: center;">0.11</td>
</tr>
<tr>
  <td style="text-align: center;">Earnings21-Test</td>
  <td style="text-align: center;"><strong>0.17</strong></td>
  <td style="text-align: center;">0.19</td>
  <td style="text-align: center;">0.65</td>
  <td style="text-align: center;">0.26</td>
  <td style="text-align: center;"><u>0.11</u></td>
</tr>
<tr>
  <td style="text-align: center;">Earnings22-Test</td>
  <td style="text-align: center;"><strong>0.20</strong></td>
  <td style="text-align: center;">0.24</td>
  <td style="text-align: center;">0.67</td>
  <td style="text-align: center;">0.36</td>
  <td style="text-align: center;"><u>0.14</u></td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-ASR-Part 1</td>
  <td style="text-align: center;"><u><strong>0.05</strong></u></td>
  <td style="text-align: center;">0.07</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">0.09</td>
  <td style="text-align: center;">0.07</td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-ASR-Part 2</td>
  <td style="text-align: center;"><u><strong>0.05</strong></u></td>
  <td style="text-align: center;">0.19</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">0.42</td>
  <td style="text-align: center;">0.33</td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-ASR-Part 3</td>
  <td style="text-align: center;"><u><strong>0.28</strong></u></td>
  <td style="text-align: center;">0.35</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">0.66</td>
  <td style="text-align: center;">0.30</td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-ASR-Part 4</td>
  <td style="text-align: center;"><u><strong>0.40</strong></u></td>
  <td style="text-align: center;">0.56</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">0.76</td>
  <td style="text-align: center;">0.48</td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-ASR-Part 5</td>
  <td style="text-align: center;"><u><strong>0.21</strong></u></td>
  <td style="text-align: center;">0.28</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">0.35</td>
  <td style="text-align: center;">0.23</td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-ASR-Part 6</td>
  <td style="text-align: center;"><u><strong>0.15</strong></u></td>
  <td style="text-align: center;">0.22</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">0.25</td>
  <td style="text-align: center;">0.18</td>
</tr>
<tr>
  <td style="text-align: center;" rowspan="6"><strong>Speech Translation</strong><br>BLEU (<span
  class="math inline">↑</span>)</td>
  <td style="text-align: center;">CoVoST 2 En <span
  class="math inline">→</span> Id</td>
  <td style="text-align: center;"><strong><u>32.62</u></strong></td>
  <td style="text-align: center;">16.33</td>
  <td style="text-align: center;">13.84</td>
  <td style="text-align: center;">14.14</td>
  <td style="text-align: center;">27.62</td>
</tr>
<tr>
  <td style="text-align: center;">CoVoST 2 En <span
  class="math inline">→</span> Zh</td>
  <td style="text-align: center;"><strong><u>37.98</u></strong></td>
  <td style="text-align: center;">25.77</td>
  <td style="text-align: center;">31.96</td>
  <td style="text-align: center;">33.89</td>
  <td style="text-align: center;">35.27</td>
</tr>
<tr>
  <td style="text-align: center;">CoVoST 2 En <span
  class="math inline">→</span> Ta</td>
  <td style="text-align: center;"><strong><u>8.50</u></strong></td>
  <td style="text-align: center;">0.03</td>
  <td style="text-align: center;">0.00</td>
  <td style="text-align: center;">0.00</td>
  <td style="text-align: center;">8.46</td>
</tr>
<tr>
  <td style="text-align: center;">CoVoST 2 Id <span
  class="math inline">→</span> En</td>
  <td style="text-align: center;"><strong>37.07</strong></td>
  <td style="text-align: center;">6.33</td>
  <td style="text-align: center;">5.93</td>
  <td style="text-align: center;">26.89</td>
  <td style="text-align: center;"><u>46.80</u></td>
</tr>
<tr>
  <td style="text-align: center;">CoVoST 2 Zh <span
  class="math inline">→</span> En</td>
  <td style="text-align: center;">15.01</td>
  <td style="text-align: center;"><strong><u>16.47</u></strong></td>
  <td style="text-align: center;">2.37</td>
  <td style="text-align: center;">5.30</td>
  <td style="text-align: center;">15.21</td>
</tr>
<tr>
  <td style="text-align: center;">CoVoST 2 Ta <span
  class="math inline">→</span> En</td>
  <td style="text-align: center;"><strong><u>3.97</u></strong></td>
  <td style="text-align: center;">0.04</td>
  <td style="text-align: center;">0.17</td>
  <td style="text-align: center;">0.36</td>
  <td style="text-align: center;">2.83</td>
</tr>
<tr>
  <td style="text-align: center;" rowspan="8"><strong>Spoken Question Answering</strong><br>LLM-as-a-Judge (<span
  class="math inline">↑</span>)</td>
  <td style="text-align: center;">SLUE-SQA-5</td>
  <td style="text-align: center;">82.94</td>
  <td style="text-align: center;">80.05</td>
  <td style="text-align: center;"><strong>83.92</strong></td>
  <td style="text-align: center;">83.48</td>
  <td style="text-align: center;"><u>88.58</u></td>
</tr>
<tr>
  <td style="text-align: center;">Spoken-SQuAD</td>
  <td style="text-align: center;">70.33</td>
  <td style="text-align: center;">64.86</td>
  <td style="text-align: center;"><strong>77.65</strong></td>
  <td style="text-align: center;">66.40</td>
  <td style="text-align: center;"><u>88.62</u></td>
</tr>
<tr>
  <td style="text-align: center;">CN-College-Listen-Test</td>
  <td style="text-align: center;"><strong>85.03</strong></td>
  <td style="text-align: center;">74.51</td>
  <td style="text-align: center;">65.43</td>
  <td style="text-align: center;">50.90</td>
  <td style="text-align: center;"><u>91.85</u></td>
</tr>
<tr>
  <td style="text-align: center;">Singapore-Public-Speech-SQA</td>
  <td style="text-align: center;"><strong>60.32</strong></td>
  <td style="text-align: center;">58.31</td>
  <td style="text-align: center;">58.55</td>
  <td style="text-align: center;">59.24</td>
  <td style="text-align: center;"><u>73.11</u></td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-SQA-Part 3</td>
  <td style="text-align: center;"><strong>51.4</strong></td>
  <td style="text-align: center;">42.0</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">40.60</td>
  <td style="text-align: center;"><u>53.20</u></td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-SQA-Part 4</td>
  <td style="text-align: center;"><strong>49.0</strong></td>
  <td style="text-align: center;">39.6</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">36.60</td>
  <td style="text-align: center;"><u>60.20</u></td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-SQA-Part 5</td>
  <td style="text-align: center;"><strong>58.2</strong></td>
  <td style="text-align: center;">51.6</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">44.60</td>
  <td style="text-align: center;"><u>67.20</u></td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-SQA-Part 6</td>
  <td style="text-align: center;"><strong>65.2</strong></td>
  <td style="text-align: center;">53.6</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">46.80</td>
  <td style="text-align: center;"><u>71.60</u></td>
</tr>
<tr>
  <td style="text-align: center;" rowspan="4"><strong>Spoken Dialogue Summarization</strong><br>LLM-as-a-Judge (<span
  class="math inline">↑</span>)</td>
  <td style="text-align: center;">MNSC-SDS-Part 3</td>
  <td style="text-align: center;"><u><strong>46.80</strong></u></td>
  <td style="text-align: center;">33.80</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">9.0</td>
  <td style="text-align: center;">45.40</td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-SDS-Part 4</td>
  <td style="text-align: center;"><u><strong>45.80</strong></u></td>
  <td style="text-align: center;">24.80</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">7.0</td>
  <td style="text-align: center;">44.00</td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-SDS-Part 5</td>
  <td style="text-align: center;"><strong>55.2</strong></td>
  <td style="text-align: center;">40.4</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">17.2</td>
  <td style="text-align: center;"><u>58.00</u></td>
</tr>
<tr>
  <td style="text-align: center;">MNSC-SDS-Part 6</td>
  <td style="text-align: center;"><strong>61.8</strong></td>
  <td style="text-align: center;">46.2</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">24.2</td>
  <td style="text-align: center;"><u>65.40</u></td>
</tr>
<tr>
  <td style="text-align: center;" rowspan="2"><strong>Speech Instruction</strong><br>LLM-as-a-Judge (<span
  class="math inline">↑</span>)</td>
  <td style="text-align: center;">OpenHermes-Audio</td>
  <td style="text-align: center;"><strong>71.4</strong></td>
  <td style="text-align: center;">44.8</td>
  <td style="text-align: center;">22.40</td>
  <td style="text-align: center;">15.80</td>
  <td style="text-align: center;"><u>72.20</u></td>
</tr>
<tr>
  <td style="text-align: center;">Alpaca-GPT4-Audio</td>
  <td style="text-align: center;"><strong>73.4</strong></td>
  <td style="text-align: center;">52.6</td>
  <td style="text-align: center;">21.60</td>
  <td style="text-align: center;">17.20</td>
  <td style="text-align: center;"><u>73.80</u></td>
</tr>
<tr>
  <td style="text-align: center;" rowspan="4"><strong>Paralinguistics</strong><br>LLM-as-a-Judge (<span
  class="math inline">↑</span>)</td>
  <td style="text-align: center;">VoxCeleb-Gender-Test</td>
  <td style="text-align: center;"><strong><u>99.53</u></strong></td>
  <td style="text-align: center;">99.12</td>
  <td style="text-align: center;">69.68</td>
  <td style="text-align: center;">88.81</td>
  <td style="text-align: center;">35.25</td>
</tr>
<tr>
  <td style="text-align: center;">VoxCeleb-Accent-Test</td>
  <td style="text-align: center;"><strong><u>46.35</u></strong></td>
  <td style="text-align: center;">29.18</td>
  <td style="text-align: center;">-</td>
  <td style="text-align: center;">34.22</td>
  <td style="text-align: center;">24.64</td>
</tr>
<tr>
  <td style="text-align: center;">MELD-Sentiment-Test</td>
  <td style="text-align: center;">42.26</td>
  <td style="text-align: center;"><strong>53.49</strong></td>
  <td style="text-align: center;">50.08</td>
  <td style="text-align: center;">42.07</td>
  <td style="text-align: center;"><u>56.67</u></td>
</tr>
<tr>
  <td style="text-align: center;">MELD-Emotion-Test</td>
  <td style="text-align: center;">30.15</td>
  <td style="text-align: center;">40.54</td>
  <td style="text-align: center;"><strong>41.07</strong></td>
  <td style="text-align: center;">30.73</td>
  <td style="text-align: center;"><u>47.39</u></td>
</tr>
</tbody>
</table>
</div>

## Uses

Here we provide a code snippet illustrating the process of loading both the processor and model, alongside detailed instructions on executing the MERaLiON-AudioLLM model for content generation.

> [!WARNING]
> **Out of Scope use**: This model is not intended for use in tool calling, math, and coding tasks.

### Inference

```python
from datasets import load_dataset
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor

repo_id = "MERaLiON/MERaLiON-AudioLLM-Whisper-SEA-LION"

processor = AutoProcessor.from_pretrained(
    repo_id, 
    trust_remote_code=True,
    )
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    repo_id,
    use_safetensors=True,
    trust_remote_code=True,
)

prompt = "Given the following audio context: <SpeechHere>\n\nText instruction: {query}"
query = "Please transcribe this speech."
conversation = [
    {"role": "user", "content": prompt.format(query=query)}
]

chat_prompt = processor.tokenizer.apply_chat_template(
    conversation=conversation,
    tokenize=False,
    add_generation_prompt=True
)

libri_data = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
audio_array = libri_data[0]["audio"]["array"]
inputs = processor(text=chat_prompt, audios=audio_array)

outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.1, top_p=0.9, repetition_penalty=1.1)
generated_ids = outputs[:, inputs['input_ids'].size(1):]
response = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
```

### Batch Inference

MERaLiON-AudioLLM also supports batch inference.

```python
from datasets import load_dataset
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor

repo_id = "MERaLiON/MERaLiON-AudioLLM-Whisper-SEA-LION"

processor = AutoProcessor.from_pretrained(
    repo_id, 
    trust_remote_code=True,
    )
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    repo_id,
    use_safetensors=True,
    trust_remote_code=True,
)

prompt = "Given the following audio context: <SpeechHere>\n\nText instruction: {query}"
transcribe_query = "Please transcribe this speech."
translate_query = "Can you please translate this speech into written Chinese?"

conversation = [
    [{"role": "user", "content": prompt.format(query=transcribe_query)}],
    [{"role": "user", "content": prompt.format(query=translate_query)}],
]

chat_prompt = processor.tokenizer.apply_chat_template(
    conversation=conversation,
    tokenize=False,
    add_generation_prompt=True
)

libri_data = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
audio_array = [libri_data[0]["audio"]["array"]]*2
inputs = processor(text=chat_prompt, audios=audio_array)

outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.1, top_p=0.9, repetition_penalty=1.1)
generated_ids = outputs[:, inputs['input_ids'].size(1):]
response = processor.batch_decode(generated_ids, skip_special_tokens=True)
```

### vLLM Inference

We support hosting the model using vLLM framework. Refer to the guide [here](vllm_plugin_meralion/README.md).

## Disclaimer

The current MERaLiON-AudioLLM has not been specifically aligned for safety and may generate content that is inappropriate, offensive, or harmful. Developers and users are responsible for performing their own safety fine-tuning and implementing necessary security measures. The authors shall not be held liable for any claims, damages, or other liabilities arising from the use of the released models, weights, or code.

## Technical Specifications 

### Training Data

MERaLiON-AudioLLM is trained on a diverse collection of publicly available datasets, alongside synthesised and augmented samples carefully curated by the team and native speakers, totaling 260,000 hours of audio.

### Compute and Infrastructure

MERaLiON-AudioLLM is trained on the **ASPIRE 2A+** Supercomputer Cluster, provided by **National Supercomputing Centre (NSCC)**, Singapore. ASPIRE 2A+ cluster provides multiple H100 nodes, with each compute node equipped with 8 Nvidia H100 GPUs, 2 TB of RAM, and 30 TB of locally attached NVMe storage. These nodes are interconnected via a rail-optimised, full fat-tree topology, utilising 400 Gb/s NDR InfiniBand cables. Additionally, the cluster incorporates a 2.5 PB SSD-based Lustre file system, linked to the H100 nodes through high-speed InfiniBand connections. 

With a global batch size of 640, we train the current release of MERaLiON-AudioLLM for around 200k steps, which took 2 days to complete using 16 nodes, 128 H100 GPUs.

## Citation

If you find our work useful, please cite our paper:

```
@misc{he2024meralionaudiollmtechnicalreport,
      title={MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Models}, 
      author={{MERaLiON Team}},
      year={2024},
      eprint={2412.09818},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.09818}, 
}
```