Update README.md

README.md

@@ -13,12 +13,11 @@ tags:
 
 # MERaLiON
 
-MERaLiON-AudioLLM is a Speech-Text Large Language Model tailored for Singapore’s multilingual and multicultural landscape. Integrating a localised [Whisper-large-v2](https://huggingface.co/openai/whisper-large-v2) speech encoder and [SEA-LION V3](https://huggingface.co/aisingapore/gemma2-9b-cpt-sea-lionv3-instruct) text decoder, MERaLiON-AudioLLM is finetuned on **260,000 hours of speech and audio data**, **8 various tasks**, to address the diverse linguistic nuances of Singapore's local accents and dialects.
+MERaLiON-AudioLLM is a Speech-Text Large Language Model tailored for Singapore’s multilingual and multicultural landscape. Integrating a localised [Whisper-large-v2](https://huggingface.co/openai/whisper-large-v2) speech encoder and [SEA-LION V3](https://huggingface.co/aisingapore/gemma2-9b-cpt-sea-lionv3-instruct) text decoder, MERaLiON-AudioLLM is finetuned on **260,000 hours of speech and audio data**, **6 various tasks**, to address the diverse linguistic nuances of Singapore's local accents and dialects.
 
 MERaLiON stands for **M**ultimodal **E**mpathetic **R**easoning **a**nd **L**earning **i**n **O**ne **N**etwork.
 
 - **Developed by:** I<sup>2</sup>R, A\*STAR
-- **Funded by:** Singapore NRF
 - **Model type:** MultiModal LLM
 - **Language(s) (Speech):** English (Global & Singapore)
 - **Language(s) (NLP):** English, Chinese, Vietnamese, Indonesian, Thai, Filipino, Tamil, Malay, Khmer, Lao, Burmese, Javanese, Sundanese
@@ -28,7 +27,7 @@ For more details, please refer to our [report]().
 
 ## Model Description
 
-MERaLiON-AudioLLM is designed to take in an **audio-text pair** as input and generates a **text output**.
+MERaLiON-AudioLLM is designed to take in an **audio-text pair** as input and generate a **text output**.
 
 The architecture comprises three key components: an **audio encoder** that transforms speech or audio inputs into sequences of vector representations, a **text decoder** that interprets and responds to natural language instructions, and an **adaptor module** that compresses the encoder representations while aligning the encoder’s hidden dimension with the text decoder’s embedding size.
 
@@ -38,15 +37,349 @@ Specifically, we fine-tuned the **MERaLiON-Whisper** encoder from Whisper-large-
 
 ## Capabilities
 
-MERaLiON-AudioLLM is trained to address 8 tasks, including `Automatic Speech Recognition` (ASR), `Speech Translation` (ST), `Spoken Question Answering` (SQA), `Spoken Dialogue Summarization` (SDS), `Speech Instruction` (SI), `Paralinguistics` (PARA), `Audio Captioning` (AC), and `Audio Scene Question Answering` (ASQA). 
-
-[More information about the 8 tasks and evaluation results]
+MERaLiON-AudioLLM is trained to mainly address 6 tasks, namely `Automatic Speech Recognition` (ASR), 
+`Speech Translation` (ST), `Spoken Question Answering` (SQA), 
+`Spoken Dialogue Summarization` (SDS), `Speech Instruction` (SI), `Paralinguistics` (PARA). 
+
+We benchmark MERaLiON-AudioLLM with a series of test sets from the [AudioBench benchmark](https://github.com/AudioLLMs/AudioBench) 
+against three well-known AudioLLMs: `Qwen2-Audio 7B`, `WavLLM`, and `SALMONN`. We also compared with a cascaded model, 
+which feeds the transcriptions recognized by Whisper-large-v2 and the instruction prompts to a Gemma2 9B CPT SEA-LIONv3 Instruct model to
+get the responses. We tuned its hyperparameters and prompt template to optimise performance across
+various speech-to-text tasks. As is shown in the following table, MERaLiON-AudioLLM performs better in the Singapore local context, 
+as evidenced by evaluation results on Singapore's [Multitask National Speech Corpus](MERaLiON/MNSC) (MNSC) datasets. 
+
+> [!NOTE]
+> MNSC is a multitask speech understanding dataset derived and further annotated from [IMDA NSC Corpus](https://www.imda.gov.sg/how-we-can-help/national-speech-corpus).
+> It focuses on the knowledge of Singapore's local accent, localised terms, and code-switching.
+
+> [!NOTE]
+> We assess ASR and ST tasks using Word Error Rate (WER) and BLEU scores, respectively.
+> For other tasks, we employ the LLM-as-a-Judge framework,
+> which uses a pre-trained large language model to evaluate task performance
+> by generating and scoring responses based on criteria such as relevance, coherence, and accuracy.
+
+<div class="table*">
+<table>
+<thead>
+<tr>
+  <th style="text-align: center;"><strong>Task</strong></th>
+  <th style="text-align: center;"><strong>Dataset</strong></th>
+  <th style="text-align: center;"><strong>MERaLiON</strong></th>
+  <th style="text-align: center;"><strong>Qwen2-Audio 7B</strong></th>
+  <th style="text-align: center;"><strong>WavLLM</strong></th>
+  <th style="text-align: center;"><strong>SALMONN-7B</strong></th>
+  <th style="text-align: center;"><strong>Cascaded Model</strong></th>
+</tr>
+</thead>
+<tbody>
+<tr>
+  <td style="text-align: center;" rowspan="11"><em>Automatic Speech Recognition<br>WER (<span
+  class="math inline">↓</span>)</em></td>
+  <td style="text-align: center;">LibriSpeech-Test-Clean</td>
+  <td style="text-align: center;">0.03</td>
+  <td style="text-align: center;">0.03</td>
+  <td style="text-align: center;"><strong><u>0.02</u></strong></td>
+  <td style="text-align: center;">0.10</td>
+  <td style="text-align: center;">0.03</td>
+</tr>
+<tr>
+  <td style="text-align: center;">LibriSpeech-Test-Other</td>
+  <td style="text-align: center;"><strong><u>0.05</u></strong></td>
+  <td style="text-align: center;">0.06</td>
+  <td style="text-align: center;"><strong><u>0.05</u></strong></td>
+  <td style="text-align: center;">0.10</td>
+  <td style="text-align: center;"><u>0.05</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">Common-Voice-15-En-Test</td>
+  <td style="text-align: center;"><strong><u>0.10</u></strong></td>
+  <td style="text-align: center;">0.11</td>
+  <td style="text-align: center;">0.15</td>
+  <td style="text-align: center;">0.31</td>
+  <td style="text-align: center;">0.11</td>
+</tr>
+<tr>
+  <td style="text-align: center;">Earnings21-Test</td>
+  <td style="text-align: center;"><strong>0.17</strong></td>
+  <td style="text-align: center;">0.19</td>
+  <td style="text-align: center;">0.65</td>
+  <td style="text-align: center;">0.26</td>
+  <td style="text-align: center;"><u>0.11</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">Earnings22-Test</td>
+  <td style="text-align: center;"><strong>0.20</strong></td>
+  <td style="text-align: center;">0.24</td>
+  <td style="text-align: center;">0.67</td>
+  <td style="text-align: center;">0.36</td>
+  <td style="text-align: center;"><u>0.14</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-ASR-Part 1</td>
+  <td style="text-align: center;"><u><strong>0.05</strong></u></td>
+  <td style="text-align: center;">0.07</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">0.09</td>
+  <td style="text-align: center;">0.07</td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-ASR-Part 2</td>
+  <td style="text-align: center;"><u><strong>0.05</strong></u></td>
+  <td style="text-align: center;">0.19</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">0.42</td>
+  <td style="text-align: center;">0.33</td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-ASR-Part 3</td>
+  <td style="text-align: center;"><u><strong>0.28</strong></u></td>
+  <td style="text-align: center;">0.35</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">0.66</td>
+  <td style="text-align: center;">0.30</td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-ASR-Part 4</td>
+  <td style="text-align: center;"><u><strong>0.40</strong></u></td>
+  <td style="text-align: center;">0.56</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">0.76</td>
+  <td style="text-align: center;">0.48</td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-ASR-Part 5</td>
+  <td style="text-align: center;"><u><strong>0.21</strong></u></td>
+  <td style="text-align: center;">0.28</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">0.35</td>
+  <td style="text-align: center;">0.23</td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-ASR-Part 6</td>
+  <td style="text-align: center;"><u><strong>0.15</strong></u></td>
+  <td style="text-align: center;">0.22</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">0.25</td>
+  <td style="text-align: center;">0.18</td>
+</tr>
+<tr>
+  <td style="text-align: center;" rowspan="6"><em>Speech Translation<br>BLEU (<span
+  class="math inline">↑</span>)</em></td>
+  <td style="text-align: center;">CoVoST 2 En <span
+  class="math inline">→</span> Id</td>
+  <td style="text-align: center;"><strong><u>32.62</u></strong></td>
+  <td style="text-align: center;">16.33</td>
+  <td style="text-align: center;">13.84</td>
+  <td style="text-align: center;">14.14</td>
+  <td style="text-align: center;">27.62</td>
+</tr>
+<tr>
+  <td style="text-align: center;">CoVoST 2 En <span
+  class="math inline">→</span> Zh</td>
+  <td style="text-align: center;"><strong><u>37.98</u></strong></td>
+  <td style="text-align: center;">25.77</td>
+  <td style="text-align: center;">31.96</td>
+  <td style="text-align: center;">33.89</td>
+  <td style="text-align: center;">35.27</td>
+</tr>
+<tr>
+  <td style="text-align: center;">CoVoST 2 En <span
+  class="math inline">→</span> Ta</td>
+  <td style="text-align: center;"><strong><u>8.50</u></strong></td>
+  <td style="text-align: center;">0.03</td>
+  <td style="text-align: center;">0.00</td>
+  <td style="text-align: center;">0.00</td>
+  <td style="text-align: center;">8.46</td>
+</tr>
+<tr>
+  <td style="text-align: center;">CoVoST 2 Id <span
+  class="math inline">→</span> En</td>
+  <td style="text-align: center;"><strong>37.07</strong></td>
+  <td style="text-align: center;">6.33</td>
+  <td style="text-align: center;">5.93</td>
+  <td style="text-align: center;">26.89</td>
+  <td style="text-align: center;"><u>46.80</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">CoVoST 2 Zh <span
+  class="math inline">→</span> En</td>
+  <td style="text-align: center;">15.01</td>
+  <td style="text-align: center;"><strong><u>16.47</u></strong></td>
+  <td style="text-align: center;">2.37</td>
+  <td style="text-align: center;">5.30</td>
+  <td style="text-align: center;">15.21</td>
+</tr>
+<tr>
+  <td style="text-align: center;">CoVoST 2 Ta <span
+  class="math inline">→</span> En</td>
+  <td style="text-align: center;"><strong><u>3.97</u></strong></td>
+  <td style="text-align: center;">0.04</td>
+  <td style="text-align: center;">0.17</td>
+  <td style="text-align: center;">0.36</td>
+  <td style="text-align: center;">2.83</td>
+</tr>
+<tr>
+  <td style="text-align: center;" rowspan="8"><em>Spoken Question Answering<br>LLM-as-a-Judge (<span
+  class="math inline">↑</span>)</em></td>
+  <td style="text-align: center;">SLUE-SQA-5</td>
+  <td style="text-align: center;">82.94</td>
+  <td style="text-align: center;">80.05</td>
+  <td style="text-align: center;"><strong>83.92</strong></td>
+  <td style="text-align: center;">83.48</td>
+  <td style="text-align: center;"><u>88.58</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">Spoken-SQuAD</td>
+  <td style="text-align: center;">70.33</td>
+  <td style="text-align: center;">64.86</td>
+  <td style="text-align: center;"><strong>77.65</strong></td>
+  <td style="text-align: center;">66.40</td>
+  <td style="text-align: center;"><u>88.62</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">CN-College-Listen-Test</td>
+  <td style="text-align: center;"><strong>85.03</strong></td>
+  <td style="text-align: center;">74.51</td>
+  <td style="text-align: center;">65.43</td>
+  <td style="text-align: center;">50.90</td>
+  <td style="text-align: center;"><u>91.85</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">Singapore-Public-Speech-SQA</td>
+  <td style="text-align: center;"><strong>60.32</strong></td>
+  <td style="text-align: center;">58.31</td>
+  <td style="text-align: center;">58.55</td>
+  <td style="text-align: center;">59.24</td>
+  <td style="text-align: center;"><u>73.11</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-SQA-Part 3</td>
+  <td style="text-align: center;"><strong>51.4</strong></td>
+  <td style="text-align: center;">42.0</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">40.60</td>
+  <td style="text-align: center;"><u>53.20</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-SQA-Part 4</td>
+  <td style="text-align: center;"><strong>49.0</strong></td>
+  <td style="text-align: center;">39.6</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">36.60</td>
+  <td style="text-align: center;"><u>60.20</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-SQA-Part 5</td>
+  <td style="text-align: center;"><strong>58.2</strong></td>
+  <td style="text-align: center;">51.6</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">44.60</td>
+  <td style="text-align: center;"><u>67.20</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-SQA-Part 6</td>
+  <td style="text-align: center;"><strong>65.2</strong></td>
+  <td style="text-align: center;">53.6</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">46.80</td>
+  <td style="text-align: center;"><u>71.60</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;" rowspan="4"><em>Spoken Dialogue Summarization<br>LLM-as-a-Judge (<span
+  class="math inline">↑</span>)</em></td>
+  <td style="text-align: center;">MNSC-SDS-Part 3</td>
+  <td style="text-align: center;"><u><strong>46.80</strong></u></td>
+  <td style="text-align: center;">33.80</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">9.0</td>
+  <td style="text-align: center;">45.40</td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-SDS-Part 4</td>
+  <td style="text-align: center;"><u><strong>45.80</strong></u></td>
+  <td style="text-align: center;">24.80</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">7.0</td>
+  <td style="text-align: center;">44.00</td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-SDS-Part 5</td>
+  <td style="text-align: center;"><strong>55.2</strong></td>
+  <td style="text-align: center;">40.4</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">17.2</td>
+  <td style="text-align: center;"><u>58.00</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">MNSC-SDS-Part 6</td>
+  <td style="text-align: center;"><strong>61.8</strong></td>
+  <td style="text-align: center;">46.2</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">24.2</td>
+  <td style="text-align: center;"><u>65.40</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;" rowspan="2"><em>Speech Instruction<br>LLM-as-a-Judge (<span
+  class="math inline">↑</span>)</em></td>
+  <td style="text-align: center;">OpenHermes-Audio</td>
+  <td style="text-align: center;"><strong>71.4</strong></td>
+  <td style="text-align: center;">44.8</td>
+  <td style="text-align: center;">22.40</td>
+  <td style="text-align: center;">15.80</td>
+  <td style="text-align: center;"><u>72.20</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">Alpaca-GPT4-Audio</td>
+  <td style="text-align: center;"><strong>73.4</strong></td>
+  <td style="text-align: center;">52.6</td>
+  <td style="text-align: center;">21.60</td>
+  <td style="text-align: center;">17.20</td>
+  <td style="text-align: center;"><u>73.80</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;" rowspan="4"><em>Paralinguistics<br>LLM-as-a-Judge (<span
+  class="math inline">↑</span>)</em></td>
+  <td style="text-align: center;">VoxCeleb-Gender-Test</td>
+  <td style="text-align: center;"><strong><u>99.53</u></strong></td>
+  <td style="text-align: center;">99.12</td>
+  <td style="text-align: center;">69.68</td>
+  <td style="text-align: center;">88.81</td>
+  <td style="text-align: center;">35.25</td>
+</tr>
+<tr>
+  <td style="text-align: center;">VoxCeleb-Accent-Test</td>
+  <td style="text-align: center;"><strong><u>46.35</u></strong></td>
+  <td style="text-align: center;">29.18</td>
+  <td style="text-align: center;">-</td>
+  <td style="text-align: center;">34.22</td>
+  <td style="text-align: center;">24.64</td>
+</tr>
+<tr>
+  <td style="text-align: center;">MELD-Sentiment-Test</td>
+  <td style="text-align: center;">42.26</td>
+  <td style="text-align: center;"><strong>53.49</strong></td>
+  <td style="text-align: center;">50.08</td>
+  <td style="text-align: center;">42.07</td>
+  <td style="text-align: center;"><u>56.67</u></td>
+</tr>
+<tr>
+  <td style="text-align: center;">MELD-Emotion-Test</td>
+  <td style="text-align: center;">30.15</td>
+  <td style="text-align: center;">40.54</td>
+  <td style="text-align: center;"><strong>41.07</strong></td>
+  <td style="text-align: center;">30.73</td>
+  <td style="text-align: center;"><u>47.39</u></td>
+</tr>
+</tbody>
+</table>
+</div>
 
 ## Uses
 
 Here we provide a code snippet illustrating the process of loading both the processor and model, alongside detailed instructions on executing the MERaLiON-AudioLLM model for content generation.
 
-**NOTE** This model has not been trained to use a system prompt or to use tool calling.
+> [!WARNING]
+> This model has not been trained to use a system prompt or to use tool calling.
 
 ### Inference