README.md

---
language:
- th
license: mit
library_name: transformers
tags:
- whisper-event
- generated_from_trainer
datasets:
- CMKL/Porjai-Thai-voice-dataset-central
metrics:
- wer
base_model: biodatlab/whisper-th-medium-combined
model-index:
- name: Whisper Medium Thai Timestamp - biodatlab
  results:
  - task:
      type: automatic-speech-recognition
      name: Automatic Speech Recognition
    dataset:
      name: mozilla-foundation/common_voice_13_0 th
      type: mozilla-foundation/common_voice_13_0
      config: th
      split: test
      args: th
    metrics:
    - type: wer
      value: 15.57
      name: Wer
---

# Whisper Medium (Thai) Timestamp

This model is a fine-tuned version of [biodatlab/whisper-th-medium-combined](biodatlab/whisper-th-medium-combined) on a custom-created longform dataset derived from the CMKL/Porjai-Thai-voice-dataset-central. It achieves the following results on the common-voice-13 test set:
- WER: 15.57 (with Deepcut Tokenizer)

## Model description

This model is designed to perform automatic speech recognition (ASR) for the Thai language, with the added capability of generating timestamps for the transcribed text. It's based on the Whisper medium architecture and has been fine-tuned on a specially crafted dataset to enable timestamp generation.

Use the model with Hugging Face's `transformers` as follows:

```py
from transformers import pipeline
import torch

MODEL_NAME = "biodatlab/whisper-th-medium-timestamp"  # specify the model name
lang = "th"  # Thai language

device = 0 if torch.cuda.is_available() else "cpu"

pipe = pipeline(
    task="automatic-speech-recognition",
    model=MODEL_NAME,
    chunk_length_s=30,
    device=device,
    return_timestamps=True,
)
pipe.model.config.forced_decoder_ids = pipe.tokenizer.get_decoder_prompt_ids(
  language=lang,
  task="transcribe"
)
result = pipe("audio.mp3", return_timestamps=True)
text = result["text"]
timestamps = result["chunks"]
```

## Intended uses & limitations
This model is intended for Thai automatic speech recognition tasks, particularly where timestamp information is required. It can be used for transcribing Thai audio content, creating subtitles, or any application that needs to align text with specific time points in audio.
The model's performance on speech recognition may be lower compared to non-timestamped versions due to the additional complexity of the task and the pseudo-timestamp generation method used in training.
## Training and evaluation data
The model was trained on a custom-created longform dataset derived from the CMKL/Porjai-Thai-voice-dataset-central. The dataset creation process involved the following steps:

- Combining multiple short audio clips from the original dataset into longer audio segments (up to 30 seconds).
- Adding environmental noises and silences between clips to simulate more realistic speech scenarios.
- Generating pseudo-timestamps for the combined audio using a Voice Activity Detection (VAD) model (Silero VAD).

This approach allowed us to create a dataset with longer, more diverse audio samples and approximate timestamp information, which is crucial for training a model capable of generating timestamps.
## Training procedure
The model was fine-tuned using a custom training script that incorporates the following:

- Mixed precision training (FP16)
- Gradient accumulation
- SpecAugment for data augmentation during training

## Training hyperparameters
The following hyperparameters were used during training:

- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 8
- gradient_accumulation_steps: 1
- num_train_iters: ~50000
- warmup_steps: 500
- fp16: True
- optimizer: AdamW
- lr_scheduler_type: linear

## Framework versions

- Transformers 4.44.2
- Pytorch 2.4.1
- Datasets 3.0.0
- Tokenizers 0.20.0

## Performance and Limitations
The WER (Word Error Rate) of 15.57 on the Common Voice 13 test set indicates good performance for Thai ASR. However, it's important to note that the timestamp generation model has a lower accuracy compared to the non-timestamped version of the model. This is due to several factors:

- The use of pseudo-timestamps in training data, which are approximations based on VAD rather than precise human annotations.
- The additional complexity of the timestamp prediction task, which requires the model to learn both transcription and temporal alignment.
- Potential discrepancies between the VAD-generated timestamps and actual word boundaries in continuous speech.

Users should be aware that while the timestamps provide a general indication of when words or phrases occur in the audio, they may not be as precise as manually annotated timestamps. The model's performance may also vary depending on the acoustic conditions, speaker variability, and the presence of background noise in the input audio.
## Citation
If you use this model in your research or applications, please cite it as follows:
```
@misc{biodatlab_whisper_th_medium_timestamp,
    author       = {Atirut Boribalburephan, Zaw Htet Aung, Knot Pipatsrisawat, Titipat Achakulvisut},
    title        = {Whisper Medium Thai Timestamp: A fine-tuned Whisper model for Thai automatic speech recognition with timestamp generation},
    year         = 2024,
    publisher    = {Hugging Face},
    howpublished = {\url{https://huggingface.co./biodatlab/whisper-th-medium-timestamp}}
}
```