metadata

license: apache-2.0
language: en
datasets:
  - Jzuluaga/atco2_corpus_1h
tags:
  - text
  - token-classification
  - en-atc
  - en
  - generated_from_trainer
  - bert
  - ner-for-atc
metrics:
  - Precision
  - Recall
  - Accuracy
  - F1
widget:
  - text: >-
      lining up runway three one csa five bravo easy five three kilo romeo
      contact ruzyne ground one two one decimal nine good bye
  - text: >-
      csa seven three two zero so change of taxi quality eight nine sierra we
      need to full length britair five nine zero bravo contact ruzyne ground one
      two one decimal nine good bye
  - text: >-
      swiss four six one foxtrot line up runway three one and wait one two one
      nine csa four yankee alfa
  - text: >-
      tower klm five five tango ils three one wizz air four papa uniform tower
      roger
model-index:
  - name: bert-base-ner-atc-en-atco2-1h
    results:
      - task:
          type: token-classification
          name: ner
        dataset:
          type: Jzuluaga/atco2_corpus_1h
          name: ATCO2 corpus (Air Traffic Control Communications)
          config: test
          split: test
        metrics:
          - type: F1
            value: 0.94
            name: TEST F1 (callsign)
            verified: false
          - type: F1
            value: 0.74
            name: TEST F1 (command)
            verified: false
          - type: F1
            value: 0.81
            name: TEST F1 (value)
            verified: false

bert-base-ner-atc-en-atco2-1h

This model allow to perform named-entity recognition (NER) on air traffic control communications data. We solve this challenge by performing token classification (NER) with a BERT model. We fine-tune a pretrained BERT model on the ner task.

For instance, if you have the following transcripts/gold annotations:

Utterance: lufthansa three two five cleared to land runway three four left

Could you tell what are the main entities in the communication? The desired output is shown below:

Named-entity module output: [call] lufthansa three two five [/call] [cmd] cleared to land [/cmd] [val] runway three four left [/val]

This model is a fine-tuned version of bert-base-uncased on the atco2_corpus_1h.

It achieves the following results on the development set:

Loss: 1.4282
Precision: 0.6195
Recall: 0.7071
F1: 0.6604
Accuracy: 0.8182

Paper: ATCO2 corpus: A Large-Scale Dataset for Research on Automatic Speech Recognition and Natural Language Understanding of Air Traffic Control Communications

Authors: Juan Zuluaga-Gomez, Karel Veselý, Igor Szöke, Petr Motlicek, Martin Kocour, Mickael Rigault, Khalid Choukri, Amrutha Prasad and others

Abstract: Personal assistants, automatic speech recognizers and dialogue understanding systems are becoming more critical in our interconnected digital world. A clear example is air traffic control (ATC) communications. ATC aims at guiding aircraft and controlling the airspace in a safe and optimal manner. These voice-based dialogues are carried between an air traffic controller (ATCO) and pilots via very-high frequency radio channels. In order to incorporate these novel technologies into ATC (low-resource domain), large-scale annotated datasets are required to develop the data-driven AI systems. Two examples are automatic speech recognition (ASR) and natural language understanding (NLU). In this paper, we introduce the ATCO2 corpus, a dataset that aims at fostering research on the challenging ATC field, which has lagged behind due to lack of annotated data. The ATCO2 corpus covers 1) data collection and pre-processing, 2) pseudo-annotations of speech data, and 3) extraction of ATC-related named entities. The ATCO2 corpus is split into three subsets. 1) ATCO2-test-set corpus contains 4 hours of ATC speech with manual transcripts and a subset with gold annotations for named-entity recognition (callsign, command, value). 2) The ATCO2-PL-set corpus consists of 5281 hours of unlabeled ATC data enriched with automatic transcripts from an in-domain speech recognizer, contextual information, speaker turn information, signal-to-noise ratio estimate and English language detection score per sample. Both available for purchase through ELDA at this http URL. 3) The ATCO2-test-set-1h corpus is a one-hour subset from the original test set corpus, that we are offering for free at this https URL. We expect the ATCO2 corpus will foster research on robust ASR and NLU not only in the field of ATC communications but also in the general research community.

Code — GitHub repository: https://github.com/idiap/atco2-corpus

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 32
eval_batch_size: 16
seed: 42
gradient_accumulation_steps: 2
total_train_batch_size: 64
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 500
training_steps: 3000

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
No log	125.0	500	0.8692	0.6396	0.7172	0.6762	0.8307
0.2158	250.0	1000	1.0074	0.5702	0.6970	0.6273	0.8245
0.2158	375.0	1500	1.3560	0.6577	0.7374	0.6952	0.8119
0.0184	500.0	2000	1.3393	0.6182	0.6869	0.6507	0.8056
0.0184	625.0	2500	1.3528	0.6087	0.7071	0.6542	0.8213
0.0175	750.0	3000	1.4282	0.6195	0.7071	0.6604	0.8182

Framework versions

Transformers 4.24.0
Pytorch 1.13.0+cu117
Datasets 2.7.0
Tokenizers 0.13.2