Unihan LM: Coarse-to-Fine Chinese-Japanese Language Model Pretraining with the Unihan Database

Model description

Chinese and Japanese share many characters with similar surface morphology. To better utilize the shared knowledge across the languages, we propose UnihanLM, a self-supervised Chinese-Japanese pretrained masked language model (MLM) with a novel two-stage coarse-to-fine training approach. We exploit Unihan, a ready-made database constructed by linguistic experts to first merge morphologically similar characters into clusters. The resulting clusters are used to replace the original characters in sentences for the coarse-grained pretraining of the MLM. Then, we restore the clusters back to the original characters in sentences for the fine-grained pretraining to learn the representation of the specific characters. We conduct extensive experiments on a variety of Chinese and Japanese NLP benchmarks, showing that our proposed UnihanLM is effective on both mono- and cross-lingual Chinese and Japanese tasks, shedding light on a new path to exploit the homology of languages. Paper

Intended uses & limitations

How to use

Use it like how you use XLM :)

Limitations and bias

The training corpus is solely from Wikipedia so the model may perform worse on informal text data. Be careful with English words! The tokenizer would cut it to characters.

Training data

We use Chinese and Japanese Wikipedia to train the model.

Training procedure

Please refer to our paper: https://www.aclweb.org/anthology/2020.aacl-main.24/

Eval results

Please refer to our paper: https://www.aclweb.org/anthology/2020.aacl-main.24/

BibTeX entry and citation info

@inproceedings{xu-etal-2020-unihanlm,
    title = "{U}nihan{LM}: Coarse-to-Fine {C}hinese-{J}apanese Language Model Pretraining with the Unihan Database",
    author = "Xu, Canwen  and
      Ge, Tao  and
      Li, Chenliang  and
      Wei, Furu",
    booktitle = "Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing",
    month = dec,
    year = "2020",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.aacl-main.24",
    pages = "201--211"
}