Model Card for COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
Model Details
Model Description
This model card contains the COCO-LM model (base++ version) pretrained models on GLUE and SQuAD 2.0 benchmarks.
- Developed by: Microsoft
- Shared by [Optional]: HuggingFace
- Model type: Language model
- Language(s) (NLP): en
- License: MIT
- Related Models: More information needed
- Parent Model: More information needed
- Resources for more information:
- GitHub Repo
- Associated Paper
Uses
Direct Use
Correcting and Contrasting Text Sequences for Language Model Pretraining
Downstream Use [Optional]
More information needed.
Out-of-Scope Use
The model should not be used to intentionally create hostile or alienating environments for people.
Bias, Risks, and Limitations
Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recomendations.
Training Details
Training Data
See the associated dataset card for further details
Training Procedure
Preprocessing
The model devloeprs note in the associated paper:
We employ three standard settings, base, base++, and large++. Base is the BERTBase training configuration: Pretraining on Wikipedia and BookCorpus (16 GB of texts) for 256 million samples on 512 token sequences (125K batches with 2048 batch size). We use the same corpus and 32, 768 uncased BPE vocabulary as with TUPE. Base++ trains the base size model with larger corpora and/or more training steps. Following recent research, we add in OpenWebText, CC-News , and STORIES, to a total of 160 GB texts, and train for 4 billion (with 2048 batch size) samples. We follow the prepossessing of UniLMV2 and use 64, 000 cased BPE vocabulary. Large++ uses the same training corpora as base++ and pretrains for 4 billion samples (2048 batch size). Its Transformer configuration is the same with BERTLarge.
Speeds, Sizes, Times
More information needed
Evaluation
Testing Data, Factors & Metrics
Testing Data
GLUE SQuAD 2.0
Factors
All results are single-task, single-model fine-tuning.
Metrics
More information needed
Results
GLUE Fine-Tuning Results
The General Language Understanding Evaluation (GLUE) benchmark is a collection of sentence- or sentence-pair language understanding tasks for evaluating and analyzing natural language understanding systems.
GLUE dev set results of COCO-LM base++ and large++ models are as follows (median of 5 different random seeds):
| Model | MNLI-m/mm | QQP | QNLI | SST-2 | CoLA | RTE | MRPC | STS-B | AVG |
|---|---|---|---|---|---|---|---|---|---|
| COCO-LM base++ | 90.2/90.0 | 92.2 | 94.2 | 94.6 | 67.3 | 87.4 | 91.2 | 91.8 | 88.6 |
| COCO-LM large++ | 91.4/91.6 | 92.8 | 95.7 | 96.9 | 73.9 | 91.0 | 92.2 | 92.7 | 90.8 |
GLUE test set results of COCO-LM base++ and large++ models are as follows (no ensemble, task-specific tricks, etc.):
| Model | MNLI-m/mm | QQP | QNLI | SST-2 | CoLA | RTE | MRPC | STS-B | AVG |
|---|---|---|---|---|---|---|---|---|---|
| COCO-LM base++ | 89.8/89.3 | 89.8 | 94.2 | 95.6 | 68.6 | 82.3 | 88.5 | 90.3 | 87.4 |
| COCO-LM large++ | 91.6/91.1 | 90.5 | 95.8 | 96.7 | 70.5 | 89.2 | 88.4 | 91.8 | 89.3 |
SQuAD 2.0 Fine-Tuning Results
Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.
SQuAD 2.0 dev set results of COCO-LM base++ and large++ models are as follows (median of 5 different random seeds):
| Model | EM | F1 |
|---|---|---|
| COCO-LM base++ | 85.4 | 88.1 |
| COCO-LM large++ | 88.2 | 91.0 |
Model Examination
The model delevopers note in the associated paper:
Architecture. Removing relative position encoding (Rel-Pos) leads to better numbers on some tasks but significantly hurts MNLI. Using a shallow auxiliary network and keeping the same hidden dimension (768) is more effective than ELECTRA’s 12-layer but 256-hidden dimension generator.
One limitation of this work is that the contrastive pairs are constructed by simple cropping and MLM replacements. Recent studies have shown the effectiveness of advanced data augmentation techniques in fine-tuning language models [16, 38, 51]. A future research direction is to explore better ways to construct contrastive pairs in language model pretraining.
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
- Hardware Type: More information needed
- Hours used: More information needed
- Cloud Provider: More information needed
- Compute Region: More information needed
- Carbon Emitted: More information needed}
Technical Specifications [optional]
Model Architecture and Objective
The model delevopers note in the associated paper:
Model Architecture. Our base/base++ model uses the BERTBase architecture: 12 layer Transformer, 768 hidden size, plus T5 relative position encoding. Our large++ model is the same with BERTLarge, 24 layer and 1024 hidden size, plus T5 relative position encoding. Our auxiliary network uses the same hidden size but a shallow 4-layer Transformer in base/base++ and a 6-layer one in large++. When generating XMLM we disable dropout in the auxiliary model
Compute Infrastructure
More information needed
Hardware
More information needed
Software
More information needed
Citation
BibTeX:
If you find the code and models useful for your research, please cite the following paper:
@inproceedings{meng2021cocolm,
title={{COCO-LM}: Correcting and contrasting text sequences for language model pretraining},
author={Meng, Yu and Xiong, Chenyan and Bajaj, Payal and Tiwary, Saurabh and Bennett, Paul and Han, Jiawei and Song, Xia},
booktitle={Conference on Neural Information Processing Systems},
year={2021}
}
Glossary [optional]
More information needed
More Information [optional]
More information needed
Model Card Authors [optional]
Microsoft in collaboration with Ezi Ozoani and the HuggingFace team
Model Card Contact
More information needed
How to Get Started with the Model
Use the code below to get started with the model.
Click to expand
from transformers import AutoModel
model = AutoModel.from_pretrained("microsoft/cocolm-base")