YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co./docs/hub/model-cards#model-card-metadata)

CodeBERT-base-mlm

Pretrained weights for CodeBERT: A Pre-Trained Model for Programming and Natural Languages.

Training Data

The model is trained on the code corpus of CodeSearchNet

Training Objective

This model is initialized with Roberta-base and trained with a simple MLM (Masked Language Model) objective.

Usage

from transformers import RobertaTokenizer, RobertaForMaskedLM, pipeline

model = RobertaForMaskedLM.from_pretrained('microsoft/codebert-base-mlm')
tokenizer = RobertaTokenizer.from_pretrained('microsoft/codebert-base-mlm')

code_example = "if (x is not None) <mask> (x>1)"
fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)

outputs = fill_mask(code_example)
print(outputs)

Expected results:

{'sequence': '<s> if (x is not None) and (x>1)</s>', 'score': 0.6049249172210693, 'token': 8}
{'sequence': '<s> if (x is not None) or (x>1)</s>', 'score': 0.30680200457572937, 'token': 50}
{'sequence': '<s> if (x is not None) if (x>1)</s>', 'score': 0.02133703976869583, 'token': 114}
{'sequence': '<s> if (x is not None) then (x>1)</s>', 'score': 0.018607674166560173, 'token': 172}
{'sequence': '<s> if (x is not None) AND (x>1)</s>', 'score': 0.007619690150022507, 'token': 4248}

Reference

  1. Bimodal CodeBERT trained with MLM+RTD objective (suitable for code search and document generation)
  2. ๐Ÿค— Hugging Face's CodeBERTa (small size, 6 layers)

Citation

@misc{feng2020codebert,
    title={CodeBERT: A Pre-Trained Model for Programming and Natural Languages},
    author={Zhangyin Feng and Daya Guo and Duyu Tang and Nan Duan and Xiaocheng Feng and Ming Gong and Linjun Shou and Bing Qin and Ting Liu and Daxin Jiang and Ming Zhou},
    year={2020},
    eprint={2002.08155},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
71,704
Inference API
Examples
Mask token: <mask>

Model tree for microsoft/codebert-base-mlm

Adapters
2 models
Finetunes
16 models

Space using microsoft/codebert-base-mlm 1