Update README.md

README.md

@@ -1,3 +1,75 @@
 ---
+language: 
+  - code
 license: apache-2.0
+tags:
+- code
+- gpt2
+- generation
+datasets:
+- "codeparrot/github-code-clean"
+- "openai_humaneval"
+metrics:
+- "evaluate-metric/code_eval"
 ---
+
+# CodeParrot-Multi 🦜 (small)
+
+CodeParrot-Multi 🦜 is a GPT-2 model (110M parameters) trained to generate code in 32 programming languages (Python, Java, C, JavaScript...)
+
+## Usage
+
+You can load the CodeParrot-Multi model and tokenizer directly in `transformers`:
+
+```Python
+from transformers import AutoTokenizer, AutoModelWithLMHead
+  
+tokenizer = AutoTokenizer.from_pretrained("codeparrot/codeparrot-small-multi")
+model = AutoModelWithLMHead.from_pretrained("codeparrot/codeparrot-small-multi")
+
+inputs = tokenizer("def hello_world():", return_tensors="pt")
+outputs = model(**inputs)
+```
+
+or with a `pipeline`:
+
+```Python
+from transformers import pipeline
+
+pipe = pipeline("text-generation", model="codeparrot/codeparrot-small-multi")
+outputs = pipe("def hello_world():")
+```
+
+## Training
+
+The model was trained on the cleaned [Github code dataset](https://huggingface.co/datasets/codeparrot/github-code-clean) with the following settings:
+
+|Config|Value|
+|-------|-----|
+|Batch size| 192 |
+|Context size| 1024 |
+|Training steps| 300'000|
+|Gradient accumulation| 2|
+|Gradient checkpointing| False|
+|Learning rate| 5e-4 |
+|Weight decay | 0.1 |
+|Warmup steps| 2000 |
+|Schedule| Cosine |
+
+The training was executed on 16 x A100 (40GB) GPUs. This setting amounts to roughly 58 billion tokens.
+
+## Performance
+
+We evaluated the model on OpenAI's [HumanEval](https://huggingface.co/datasets/openai_humaneval) benchmark which consists of programming challenges:
+
+| Metric | Value |
+|-------|-----|
+|pass@1 | --% |
+|pass@10 | --%	 |
+|pass@100 | --% |
+
+The [pass@k metric](https://huggingface.co/metrics/code_eval) tells the probability that at least one out of k generations passes the tests. 
+
+## Resources
+
+- Code: [repository](https://github.com/huggingface/transformers/tree/master/examples/research_projects/codeparrot)