evaluation/intro.txt

A popular evaluatrion framework for code generation models is the [pass@k](https://huggingface.co./metrics/code_eval) metric on [HumanEval](https://huggingface.co./datasets/openai_humaneval) dataset, which was introduced in [Codex paper](https://arxiv.org/pdf/2107.03374v2.pdf). The dataset includes 164 handwritten programming problems. In the pass@k metric, k code samples are generated per problem, a problem is considered solved if any sample passes the unit tests and the total fraction of problems solved is reported. Below are some examples for the selcted models.