OpenCoder-LLM/OpenCoder-1.5B-Base-Checkpoints

🏠 Home Page   |    🤗 Model   |    📊 Dataset   |    📄Paper  

1. Introduction

OpenCoder is an open and reproducible code LLM family which includes 1.5B and 8B base and chat models, supporting both English and Chinese languages. Starting from scratch, OpenCoder is pretrained on 2.5 trillion tokens composed of 90% raw code and 10% code-related web data, and supervised finetuned on over 4.5M high-quality SFT examples, finally reaching the performance of top-tier code LLMs.

This repository contains all the intermediate checkpoints of OpenCoder-1.5B-Base, saving in different branches. For the final results, please refer to 🤗 OpenCoder-1.5B-Base

2. Branches Overview

• pretrain_iter_0001000 - pretrain_iter_0484865: Intermediate checkpoints during the pretraining stage.

• anneal_iter_0001000 - anneal_iter_0023841: Intermediate checkpoints during the annealing stage.

The number in each branch name indicates the corresponding current training step, where each step consumes 419,4304 training tokens (1,024 batch size * 4,096 sequence length).

We use pretrain_iter_0373000 as the starting point for the annealing stage, and use anneal_iter_0023000 as the final base model.