Update README.md

README.md

@@ -5,14 +5,59 @@ datasets:
 - infly/INF-ORM-Preference-Magnitude-80K
 pipeline_tag: text-classification
 ---
+<div align="center">
+<img src="INF.jpg" width="300"/>
 
+🤗 <a href="https://huggingface.co/infly" target="_blank">Hugging Face</a> 
+<br>
+<br>
+<br>
+</div>
 
 # INF Outcome Reward Model
 ## Introduction
 
-[**INF-ORM-Llama3.1-70B**](https://huggingface.co/Skywork/Skywork-Reward-Gemma-2-27B-v0.2) is the outcome reward model roughly built on the [Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) architecture and trained with the dataset [INF-ORM-Preference-Magnitude-80K](https://huggingface.co/datasets/infly/INF-ORM-Preference-Magnitude-80K).
+[**INF-ORM-Llama3.1-70B**](https://huggingface.co/Skywork/Skywork-Reward-Gemma-2-27B-v0.2) is the outcome reward model roughly built on the [Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) architecture and trained with the dataset [INF-ORM-Preference-Magnitude-80K](https://huggingface.co/datasets/infly/INF-ORM-Preference-Magnitude-80K). 
+
+We did the following three things to improve the performance of our model.
+### Data Pre-processing
+We trained it on the dataset [INF-ORM-Preference-Magnitude-80K](https://huggingface.co/datasets/infly/INF-ORM-Preference-Magnitude-80K), which is derived from the **decontaminated dataset** [Skywork/Skywork-Reward-Perference-80k-v0.2](https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.2).
+
+We use GPT-4o to evaluate the difference between the chosen answer and the rejected answer in the [Skywork/Skywork-Reward-Perference-80k-v0.2](https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.2) and then add the 'Magnitude' column in the dataset.
+
+The evaluation follows the following rules:
+1. If the chosen answer is much better than rejected answer, set 'Magnitude'  value $d$ to 3.
+2. If the chosen answer is better than the rejected answer, set 'Magnitude'  value  $d$ to 2.
+3. If the chosen answer is slightly better than rejected answer, set 'Magnitude'  value  $d$ to 1.
+
+After that, we train our model with the scaled BT loss. The scaled BT loss is defined as:
+$$\mathcal{L}_{Scaled-BT} = -\alpha*d*log(\sigma(r_{\theta}(x, y_{c}))-\sigma(r_{\theta}(x, y_{r})))$$
+where $\alpha$ is the scaling factor. You can find more details about scaled BT loss here [1](https://arxiv.org/pdf/2410.01257). 
+
+> Here we look at the performance gains of scaled BT loss from a different perspective than [1](https://arxiv.org/pdf/2410.01257). The scaled BT loss can be thought of as a form of cross-entropy, where the distribution of the difference of the logits produced by the model is sensitive to the distribution of the magnitude. Therefore, we improve the difference of the values in the 'Magnitude' column from 1, 2, 3 to 1, 3, 10 and finally get better performance.
+
+### Modified Score Head
+We use the modified score head instead of origin score head.
+```python
+        # modified score head
+        self.score = nn.Sequential(
+            nn.Linear(config.hidden_size, config.hidden_size),
+            nn.ReLU(),
+            nn.Linear(config.hidden_size, 1)
+        )
+        # origin score head
+        self.score = nn.linear(config.hidden_size, 1)
+```
+
+### Model Merge
+We trained two models and merge them with the weight $0.5$.
+| Model        | Score | Chat  | Chat Hard | Safety | Reasoning |
+| ----------------- | :---: | :---: | :-------: | :----: | :-------: |
+| INF-ORM-v1   | 94.3  | 96.1  |   88.2    |  94.6  |   98.2    |
+| INF-ORM-v2   | 94.4  | 95.5  |   90.8    |  93  |   99.1    |
+| INF-ORM-v3(Averaged)   | 95.1  | 96.6  |   91.0    |  93.6  |   99.1    |
+
 
-**Note: Train Details are coming soon!**
 
 ## RewardBench Leaderboard
 
@@ -20,7 +65,7 @@ We evaluate our model on [RewardBench](https://huggingface.co/spaces/allenai/rew
 
 | Rank  | Model                                        | Model Type        | Score | Chat  | Chat Hard | Safety | Reasoning |
 | :---: | -------------------------------------------- | ----------------- | :---: | :---: | :-------: | :----: | :-------: |
-|   1   | **infly/INF-ORM-Llama3.1-70B**  | Custom Classifier   | 95.2  | 96.9  |   91.0    |  93.8  |   99.1    |
+|   1   | **infly/INF-ORM-Llama3.1-70B**  | Seq. Classifier   | 95.1  | 96.6  |   91.0    |  93.6  |   99.1    |
 |   2   | Skywork/Skywork-Reward-Gemma-2-27B-v0.2  | Seq. Classifier   | 94.3  | 96.1  |   89.9    |  93.0  |   98.1    |
 |   3   | nvidia/Llama-3.1-Nemotron-70B-Reward         | Custom Classifier | 94.1  | 97.5  |   85.7    |  95.1  |   98.1    |
 |   4   | Skywork/Skywork-Reward-Gemma-2-27B           | Seq. Classifier   | 93.8  | 95.8  |   91.4    |  91.9  |   96.1    |
@@ -135,20 +180,19 @@ print(f"Score for response 1: {score1}")
 print(f"Score for response 2: {score2}")
 
 # Output:
-
 # Score for response 1: 4.96875
 # Score for response 2: 2.890625
 
 ```
 
-## Declaration and License Agreement
+## License Agreement
+INF-ORM-Llama3.1-70B support commercial applications under a permissive [License](https://huggingface.co/infly/INF-ORM-Llama3.1-70B/blob/main/LICENSE).
 
-### Declaration
+## Contact
+If you have any questions, please feel free to reach us at <[email protected]>, <[email protected]> and <[email protected]>.
 
-### License Agreement
+## Acknowledgement 
+This work was done during my internship at INF. I would like to thank my mentor (quchao, tanxiaoyu) and the INF team for their support. Their insights and expertise greatly contributed to the successful completion of this work.
 
-## Contact
-If you have any questions, please feel free to reach us at <[email protected]>.
-## Citation