Update README.md

README.md

@@ -0,0 +1,99 @@
+---
+datasets:
+  - PKU-Alignment/PKU-SafeRLHF
+language:
+  - en
+tags:
+  - reinforcement-learning-from-human-feedback
+  - reinforcement-learning
+  - beaver
+  - safety
+  - llama
+  - ai-safety
+  - deepspeed
+  - rlhf
+  - alpaca
+library_name: safe-rlhf
+---
+
+# 🦫 Beaver's Reward Model
+
+## Model Details
+
+The Beaver reward model is a preference model trained using the [PKU-SafeRLHF](https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF) dataset.
+It can play a role in the safe RLHF algorithm, helping the Beaver model become more helpful.
+
+- **Developed by:** the [PKU-Alignment](https://github.com/PKU-Alignment) Team.
+- **Model Type:** An auto-regressive language model based on the transformer architecture.
+- **License:** Non-commercial license.
+- **Fine-tuned from model:** [LLaMA](https://arxiv.org/abs/2302.13971), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca).
+
+## Model Sources
+
+- **Repository:** <https://github.com/PKU-Alignment/safe-rlhf>
+- **Beaver:** <https://huggingface.co/PKU-Alignment/beaver-7b-v3.0>
+- **Dataset:** <https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF>
+- **Reward Model:** <https://huggingface.co/PKU-Alignment/beaver-7b-unified-reward>
+- **Cost Model:** <https://huggingface.co/PKU-Alignment/beaver-7b-unified-cost>
+- **Dataset Paper:** <https://arxiv.org/abs/2307.04657>
+- **Paper:** <https://arxiv.org/abs/2310.12773>
+
+## How to Use the Reward Model
+
+```python
+import torch
+from transformers import AutoTokenizer
+from safe_rlhf.models import AutoModelForScore
+
+model = AutoModelForScore.from_pretrained('PKU-Alignment/beaver-7b-unified-reward', torch_dtype=torch.bfloat16, device_map='auto')
+tokenizer = AutoTokenizer.from_pretrained('PKU-Alignment/beaver-7b-unified-reward')
+
+input = 'BEGINNING OF CONVERSATION: USER: hello ASSISTANT:Hello! How can I help you today?'
+
+input_ids = tokenizer(input, return_tensors='pt')
+output = model(**input_ids)
+print(output)
+
+# ScoreModelOutput(
+#     scores=tensor([[[-7.2812],
+#          [-0.8203],
+#          [-0.3535],
+#          [-0.5781],
+#          [-0.5781],
+#          [-1.2578],
+#          [-2.9219],
+#          [-2.8594],
+#          [-2.0469],
+#          [-0.8789],
+#          [-1.2422],
+#          [-1.5312],
+#          [-0.7500],
+#          [-1.4688],
+#          [-0.9141],
+#          [-1.0469],
+#          [-1.2266],
+#          [-1.4062],
+#          [-1.4297],
+#          [-1.1016],
+#          [-0.9688],
+#          [ 0.5977],
+#          [ 0.6211],
+#          [ 0.4238],
+#          [ 0.8906],
+#          [ 0.4277],
+#          [ 0.6680],
+#          [ 0.3789]]], grad_fn=<ToCopyBackward0>),
+#     end_scores=tensor([[0.3789]], grad_fn=<ToCopyBackward0>),
+#     last_hidden_state=tensor([[[-0.0552, -0.3203, -0.9180,  ...,  0.1719,  0.1309,  0.2988],
+#          [-1.9609, -0.2617, -0.7227,  ...,  0.3535,  0.8945,  1.6719],
+#          [-1.1016, -0.3984, -0.3398,  ...,  0.5820,  0.9062,  1.6172],
+#          ...,
+#          [-0.4844,  0.1387, -0.6562,  ...,  0.3789,  0.2910,  1.5625],
+#          [-0.3125,  0.0811, -0.7969,  ...,  0.4688,  0.2344,  1.4453],
+#          [-0.7148, -0.2139, -0.4336,  ...,  0.9219, -0.1050,  1.3594]]],
+#        dtype=torch.bfloat16, grad_fn=<ToCopyBackward0>),
+#     end_last_hidden_state=tensor([[-0.7148, -0.2139, -0.4336,  ...,  0.9219, -0.1050,  1.3594]],
+#        dtype=torch.bfloat16, grad_fn=<ToCopyBackward0>),
+#     end_index=tensor([27])
+# )
+```