Upload README.md

README.md

@@ -10,19 +10,20 @@ model_type: llama
 
 # Swallow
 
-Our Swallow model has undergone continuous pre-training from the [Llama 2 family](https://huggingface.co/meta-llama), primarily with the addition of Japanese language data. The tuned versions use supervised fine-tuning (SFT). 
+Our Swallow model has undergone continual pre-training from the [Llama 2 family](https://huggingface.co/meta-llama), primarily with the addition of Japanese language data. The tuned versions use supervised fine-tuning (SFT). 
 Links to other models can be found in the index.
 
 # Model Release Updates
 
 We are excited to share the release schedule for our latest models:
-- **April 25, 2024**: Released version 1.0 of our enhanced instruction-tuned models: [Swallow-7b-instruct-v1.0](https://huggingface.co/tokyotech-llm/Swallow-7b-instruct-v1.0), [Swallow-13b-instruct-v1.0](https://huggingface.co/tokyotech-llm/Swallow-13b-instruct-v1.0), and [Swallow-70b-instruct-v1.0](https://huggingface.co/tokyotech-llm/Swallow-70b-instruct-v1.0). 
+- **April 26, 2024**: Released version 0.1 of our enhanced instruction-tuned models: [Swallow-7b-instruct-v0.1](https://huggingface.co/tokyotech-llm/Swallow-7b-instruct-v0.1), [Swallow-13b-instruct-v0.1](https://huggingface.co/tokyotech-llm/Swallow-13b-instruct-v0.1), and [Swallow-70b-instruct-v0.1](https://huggingface.co/tokyotech-llm/Swallow-70b-instruct-v0.1) as preview versions.
 - **March 2, 2024**: Released the [Swallow-7b-plus-hf](https://huggingface.co/tokyotech-llm/Swallow-7b-plus-hf), a model trained with approximately twice as many Japanese tokens as [Swallow-7b-hf](https://huggingface.co/tokyotech-llm/Swallow-7b-hf).
 - **February 4, 2024**: Released the [Swallow-13b-NVE-hf](https://huggingface.co/tokyotech-llm/Swallow-13b-NVE-hf).
 - **January 26, 2024**: Released the [Swallow-7b-NVE-hf](https://huggingface.co/tokyotech-llm/Swallow-7b-NVE-hf), [Swallow-7b-NVE-instruct-hf](https://huggingface.co/tokyotech-llm/Swallow-7b-NVE-instruct-hf), [Swallow-70b-NVE-hf](https://huggingface.co/tokyotech-llm/Swallow-70b-NVE-hf), and [Swallow-70b-NVE-instruct-hf](https://huggingface.co/tokyotech-llm/Swallow-70b-NVE-instruct-hf)
 - **December 19, 2024**: Released the [Swallow-7b-hf](https://huggingface.co/tokyotech-llm/Swallow-7b-hf), [Swallow-7b-instruct-hf](https://huggingface.co/tokyotech-llm/Swallow-7b-instruct-hf), [Swallow-13b-hf](https://huggingface.co/tokyotech-llm/Swallow-13b-hf), [Swallow-13b-instruct-hf](https://huggingface.co/tokyotech-llm/Swallow-13b-instruct-hf), [Swallow-70b-hf](https://huggingface.co/tokyotech-llm/Swallow-70b-hf), and [Swallow-70b-instruct-hf](https://huggingface.co/tokyotech-llm/Swallow-70b-instruct-hf).
+
 ## Swallow Model Index
-|Model|Swallow-hf|Swallow-instruct-hf|Swallow-instruct-v1.0|
+|Model|Swallow-hf|Swallow-instruct-hf|Swallow-instruct-v0.1|
 |---|---|---|---|
 |7B| [Link](https://huggingface.co/tokyotech-llm/Swallow-7b-hf) | [Link](https://huggingface.co/tokyotech-llm/Swallow-7b-instruct-hf)|[Link](https://huggingface.co/tokyotech-llm/Swallow-7b-instruct-v1.0)|
 |7B-Plus| [Link](https://huggingface.co/tokyotech-llm/Swallow-7b-plus-hf) | N/A | N/A |
@@ -39,13 +40,11 @@ We are excited to share the release schedule for our latest models:
 ![logo](./logo.png)
 
 This repository provides large language models developed by [TokyoTech-LLM](https://tokyotech-llm.github.io/).
-Read our [blog post](https://zenn.dev/tokyotech_lm/articles/d6cb3a8fdfc907) or our [paper](https://www.anlp.jp/proceedings/annual_meeting/2024/pdf_dir/A8-5.pdf)
 
 ## Model Details
 
 * **Model type**: Please refer to LLaMA-2 technical report for details on the model architecture. 
 * **Language(s)**: Japanese English
-* **Library**: [Megatron-LM](https://github.com/rioyokotalab/Megatron-Llama2) 
 * **Tokenizer**: This model employs a tokenizer that features a broadened vocabulary based on Japanese data. This allows for a more efficient representation of text using fewer tokens, leading to a notably faster inference process.
 * **Contact**: swallow[at]nlp.c.titech.ac.jp 
 
@@ -53,66 +52,29 @@ Read our [blog post](https://zenn.dev/tokyotech_lm/articles/d6cb3a8fdfc907) or o
 
 ### MT-Bench JA
 
-TODO
-
-## Base Model Performance
-
-### Japanese tasks
+* NOTE that the models with the `v0.1` suffix are newer versions compared to their original counterparts with the `hf`.
+* We will update the score of `Swallow-70b-instruct-hf` soon.
 
-|Model|Size|JCommonsenseQA|JEMHopQA|NIILC|JSQuAD|XL-Sum|MGSM|WMT20-en-ja|WMT20-ja-en|
+|Model|Average|Writing|Roleplay|Reasoning|Math|Coding|Extraction|STEM|Humanities|
 |---|---|---|---|---|---|---|---|---|---|
-|   |   |4-shot|4-shot|4-shot|4-shot|1-shot|4-shot|4-shot|4-shot|
-| Llama 2 | 7B | 0.3852 | 0.4240 | 0.3410 | 0.7917 | 0.1905 | 0.0760 | 0.1783 | 0.1738 |
-| Swallow | 7B | 0.4808 | 0.5078 | 0.5968 | 0.8573 | 0.1830 | 0.1240 | 0.2510 | 0.1511 |
-| Swallow-Plus | 7B | 0.5478 | 0.5493 | 0.6030 | 0.8544 | 0.1806 | 0.1360 | 0.2568 | 0.1441 |
-| Swallow-NVE | 7B | 0.5433 | 0.5425 | 0.5729 | 0.8684 | 0.2117 | 0.1200 | 0.2405 | 0.1512 |
-| Llama 2 | 13B | 0.6997 | 0.4415 | 0.4170 | 0.8533 | 0.2139 | 0.1320 | 0.2146 | 0.1982 |
-| Swallow | 13B | 0.7837 | 0.5063 | 0.6398 | 0.9005 | 0.2168 | 0.2040 | 0.2720 | 0.1771 |
-| Swallow-NVE | 13B | 0.7712 | 0.5438 | 0.6351 | 0.9030 | 0.2294 | 0.2120 | 0.2735 | 0.1817 |
-| Llama 2 | 70B | 0.8686 | 0.4656 | 0.5256 | 0.9080 | 0.2361 | 0.3560 | 0.2643 | **0.2398** |
-| Swallow | 70B | 0.9348 | **0.6290** | 0.6960 | 0.9176 | 0.2266 | **0.4840** | **0.3043** | 0.2298 |
-| Swallow-NVE | 70B | **0.9410** | 0.5759 | **0.7024** | **0.9254** | **0.2758** | 0.4720 | 0.3042 | 0.2322 |
-### English tasks
-
-|Model|Size|OpenBookQA|TriviaQA|HellaSwag|SQuAD2.0|XWINO|GSM8K|
-|---|---|---|---|---|---|---|---|
-|   |   |8-shot|8-shot|8-shot|8-shot|8-shot|8-shot|
-| Llama 2 | 7B    | 0.3580     | 0.6265   | 0.5860    | 0.3207   | 0.9049 | 0.1410 |
-| Swallow | 7B    | 0.3180     | 0.4836   | 0.5308    | 0.3125   | 0.8817 | 0.1130 |
-| Swallow-Plus | 7B | 0.3280     | 0.4558   | 0.5259    | 0.3134   | 0.8929 | 0.1061 |
-| Swallow-NVE | 7B | 0.3180     | 0.5079   | 0.5329    | 0.2919   | 0.8817 | 0.0986 |
-| Llama 2 | 13B   | 0.3760     | 0.7255   | 0.6148    | 0.3681   | 0.9140 | 0.2403 |
-| Swallow | 13B   | 0.3500     | 0.5852   | 0.5660    | 0.3406   | 0.9075 | 0.2039 |
-| Swallow-NVE | 13B | 0.3460     | 0.6025   | 0.5700    | 0.3478   | 0.9006 | 0.1751 |
-| Llama 2 | 70B   | **0.4280** | **0.8239** | **0.6742** | **0.3770** | **0.9290** | **0.5284** |
-| Swallow | 70B   | 0.4220     | 0.7756   | 0.6458    | 0.3745   | 0.9204 | 0.4867 |
-| Swallow-NVE | 70B | 0.4240     | 0.7817   | 0.6439    | 0.3451   | 0.9256 | 0.4943 |
+| Swallow-7b-instruct-v0.1 |0.3435|0.4450|0.4720|0.1853|0.1920|0.2204|0.3015|0.4594|0.4720|
+| Swallow-7b-instruct-hf |0.1833|0.2205|0.1975|0.1593|0.1045|0.1282|0.2672|0.1908|0.1980|
+| Swallow-13b-instruct-v0.1 |0.3669|0.4816|0.5562|0.2769|0.1020|0.1505|0.4179|0.4347|0.5150|
+| Swallow-13b-instruct-hf |0.2004|0.1932|0.2552|0.1507|0.1184|0.1285|0.2641|0.2434|0.2500|
+| Swallow-70b-instruct-v0.1 |0.4513|0.4822|0.5353|0.3497|0.3492|0.2668|0.5553|0.4955|0.5767|
+| Swallow-70b-instruct-hf |N/A|N/A|N/A|N/A|N/A|N/A|N/A|N/A|N/A|
 
 ## Evaluation Benchmarks
 
-### Japanese evaluation benchmarks
-
-We used llm-jp-eval(v1.0.0) and JP Language Model Evaluation Harness(commit #9b42d41). The details are as follows:
-
-- Multiple-choice question answering (JCommonsenseQA [Kurihara+, 2022])
-- Open-ended question answering (JEMHopQA [Ishii+, 2023])
-- Open-ended question answering (NIILC [Sekine, 2003])
-- Machine reading comprehension (JSQuAD [Kurihara+, 2022])
-- Automatic summarization (XL-Sum [Hasan+, 2021])
-- Machine translation (WMT2020 ja-en [Barrault+, 2020])
-- Machine translation (WMT2020 en-ja [Barrault+, 2020])
-- Mathematical reasoning (MGSM [Shi+, 2023])
-
-### English evaluation benchmarks
+### MT-Bench JA
 
-We used the Language Model Evaluation Harness(v.0.3.0). The details are as follows:
+We used [Japanese MT-Bench](https://wandb.ai/wandb-japan/llm-leaderboard/artifacts/dataset/mtbench_ja_question).
+We utilized the following artifacts:
 
-- Multiple-choice question answering (OpenBookQA [Mihaylov+, 2018])
-- Open-ended question answering (TriviaQA [Joshi+, 2017])
-- Machine reading comprehension (SQuAD 2.0 [Rajpurkar+, 2018])
-- Commonsense reasoning (XWINO [Tikhonov & Ryabinin, 2021])
-- Natural language inference (HellaSwag [Zellers+, 2019])
-- Mathematical reasoning (GSM8k [Cobbe+, 2021])
+- Implemantation: FastChat [Zheng+, 2023] (commit #e86e70d0)
+- Question: [Nejumi LLM-Leaderboard NEO, mtbench_ja_question_v3](https://wandb.ai/wandb-japan/llm-leaderboard/artifacts/dataset/mtbench_ja_question/v3)
+- Reference Answer: [Nejumi LLM-Leaderboard NEO, mtbench_ja_referenceanswer_v1](https://wandb.ai/wandb-japan/llm-leaderboard/artifacts/dataset/mtbench_ja_referenceanswer/v1)
+- Prompt for Judge: [Nejumi LLM-Lederboard NEO, mtbench_ja_prompt_v1](https://wandb.ai/wandb-japan/llm-leaderboard/artifacts/dataset/mtbench_ja_prompt/v1)
 
 
 ## Usage
@@ -123,7 +85,7 @@ First install additional dependencies in [requirements.txt](./requirements.txt):
 pip install -r requirements.txt
 ```
 
-### Instruction format Ver1.0
+### Instruction format Ver0.1
 This format must be adhered to strictly, as deviations may result in less optimal outputs from the model.
 
 The template used to construct a prompt for the Instruct model is specified as follows:
@@ -132,15 +94,18 @@ The template used to construct a prompt for the Instruct model is specified as f
 <s>[INST] <<SYS>>\n{Instruction}\n<</SYS>>\n\n{USER_MESSAGE_1} [INST] {BOT_MESSAGE_1} </s>[INST] {USER_MESSAGE_2}[/INST] 
 ```
 
-Please be aware that  ``<s> `` and  ``</s> `` are special tokens used for the beginning of string (BOS) and end of string (EOS), respectively, while [INST] and [/INST] are considered regular strings.
+Please be aware that ``<s>`` and ``</s>`` are special tokens used for the beginning of string (BOS) and end of string (EOS), respectively, while [INST] and [/INST] are considered regular strings.
+
+For the "{Instruction}" part, We recommend using "あなたは誠実で優秀な日本人のアシスタ��トです。"
 
-### Use the instruct model Ver1.0
+
+### Use the instruct model Ver0.1
 
 ```python
 import torch
 from transformers import AutoTokenizer, AutoModelForCausalLM
 
-model_name = "tokyotech-llm/Swallow-70b-instruct-v1.0"
+model_name = "tokyotech-llm/Swallow-70b-instruct-v0.1"
 model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
 tokenizer = AutoTokenizer.from_pretrained(model_name)
 
@@ -161,135 +126,14 @@ decoded = tokenizer.batch_decode(generated_ids)
 print(decoded[0])
 ```
 
-### Use the instruct model
-
-**Note:** Please be aware that the inference example is based on a model version older than 1.0.
-
-
-```python
-import torch
-from transformers import AutoTokenizer, AutoModelForCausalLM
-
-model_name = "tokyotech-llm/Swallow-7b-instruct-hf"
-
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, device_map="auto")
-
-
-PROMPT_DICT = {
-    "prompt_input": (
-        "以下に、あるタスクを説明する指示があり、それに付随する入力が更なる文脈を提供しています。"
-        "リクエストを適切に完了するための回答を記述してください。\n\n"
-        "### 指示:\n{instruction}\n\n### 入力:\n{input}\n\n### 応答:"
-
-    ),
-    "prompt_no_input": (
-        "以下に、あるタスクを説明する指示があります。"
-        "リクエストを適切に完了するための回答を記述してください。\n\n"
-        "### 指示:\n{instruction}\n\n### 応答:"
-    ),
-}
-
-def create_prompt(instruction, input=None):
-    """
-    Generates a prompt based on the given instruction and an optional input.
-    If input is provided, it uses the 'prompt_input' template from PROMPT_DICT.
-    If no input is provided, it uses the 'prompt_no_input' template.
-
-    Args:
-        instruction (str): The instruction describing the task.
-        input (str, optional): Additional input providing context for the task. Default is None.
-
-    Returns:
-        str: The generated prompt.
-    """
-    if input:
-        # Use the 'prompt_input' template when additional input is provided
-        return PROMPT_DICT["prompt_input"].format(instruction=instruction, input=input)
-    else:
-        # Use the 'prompt_no_input' template when no additional input is provided
-        return PROMPT_DICT["prompt_no_input"].format(instruction=instruction)
-
-# Example usage
-instruction_example = "以下のトピックに関する詳細な情報を提供してください。"
-input_example = "東京工業大学の主なキャンパスについて教えてください"
-prompt = create_prompt(instruction_example, input_example)
-
-input_ids = tokenizer.encode(
-    prompt,
-    add_special_tokens=False,
-    return_tensors="pt"
-)
-
-tokens = model.generate(
-    input_ids.to(device=model.device),
-    max_new_tokens=128,
-    temperature=0.99,
-    top_p=0.95,
-    do_sample=True,
-)
-
-out = tokenizer.decode(tokens[0], skip_special_tokens=True)
-print(out)
-
-```
-
-### Use the base model
-
-```python
-import torch
-from transformers import AutoTokenizer, AutoModelForCausalLM
-
-model_name = "tokyotech-llm/Swallow-7b-hf"
-
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
-
-prompt = "東京工業大学の主なキャンパスは、"
-input_ids = tokenizer.encode(
-    prompt,
-    add_special_tokens=False,
-    return_tensors="pt"
-)
-tokens = model.generate(
-    input_ids.to(device=model.device),
-    max_new_tokens=128,
-    temperature=0.99,
-    top_p=0.95,
-    do_sample=True,
-)
-
-out = tokenizer.decode(tokens[0], skip_special_tokens=True)
-print(out)
-```
-
 ## Training Datasets
 
-### Continual Pre-Training
-The following datasets were used for continual pre-training.
-
-- [Japanese Wikipedia](https://dumps.wikimedia.org/other/cirrussearch)
-- [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)
-- Swallow Corpus
-- [The Pile](https://huggingface.co/datasets/EleutherAI/pile)
-
-
-### Instruction Tuning
-
-#### Ver1.0
+### Instruction Tuning Ver0.1
 
 The following datasets were used for the instruction tuning. 
 
 - [OpenAssistant Conversations Dataset EN top-1 thread](https://huggingface.co/datasets/OpenAssistant/oasst2)
 - [OpenAssistant Conversations Dataset](https://huggingface.co/datasets/llm-jp/oasst1-21k-ja) was used, where human utterances are included but the responses are not used. Instead, the responses were generated using the [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/datasets/llm-jp/oasst1-21k-jahttps://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) model.
-
-#### Old
-
-The following datasets were used for the instruction tuning. 
-
-- [Anthropic HH-RLHF](https://huggingface.co/datasets/kunishou/hh-rlhf-49k-ja)
-- [Databricks Dolly 15-k](https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja)
-- [OpenAssistant Conversations Dataset](https://huggingface.co/datasets/kunishou/oasst1-89k-ja)
  
 ## Risks and Limitations