Upload folder using huggingface_hub

README.md

@@ -0,0 +1,119 @@
+---
+license: apache-2.0
+datasets:
+- PleIAs/common_corpus
+language:
+- en
+- fr
+- de
+- es
+- it
+- nl
+- la
+- pt
+---
+
+<div style="text-align: center;">
+  <img src="https://raw.githubusercontent.com/Pleias/logos/d6152d7943905da32a1e04fdfd7708ed9c7eed5e/PleIAs%201_0%20Full%20Logo%20(Black).png" style="width: 80%; margin: 0 auto; display: inline-block;"/>
+</div>
+
+**Pleias-nano-1.2b-RAG 0.1** is a specialized language model designed by [Pleias](https://huggingface.co/PleIAs) and trained with [Tracto AI](https://tracto.ai/) for Retrieval-Augmented Generation.
+
+Similarly to its base model, [Pleias-nano-1.2b-Preview](https://huggingface.co/PleIAs/Pleias-nano-1.2b-Preview), Pleias-nano-1.2b-RAG 0.1 aims to be a fully open model (weights, code, data), only trained on content with a permissible license and fully compliant with the upcoming European AI Act.
+
+## Description
+Pleias-nano-1.2b-RAG is continuous pretraining of Pleias-nano-1.2b-Preview on a new dataset of 45,088,768,000 tokens modeling common retrieval tasks. All the content of the dataset is ultimately coming from Common Corpus.
+
+Pleias-nano-1.2b-RAG includes the main features of the original base model:
+* Only trained on open data under a permissible license and in compliance with the European AI Act. By design, all Pleias model are unable to output copyrighted content.
+* Extensive multilingual support for main European languages: English, French, German, Spanish, Italian, Dutch, Latin, Portuguese and Polish.
+* Extremely low level of toxicity and problematic content.
+
+Pleias-nano-1.2b-RAG supports retrieval-augmented generation with enhanced verifiability, source analysis and grounding on submitted sources. This includes:
+* Standardized structure and special tokens to include queries, sources, .
+* Anticipation of various query forms in multiple languages, from actual drafted questions to unstructured list of keyword search.
+* Source analysis/criticism which also acts as an integrated reranker step.
+* Generation of ground answers with references and excerpts linked to the original sources.
+
+While the base model Pleias-nano-1.2b-RAG has been made available as an experimental preview, we release Pleias-nano-1.2b-RAG 0.1 as an early version. Pleias-nano-1.2b-RAG 0.1 has been already tested and integrated into multiple applied RAG projects, including Pleias flagship application Scholasticai.
+
+## Training
+Pleias-nano-1.2b-RAG was trained pretrained on TractoAI on ISEG GPU cluster by Nebius AI on the fork Nanotron developed by TractoAI. We provide the complete settings as a yaml file as part of our release.
+
+Pleias-nano-1.2b-RAG derives from the last checkpoint of Pleias-nano-1.2b-Preview (369,000). The training schedule reused the last learning rate value (5e-6) without decay for 43,000 steps. Each step is about 10 time smaller than the original steps from the base model training (roughly 1M tokens per step vs. 12M tokens)
+
+Training covers the entire RAG dataset we have been designing based on [Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus) for 3 epochs.
+
+Further experiments were made with different learning rate values: none of theses tests have provided a better convergence than the one obtained with the final learning rate from the base model.
+
+## Inference
+Pleias-nano-1.2b-RAG relies on special tokens to encode the core RAG functionalities:
+
+A typical example, with excerpts drawn from a Wikipedia article on Wikipedia
+```bash
+<|query_start|>Is Wikipedia reliable?<|query_end|>
+<|source_start|><|source_id_start|>ebea70a3502acfbd<|source_id_end|>Articles for traditional encyclopedias such as Encyclopædia Britannica are written by experts, lending such encyclopedias a reputation for accuracy.[144] However, a peer review in 2005 of forty-two scientific entries on both Wikipedia and Encyclopædia Britannica by the science journal Nature found few differences in accuracy, and concluded that "the average science entry in Wikipedia contained around four inaccuracies; Britannica, about three."[145] Joseph Reagle suggested that while the study reflects "a topical strength of Wikipedia contributors" in science articles, "Wikipedia may not have fared so well using a random sampling of articles or on humanities subjects."<|source_end|>
+<|source_start|><|source_id_start|>5f862e733d38288e<|source_id_end|>As a consequence of the open structure, Wikipedia "makes no guarantee of validity" of its content, since no one is ultimately responsible for any claims appearing in it.[W 54] Concerns have been raised by PC World in 2009 regarding the lack of accountability that results from users' anonymity, the insertion of false information,[152] vandalism, and similar problems. Legal Research in a Nutshell (2011), cites Wikipedia as a "general source" that "can be a real boon" in "coming up to speed in the law governing a situation" and, "while not authoritative, can provide basic facts as well as leads to more in-depth resources".<|source_end|>
+<|source_start|><|source_id_start|>354fa4908152b336<|source_id_end|>Wikipedia's open structure inherently makes it an easy target for Internet trolls, spammers, and various forms of paid advocacy seen as counterproductive to the maintenance of a neutral and verifiable online encyclopedia.[70][W 55] In response to paid advocacy editing and undisclosed editing issues, Wikipedia was reported in an article in The Wall Street Journal to have strengthened its rules and laws against undisclosed editing.[162] The article stated that: "Beginning Monday [from the date of the article, June 16, 2014], changes in Wikipedia's terms of use will require anyone paid to edit articles to disclose that arrangement. Katherine Maher, the nonprofit Wikimedia Foundation's chief communications officer, said the changes address a sentiment among volunteer editors that 'we're not an advertising service; we're an encyclopedia.'"<|source_end|>
+<|source_analysis_start|>
+```
+
+As a specialized language model, PleIAs-1.2b-RAG will be unable to work properly with prompts that detracts from that design.
+
+### RAG Evaluation
+
+We evaluate Pico and Nano models on a RAG task. As existing benchmarks are largely limited to English, we develop a custom multilingual RAG benchmark. We synthetically generate queries and small sets of documents. To evaluate, we prompted models with the query and documents. We then ran a head-to-head ELO-based tournament with GPT-4o as judge. We [release the prompts and generations for all models we compared](https://huggingface.co/datasets/PleIAs/Pleias-1.0-eval/tree/main/RAGarena). Our nano (1.2B) model outperforms Llama 3.2 1.1B and EuroLLM 1.7B. Our pico (350M) model outperforms other models in its weight class, such as SmolLM 360M and Qwen2.5 500M, in addition to much larger models, such as Llama 3.2 1.1B and EuroLLM 1.7B.
+  
+| **Rank** | **Model**                | **ELO**    |
+|----------|--------------------------|------------|
+| 1        | Qwen2.5-Instruct-7B      | 1294.6     |
+| 2        | Llama-3.2-Instruct-8B    | 1269.8     |
+| 3        | **Pleias-nano-1.2B-RAG**   | **1137.5** |
+| 4        | Llama-3.2-Instruct-3B    | 1118.1     |
+| 5        | Qwen2.5-Instruct-3B      | 1078.1     |
+| 6        | **Pleias-pico-350M-RAG** | **1051.2** |
+| 7        | Llama-3.2-1B-Instruct    | 872.3      |
+| 8        | EuroLLM-1.7B-Instruct    | 860.0      |
+| 9        | SmolLM-360M-Instruct     | 728.6      |
+| 10       | Qwen2.5-0.5B-Instruct    | 722.2      |
+| 11       | SmolLM-1.7B-Instruct     | 706.3      |
+
+## Acceptable use
+Pleias-nano-1.2b-RAG includes a much wider range of support for verifiability and grounding than most generalist models. 
+
+The model is not a substitute for an integrated RAG application. Retrieval errors as well as challenging texts and questions can still create a range of issues. We especially encourage end users to take advantage of the citations and the references to provide better indicators of accuracy.
+
+For best results we recommend the following setting: 
+* Deterministic generation (temp = 0) and no repetition penalty (which is unsurprisingly detrimental to the accuracy of citations).
+* Standardized hashes of 16 characters. While the model has been trained on many other patterns (including full bibliographic entries), this has proven the most convenient for systematic citation parsing.
+
+Training Greenhouse Gas Emissions: Estimated total location-based greenhouse gas emissions were 4 tons CO2eq for training. 
+
+## Ethical Considerations
+
+pleias-nano-1.2b-RAG model, like all large language models, carries inherent ethical risks that require careful consideration. Our approach to mitigating these risks begins at the data level, where we exclusively use vetted sources, deliberately excluding CommonCrawl. The primary challenge comes from our public domain dataset component, which contains historical texts that may reflect outdated social norms and potentially harmful language, particularly regarding minoritized groups.
+
+To address this, we implemented a systematic ethical filtering process using toxicity classifiers to identify extremely harmful content. We also employed synthetic rewriting techniques to transform mildly problematic passages while preserving the underlying informational value. This process significantly reduced potential societal harm without compromising the dataset's size or textual quality, resulting in notably low toxicity scores in benchmarks compared to other models.
+
+Despite these preventive measures, users should be aware that the model has not undergone additional safety alignment procedures and may still produce problematic outputs. The model's capabilities in generative AI tasks must be balanced against the risks of bias, misinformation propagation, and autonomous decision-making challenges. We explicitly prohibit any malicious utilization and emphasize the responsibility of users to implement appropriate safeguards.
+
+At Pleias, we continue to research and develop improved methods for creating safer and more equitable models and datasets. This includes ongoing work in toxicity reduction, bias mitigation, and the development of more sophisticated ethical filtering techniques.
+
+## Acknowledgements
+
+This work would not have been possible without the substantial support and technical expertise from TractoAI, a serverless AI platform for running data and compute-intensive workloads at scale.
+
+We are deeply grateful to the Mozilla Foundation Local AI Program for their generous support. 
+
+Finally, we acknowledge the significant contributions from the open science LLM community, particularly HuggingFace, Eleuther AI and Allen AI whose insights and cooperation have been invaluable to our work.
+
+## Future updates
+Pleias-nano-1.2b-RAG will be continuously improved through iterative retraining/adaptation.
+
+The current roadmap includes the following features:
+* Context length expansion.
+* Better handling of multilingual sources. In its current form, Pleias-nano-1.2b-RAG will generally switch language if a query is made to sources in a different language.
+* New sampling methods inspired by Entropix for a better combined support of text creativity and accuracy.
+* Interactive/conversational RAG.
+
+End users are encouraged to update to the latest version whenever possible.

config.json

@@ -6,6 +6,7 @@
   "attention_dropout": 0.0,
   "bos_token_id": 1,
   "eos_token_id": 2,
+  "head_dim": 64,
   "hidden_act": "silu",
   "hidden_size": 2048,
   "initializer_range": 0.02,
@@ -22,7 +23,7 @@
   "rope_theta": 10000,
   "tie_word_embeddings": true,
   "torch_dtype": "bfloat16",
-  "transformers_version": "4.44.2",
+  "transformers_version": "4.46.0",
   "use_cache": true,
   "vocab_size": 65536
 }

generation_config.json

@@ -2,5 +2,5 @@
   "_from_model_config": true,
   "bos_token_id": 1,
   "eos_token_id": 2,
-  "transformers_version": "4.44.2"
+  "transformers_version": "4.46.0"
 }

pleias_1b_base/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

pleias_1b_base/README.md

@@ -0,0 +1,77 @@
+---
+license: apache-2.0
+datasets:
+- PleIAs/common_corpus
+language:
+- en
+- fr
+- es
+- de
+- it
+- la
+- nl
+- pl
+---
+<div style="text-align: center;">
+  <img src="https://raw.githubusercontent.com/Pleias/logos/d6152d7943905da32a1e04fdfd7708ed9c7eed5e/PleIAs%201_0%20Full%20Logo%20(Black).png" style="width: 80%; margin: 0 auto; display: inline-block;"/>
+</div>
+
+**Pleias-nano-1.2b-Preview** is an early preview of a 1.21 billion parameters base model trained by [Pleias](https://huggingface.co/PleIAs) with [Tracto AI](https://tracto.ai/) on [Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus).
+
+Like all the base and specialized models from Pleias, Pleias-nano-1.2b-Preview has only been trained on open data out of copyright (public domain) or under a permissible license.
+
+## Description
+Pleias-nano-1.2b-Preview is a transformer base model, entirely pretrained from scratch, using an architecture similar to Llama/GPT-Neox for easier deployment/inference.
+
+It includes the following features, that would apply to any responsibly trained variant:
+* Only trained on open data under a permissible license and in compliance with the European AI Act. By design, all Pleias model are unable to output copyrighted content.
+* Extensive multilingual support for main European languages.
+* A new tokenizer designed for enhanced document processing tasks and better multilingual support.
+* Extremely low level of toxicity and problematic content.
+
+Pleias-nano-1.2b-Preview has demonstrated unusual abilities for multilingual generation in its size range. Fully supported languages include English, French, Spanish, German, Italian, Dutch, Latin and Portuguese. 
+
+Given its size, Pleias-nano-1.2b-Preview can run on CPU without any compression loss. We provide a first GGUF variant as part of our release.
+
+## Recommended use
+As a base model, Pleias-nano-1.2b-Preview is only able to run continuation prompts.
+
+Text generation is currently able to support a range of creative writing tasks in multiple European languages. For more consistent results we recommend using a low or null temperature with a slight repetition penalty (1.2).
+
+Pleias-nano-1.2b-Preview has been successfully adapted for continuous pretraining and full-fine-tuning on document processing tasks such as RAG, translation or OCR correction. Given the small size of the model we do not recommend fine-tuning methods based on LORA.
+
+## Example
+
+
+## Training
+Pleias-nano-1.2b-Preview was fully pretrained on TractoAI on ISEG GPU cluster by Nebius AI on 192 h100s for 5 days. Pretraining code relied on [the fork of Nanotron developed by TractoAI](https://github.com/tractoai/nanotron). We provide the complete settings as a yaml file as part of our release. 
+
+Training schedule includes 518,000 steps (batch size 1,024) on over three epochs (nearly 5 trillions tokens):
+* A lightly filtered version of Common Corpus (1.6 trillion tokens)
+* A filtered and enhanced version of Common Corpus (1,086,324,736,000 tokens).
+* A repeat of the previous set.
+  
+Training Greenhouse Gas Emissions: Estimated total location-based greenhouse gas emissions were 4 tons CO2eq for training. 
+
+## Ethical Considerations
+
+pleias-1.B-Base model, like all large language models, carries inherent ethical risks that require careful consideration. Our approach to mitigating these risks begins at the data level, where we exclusively use vetted sources, deliberately excluding CommonCrawl. The primary challenge comes from our public domain dataset component, which contains historical texts that may reflect outdated social norms and potentially harmful language, particularly regarding minoritized groups.
+
+To address this, we implemented a systematic ethical filtering process using toxicity classifiers to identify extremely harmful content. We also employed synthetic rewriting techniques to transform mildly problematic passages while preserving the underlying informational value. This process significantly reduced potential societal harm without compromising the dataset's size or textual quality, resulting in notably low toxicity scores in benchmarks compared to other models.
+
+Despite these preventive measures, users should be aware that the model has not undergone additional safety alignment procedures and may still produce problematic outputs. The model's capabilities in generative AI tasks must be balanced against the risks of bias, misinformation propagation, and autonomous decision-making challenges. We explicitly prohibit any malicious utilization and emphasize the responsibility of users to implement appropriate safeguards.
+
+At Pleias, we continue to research and develop improved methods for creating safer and more equitable models and datasets. This includes ongoing work in toxicity reduction, bias mitigation, and the development of more sophisticated ethical filtering techniques.
+
+## Acknowledgements
+
+This work would not have been possible without the substantial support and technical expertise from TractoAI, a serverless AI platform for running data and compute-intensive workloads at scale.
+
+We are deeply grateful to the Mozilla Foundation Local AI Program for their generous support. 
+
+Finally, we acknowledge the significant contributions from the open science LLM community, particularly HuggingFace, Eleuther AI and Allen AI whose insights and cooperation have been invaluable to our work.
+
+## Update
+Pleias-1.2b-Preview is currently released as an early preview.
+
+The model will undergo several more round of post-training to enhance reasoning capacities and fine-tunability as well as in anticipation of a generalist instruct version.

pleias_1b_base/config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 6144,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 22,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.46.0",
+  "use_cache": true,
+  "vocab_size": 65536
+}

pleias_1b_base/generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.46.0"
+}

pleias_1b_base/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20211134a3a0ffb8efbb3c50443dc57f08390d7a4b1957f5d95896549333c2c4
+size 2390960616

pleias_1b_base/special_tokens_map.json

@@ -0,0 +1 @@
+{}

pleias_1b_base/tokenizer_config.json

@@ -0,0 +1,39 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|begin_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|end_of_text|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}

special_tokens_map.json

@@ -1,15 +1,30 @@
 {
   "additional_special_tokens": [
-    "<|source_id|>",
+    "<|tool_call_start|>",
+    "<|tool_call_end|>",
+    "<|tool_list_start|>",
+    "<|tool_list_end|>",
     "<|source_analysis_start|>",
     "<|source_analysis_end|>",
     "<|source_start|>",
     "<|source_end|>",
+    "<|im_start|>",
+    "<|im_end|>",
     "<|answer_start|>",
     "<|answer_end|>",
-    "<|query_start|>",
-    "<|query_end|>",
+    "<|text_start|>",
+    "<|text_end|>",
+    "<|translation_start|>",
+    "<|translation_end|>",
+    "<|back_translation_start|>",
+    "<|back_translation_end|>",
+    "<|ocr_correction_start|>",
+    "<|ocr_correction_end|>",
+    "<|json_scheme_start|>",
+    "<|json_scheme_end|>",
     "<|source_interpretation_start|>",
-    "<|source_interpretation_end|>"
+    "<|source_interpretation_end|>",
+    "<|query_start|>",
+    "<|query_end|>"
   ]
-}
+}

tokenizer_config.json

@@ -32,8 +32,88 @@
       "single_word": false,
       "special": true
     },
+    "65510": {
+      "content": "<|tool_call_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65511": {
+      "content": "<|tool_call_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65512": {
+      "content": "<|tool_list_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65513": {
+      "content": "<|tool_list_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65514": {
+      "content": "<|source_analysis_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65515": {
+      "content": "<|source_analysis_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65516": {
+      "content": "<|source_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65517": {
+      "content": "<|source_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65518": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65519": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
     "65520": {
-      "content": "<|source_id|>",
+      "content": "<|answer_start|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -41,7 +121,7 @@
       "special": true
     },
     "65521": {
-      "content": "<|source_analysis_start|>",
+      "content": "<|answer_end|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -49,7 +129,7 @@
       "special": true
     },
     "65522": {
-      "content": "<|source_analysis_end|>",
+      "content": "<|text_start|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -57,7 +137,7 @@
       "special": true
     },
     "65523": {
-      "content": "<|source_start|>",
+      "content": "<|text_end|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -65,7 +145,7 @@
       "special": true
     },
     "65524": {
-      "content": "<|source_end|>",
+      "content": "<|translation_start|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -73,7 +153,7 @@
       "special": true
     },
     "65525": {
-      "content": "<|answer_start|>",
+      "content": "<|translation_end|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -81,7 +161,7 @@
       "special": true
     },
     "65526": {
-      "content": "<|answer_end|>",
+      "content": "<|back_translation_start|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -89,7 +169,7 @@
       "special": true
     },
     "65527": {
-      "content": "<|query_start|>",
+      "content": "<|back_translation_end|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -97,7 +177,7 @@
       "special": true
     },
     "65528": {
-      "content": "<|query_end|>",
+      "content": "<|ocr_correction_start|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -105,7 +185,7 @@
       "special": true
     },
     "65529": {
-      "content": "<|source_interpretation_start|>",
+      "content": "<|ocr_correction_end|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -113,28 +193,83 @@
       "special": true
     },
     "65530": {
+      "content": "<|json_scheme_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65531": {
+      "content": "<|json_scheme_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65532": {
+      "content": "<|source_interpretation_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65533": {
       "content": "<|source_interpretation_end|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
       "single_word": false,
       "special": true
+    },
+    "65534": {
+      "content": "<|query_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65535": {
+      "content": "<|query_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
     }
   },
   "additional_special_tokens": [
-    "<|source_id|>",
+    "<|tool_call_start|>",
+    "<|tool_call_end|>",
+    "<|tool_list_start|>",
+    "<|tool_list_end|>",
     "<|source_analysis_start|>",
     "<|source_analysis_end|>",
     "<|source_start|>",
     "<|source_end|>",
+    "<|im_start|>",
+    "<|im_end|>",
     "<|answer_start|>",
     "<|answer_end|>",
-    "<|query_start|>",
-    "<|query_end|>",
+    "<|text_start|>",
+    "<|text_end|>",
+    "<|translation_start|>",
+    "<|translation_end|>",
+    "<|back_translation_start|>",
+    "<|back_translation_end|>",
+    "<|ocr_correction_start|>",
+    "<|ocr_correction_end|>",
+    "<|json_scheme_start|>",
+    "<|json_scheme_end|>",
     "<|source_interpretation_start|>",
-    "<|source_interpretation_end|>"
+    "<|source_interpretation_end|>",
+    "<|query_start|>",
+    "<|query_end|>"
   ],
   "clean_up_tokenization_spaces": true,
   "model_max_length": 1000000000000000019884624838656,
   "tokenizer_class": "PreTrainedTokenizerFast"
-}
+}