Update README.md

README.md

@@ -1,201 +1,238 @@
 ---
 library_name: transformers
-tags: []
+license: apache-2.0
+datasets:
+- isek-ai/danbooru-tags-2023
+base_model: p1atdev/dart-v1-base
+tags:
+- trl
+- sft
+- danbooru
 ---
 
-# Model Card for Model ID
+# Dart (Danbooru Tags Transformer) v1
 
-<!-- Provide a quick summary of what the model is/does. -->
+This model is a fine-tuned Dart (**Da**nboo**r**u **T**ags Transformer) model that generates danbooru tags.
 
+Demo: [🤗 Space](https://huggingface.co/spaces/p1atdev/danbooru-tags-transformer)
 
+If you are a developer and want to finetune, it's recommended using the base version, [p1atdev/dart-v1-base](https://huggingface.co/p1atdev/dart-v1-base), instead 
 
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+## Usage
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+### Using AutoModel
 
-### Model Sources [optional]
+🤗 Transformers library is required.
 
-<!-- Provide the basic links for the model. -->
+```bash
+pip install -U transformers
+```
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+```py
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
 
-## Uses
+MODEL_NAME = "p1atdev/dart-v1-sft"
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True) # trust_remote_code is required for tokenizer
+model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, torch_dtype=torch.bfloat16)
 
-### Direct Use
+prompt = "<|bos|><rating>rating:sfw, rating:general</rating><copyright>original</copyright><character></character><general>1girl, "
+inputs = tokenizer(prompt, return_tensors="pt").input_ids
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+with torch.no_grad():
+  outputs = model.generate(inputs, generation_config=generation_config)
 
-[More Information Needed]
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+# rating:sfw, rating:general, original, 1girl, ahoge, black hair, blue eyes, blush, closed mouth, ear piercing, earrings, jewelry, looking at viewer, mole, mole under eye, piercing, portrait, shirt, short hair, solo, white shirt
+```
 
-### Downstream Use [optional]
+#### Flash attention (optional)
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+Using flash attention can optimize computations, but it is currently only compatible with Linux.
 
-[More Information Needed]
+```bash
+pip install flash_attn
+```
 
-### Out-of-Scope Use
+### Accelerate with ORTModel
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+🤗 Optimum library is also compatible, for the high performance inference using ONNX.
 
-[More Information Needed]
+```bash
+pip install "optimum[onnxruntime]"
+```
 
-## Bias, Risks, and Limitations
+Two ONNX models are provided:
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+- [Normal](./model.onnx)
+- [Quantized](./model_quantized.onnx)
 
-[More Information Needed]
+Both can be utilized based on the following code:
 
-### Recommendations
+```py
+import torch
+from transformers import AutoTokenizer, GenerationConfig
+from optimum.onnxruntime import ORTModelForCausalLM
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+MODEL_NAME = "p1atdev/dart-v1-sft"
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
 
-## How to Get Started with the Model
+# normal version
+ort_model = ORTModelForCausalLM.from_pretrained(MODEL_NAME)
 
-Use the code below to get started with the model.
+# qunatized version
+# ort_model = ORTModelForCausalLM.from_pretrained(MODEL_NAME, file_name="model_quantized.onnx")
 
-[More Information Needed]
+prompt = "<|bos|><rating>rating:sfw, rating:general</rating><copyright>original</copyright><character></character><general>1girl, "
+inputs = tokenizer(prompt, return_tensors="pt").input_ids
 
-## Training Details
+with torch.no_grad():
+  outputs = model.generate(inputs, generation_config=generation_config)
 
-### Training Data
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+### Prompt guidde
 
-[More Information Needed]
+Due to training with a specialized prompt format, **natural language is not supported**.
 
-### Training Procedure 
+The trained sentences are essentially composed of the following elements, arranged in the strict order shown below:
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+- `<|bos|>`: The bos (begin of sentence) token
+- `<rating>[RATING_PARENT], [RATING_CHILD]</rating>`: The block of rating tags
+  - [RATING_PARENT]: `rating:sfw`, `rating:nsfw`
+  - [RATING_CHILD]:
+    - if `[RATING_PARENT]` is `rating:sfw`: `rating:general`, `rating:sensitive`
+    - else: `rating:questionable`, `rating:explicit`
+- `<copyright>[COPYRIGHT, ...]</copyright>`: The block of copyright tags.
+  - [COPYRIGHT, ...]: All supported copyright tags can be seen in [TODO]()
+- `<character>[CHARACTER, ...]</character>`: The block of character tags.
+  - [CHARACTER, ...]: All supported character tags can be seen in [TODO]()
+- `<general>[LENGTH_TOKEN][GENERAL, ...]<|input_end|>[COMPLETION]</general>`: The block of general tags.
+  - [LENGTH_TOKEN]: A token to specify **total** amount of general tags.
+    - Avaiable:
+      - `<|very_short|>`: less than 10 tags
+      - `<|short|>`: less than 20 tags
+      - `<|long|>`: less than 40 tags (recommended)
+      - `<|very_long|>`: more than 40 tags
+  - [GENERAL, ...]:  All supported general tags can be seen in [TODO]()
+  - `<|input_end|>`: A tag to show the end of input. Set this token at last of prompt.
+  - [COMPLETION]: The model complete tags in alphabetical order.
+- `<|eos|>`: The eos (end of sentence) token
 
-#### Preprocessing [optional]
+- Tags other than special tokens are separated by commas.
+- You can place tags in any order you like in each block.
 
-[More Information Needed]
+Example sentence:
 
+```
+<|bos|><rating>rating:sfw, rating:general</rating><copyright>vocaloid</copyright><character>hatsune miku</character><general><|long|>solo, 1girl, very long hair<|input_end|>blue hair, cowboy shot, ...</general><|eos|>
+```
 
-#### Training Hyperparameters
+Therefore, to complete the tags, the input prompt should be as follows:
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+1. without any copyright and character tags
 
-#### Speeds, Sizes, Times [optional]
+```
+<|bos|><rating>rating:sfw, rating:general</rating><copyright></copyright><character></character><general><|very_long|>1girl, solo, cat ears<|input_end|>
+```
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+2. specifing copyright and character tags
 
-[More Information Needed]
+```
+<|bos|><rating>rating:sfw, rating:general</rating><copyright>sousou no frieren</copyright><character>frieren</character><general><|long|>1girl, solo, from side<|input_end|>
+```
 
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
+## Model Details
 
-<!-- This should link to a Dataset Card if possible. -->
+### Model Description
 
-[More Information Needed]
+- **Developed by:** Plat
+- **Model type:** Causal language model
+- **Language(s) (NLP):** Danbooru tags
+- **License:** Apache-2.0
 
-#### Factors
+- **Demo:** Avaiable on [🤗Space](https://huggingface.co/spaces/p1atdev/danbooru-tags-transformer)
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+## Bias, Risks, and Limitations
 
-[More Information Needed]
+Since this model is a pre-trained model, it cannot accommodate flexible specifications.
 
-#### Metrics
+## Training Details
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+### Training Data
 
-[More Information Needed]
+This model was trained with:
 
-### Results
+- [isek-ai/danbooru-tags-2023](https://huggingface.co/datasets/isek-ai/danbooru-tags-2023): 6M size of danbooru tags dataset since 2005 to 2023
 
-[More Information Needed]
+Only data from 2020 onwards was used for SFT.
 
-#### Summary
+### Training Procedure 
 
+Trained using 🤗 transformers' trainer.
 
+#### Preprocessing
 
-## Model Examination [optional]
+Preprocessing was conducted through the following process:
 
-<!-- Relevant interpretability work for the model goes here -->
+1. Remove data where `general` tags is null.
+2. Remove `general` tags that appear less than 100 times.
+3. Remove undesirable tags such as `watermark` and `bad anatomy`.
+4. Remove based on the number of tags attached to a single post (following rules):
+  - Remove if more than 100 for `general` tags.
+  - Remove if more than 5 for `copyright` tags.
+  - Remove if more than 10 for `character` tags.
+5. Remove posts created before 2020
+6. Set length token according to each tags length
+7. Shuffle some tags in the following rule:
+  - Include people tags (e.g. `1girl`, `no humans`) tags in the shuffle-group with a 95% probability, and do not do so with a 5% probability.
+  - Get tags at a random percentage between 0% and 75% to create a shuffle-group.
+  - Shuffle tags in shuffle-group and concatnate with `<|input_end|>` token and remains in alphabetical order.
+8. Concatnate all categories
 
-[More Information Needed]
+#### Training Hyperparameters
 
-## Environmental Impact
+The following hyperparameters were used during training:
+- learning_rate: 0.0001
+- train_batch_size: 32
+- eval_batch_size: 8
+- seed: 42
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 64
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 500
+- num_epochs: 1
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+## Evaluation
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+Evaluation has not been done yet and it needs to evaluate.
 
-## Technical Specifications [optional]
+## Technical Specifications
 
 ### Model Architecture and Objective
 
-[More Information Needed]
+The architecture of this model is [OPT (Open Pretrained Transformer)](https://huggingface.co/docs/transformers/model_doc/opt), but the position embeddings was not trained.
 
 ### Compute Infrastructure
 
-[More Information Needed]
+In house
 
 #### Hardware
 
-[More Information Needed]
+1x RTX 3070 Ti
 
 #### Software
 
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
+- Dataset processing: [🤗 Datasets](https://github.com/huggingface/datasets)
+- Training: [🤗 Transformers](https://github.com/huggingface/transformers)
+- Optimizing: [🤗 Optimum](https://github.com/huggingface/optimum)
+- SFT: [🤗 TRL](https://github.com/huggingface/trl)
 
 ## More Information [optional]
 
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
-
-
+[More Information Needed]