Update README.md

README.md

@@ -1,6 +1,175 @@
 ---
+language:
+- en
+- ko
+pipeline_tag: text-generation
+inference: false
+tags:
+- facebook
+- meta
+- pytorch
+- llama
+- llama-2
+- llama-2-chat
+license: apache-2.0
 library_name: peft
 ---
+
+
+# komt-Llama-2-13b-hf-lora
+This model fine-tuned the aaa model using PEFT-LoRA.
+
+The "komt-Llama-2-13b-hf-lora" model was developed using a multi-task instruction technique aimed at enhancing Korean language performance. For more details, please refer to the GitHub Repository. 
+Please refer below for more detailed information.
+
+For more detailed information, please refer to the https://huggingface.co/davidkim205/komt-Llama-2-13b-hf.
+
+## Model Details
+
+* **Model Developers** :  davidkim(changyeon kim)
+* **Repository** : https://github.com/davidkim205/komt
+* **Lora target modules** : q_proj, o_proj, v_proj, gate_proj, down_proj, k_proj, up_proj
+* **Model Size** : 120MB
+* **Model Architecture** : komt-Llama-2-13b is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning by multi-task instruction
+## Dataset
+korean multi-task instruction dataset
+
+## Prompt Template
+```
+### instruction: {prompt}
+
+### Response: 
+```
+Examples:
+```
+### instruction: 자동차 종합(정기)검사 의무기간은 얼마인가요?
+
+### Response:
+
+```
+response:
+``` 
+### instruction: 자동차 종합(정기)검사 의무기간은 얼마인가요?
+
+### Response: 자동차 종합(정기)검사 의무기간은 2년입니다. 이 기간 동안 검사를 받지 않으면 과태료가 부과됩니다. 자동차 종합(정기)검사 의무기간은 2013년 12월 31일부터 시행되었습니다
+```
+
+## Usage
+
+After downloading from GitHub, please install as follows:
+``` 
+git clone https://github.com/davidkim205/komt
+cd komt
+pip install -r lora/requirements_lora.txt
+
+```
+* Requirements Python >=3.8. Linux distribution (Ubuntu, MacOS, etc.) + CUDA > 10.0.
+Refer https://github.com/TimDettmers/bitsandbytes#tldr
+
+``` 
+import torch
+
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
+from transformers import StoppingCriteria, StoppingCriteriaList
+from transformers import TextStreamer, GenerationConfig
+from peft import PeftModel, PeftConfig
+
+class LocalStoppingCriteria(StoppingCriteria):
+
+    def __init__(self, tokenizer, stop_words = []):
+        super().__init__()
+
+        stops = [tokenizer(stop_word, return_tensors='pt', add_special_tokens = False)['input_ids'].squeeze() for stop_word in stop_words]
+        print('stop_words', stop_words)
+        print('stop_words_ids', stops)
+        self.stop_words = stop_words
+        self.stops = [stop.cuda() for stop in stops]
+        self.tokenizer = tokenizer
+    def _compare_token(self, input_ids):
+        for stop in self.stops:
+            if len(stop.size()) != 1:
+                continue
+            stop_len = len(stop)
+            if torch.all((stop == input_ids[0][-stop_len:])).item():
+                return True
+
+        return False
+    def _compare_decode(self, input_ids):
+        input_str = self.tokenizer.decode(input_ids[0])
+        for stop_word in self.stop_words:
+            if input_str.endswith(stop_word):
+                return True
+        return False
+
+    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor):
+        input_str = self.tokenizer.decode(input_ids[0])
+        for stop_word in self.stop_words:
+            if input_str.endswith(stop_word):
+                return True
+        return False
+
+#
+# config
+peft_model_name = 'davidkim205/komt-Llama-2-7b-chat-hf-lora'
+model_name = 'davidkim205/komt-Llama-2-7b-chat-hf'
+instruction_prefix = "### instruction: "
+input_prefix = "### input: "
+answer_prefix = "### Response: "
+endoftext = "<|end|>"
+stop_words = [endoftext, '<s>', '###']
+generation_config = GenerationConfig(
+    temperature=0.9,
+    top_p=0.7,
+    top_k=100,
+    max_new_tokens=2048,
+    early_stopping=True,
+    do_sample=True,
+)
+#
+# create model
+config = PeftConfig.from_pretrained(peft_model_name)
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16
+)
+model = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=bnb_config,
+                                             device_map="auto")
+model = PeftModel.from_pretrained(model, peft_model_name)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+stopping_criteria = StoppingCriteriaList([LocalStoppingCriteria(tokenizer=tokenizer, stop_words=stop_words)])
+streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
+model.eval()
+
+#
+# generate
+prompt = f"### instruction: 자동차 종합(정기)검사 의무기간은 얼마인가요?.\n\n### Response:"
+gened = model.generate(
+    **tokenizer(
+        prompt,
+        return_tensors='pt',
+        return_token_type_ids=False
+    ).to('cuda'),
+    generation_config=generation_config,
+    eos_token_id=model.config.eos_token_id,
+    stopping_criteria=stopping_criteria,
+    streamer=streamer
+)
+output_text = tokenizer.decode(gened[0], skip_special_tokens=True)
+
+print('--------------------')
+print(output_text)
+
+```
+response:
+``` 
+nlp는 자연어 처리의 약자로, 자연어를 사용하여 인간과 컴퓨터 간의 상호 작용을 다루는 분야입니다. 컴퓨터와 인간이 서로 상호 작용하는 데 사용되는 언어와 기술을 포함하며, 컴퓨터는 인간의 언어를 처리하고 분석하여 인간의 작업을 돕거나 작업을 자동화하는 데 사용됩니다. 따라서 컴퓨터가 컴퓨터에서 작업하는 데 사용되는 컴퓨터 프로그램이나 프로그램과 비슷하게 인간도 자신의 작업에 사용되는 컴퓨터 프로그램과 비슷한 방식으로 작업할 수 있습니다.
+```
+## Hardware and Software
+- nvidia driver : 535.54.03
+- CUDA Version: 12.2
+- 
 ## Training procedure