update readme

README.md

@@ -11,30 +11,114 @@ model-index:
   results: []
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# Llama-3.1-Legal-ThaiCCL-8B
 
-# llama3.1-8b-legal-combine-ccl-16
-
-This model is a fine-tuned version of [/ist-project/scads/pumet/models/Meta-Llama-3.1-8B](https://huggingface.co//ist-project/scads/pumet/models/Meta-Llama-3.1-8B) on the /ist-project/scads/pumet/WangchanX/datasets/legal-combine-ccl dataset.
+Llama-3.1-Legal-ThaiCCL-8B is a large language model built upon Llama-3.1-8B, designed to answer Thai legal questions. It is full finetuned on the [WangchanX Thai Legal dataset](https://huggingface.co/datasets/airesearch/WangchanX-Legal-ThaiCCL-RAG) using the [WangchanX Finetuning pipeline](https://github.com/vistec-AI/WangchanX). The model is intended to be used with a supporting Retrieval-Augmented Generation (RAG) system which queries relevant supporting legal documents for the model to reference when responding to the questions.
 
 ## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
+- Base model: [Meta Llama 3.1 8B](https://huggingface.co/meta-llama/Llama-3.1-8B)
+- Training Repository: [WangchanX Finetuning Pipeline](https://github.com/vistec-AI/WangchanX)
+- Training Dataset: [WangchanX Thai Legal dataset](https://huggingface.co/datasets/airesearch/WangchanX-Legal-ThaiCCL-RAG) 
+- License: [Meta's Llama 3.1 Community License Agreement](https://www.llama.com/llama3_1/license/)
+
+## Model Usage
+```python
+from transformers import pipeline
+import torch
+
+EN_QA_TEMPLATE = "Given the user's query in the context of Thai legal matters, the RAG system retrieves the top_n related documents. From these documents, it's crucial to identify and utilize only the most relevant ones to craft an accurate and informative response.Context information is below.\n\n---------------------\nContext: Thai legal domain\nQuery: {query_str}\nRetrieved Documents: {context_str}\n---------------------\n\n Using the provided context information and the list of retrieved documents, you will focus on selecting the documents that are most relevant to the user's query. This selection process involves evaluating the content of each document for its pertinency to the query, ensuring that the response is based on accurate and contextually appropriate information.Based on the selected documents, you will synthesize a response that addresses the user's query, drawing directly from the content of these documents to provide a precise, legally informed answer.You must answer in Thai.\nAnswer:"
+
+EN_SYSTEM_PROMPT_STR = """You are a legal assistant named Sommai (สมหมาย in Thai). You provide legal advice in a friendly, clear, and approachable manner. When answering questions, you reference the relevant law sections, including the name of the act or code they are from. You explain what these sections entail, including any associated punishments, fees, or obligations. Your tone is polite yet informal, making users feel comfortable, like consulting a trusted friend. If a question falls outside your knowledge, you must respond with the exact phrase: 'สมหมายไม่สามารถตอบคำถามนี้ได้ครับ'. You avoid making up information and guide users based on accurate legal references relevant to their situation. Where applicable, you provide practical advice, such as preparing documents, seeking medical attention, or contacting authorities. If asked about past Supreme Court judgments, you must state that you do not have information on those judgments at this time."""
+
+query = "การร้องขอให้ศาลสั่งให้บุคคลเป็นคนไร้ความสามารถมีหลักเกณฑ์การพิจารณาอย่างไร"
+
+context = """ประมวลกฎหมายแพ่งและพาณิชย์ มาตรา 33 ในคดีที่มีการร้องขอให้ศาลสั่งให้บุคคลใดเป็นคนไร้ความสามารถเพราะวิกลจริต ถ้าทางพิจารณาได้ความว่าบุคคลนั้นไม่วิกลจริต แต่มีจิตฟั่นเฟือนไม่สมประกอบ เมื่อศาลเห็นสมควรหรือเมื่อมีคำขอของคู่ความหรือของบุคคลตามที่ระบุไว้ในมาตรา 28 ศาลอาจสั่งให้บุคคลนั้นเป็นคนเสมือนไร้ความสามารถก็ได้ หรือในคดีที่มีการร้องขอให้ศาลสั่งให้���ุคคลใดเป็นคนเสมือนไร้ความสามารถเพราะมีจิตฟั่นเฟือนไม่สมประกอบ ถ้าทางพิจารณาได้ความว่าบุคคลนั้นวิกลจริต เมื่อมีคำขอของคู่ความหรือของบุคคลตามที่ระบุไว้ในมาตรา 28 ศาลอาจสั่งให้บุคคลนั้นเป็นคนไร้ความสามารถก็ได้"""
+
+
+model_id = "airesearch/LLaMa3.1-8B-Legal-ThaiCCL-Combine"
+
+pipeline = transformers.pipeline(
+    "text-generation",
+    model=model_id,
+    model_kwargs={"torch_dtype": torch.bfloat16},
+    device_map="auto",
+)
+
+sample = [
+    {"role": "system", "content": SYSTEM_PROMPT_STR},
+    {"role": "user", "content": QA_template.format(context_str=context, query_str=query)},
+]
+
+prompt = pipeline.tokenizer.apply_chat_template(sample, 
+                                                tokenize=False, 
+                                                add_generation_prompt=True)
+
+outputs = pipeline(
+    prompt,
+    max_new_tokens = 512,
+    eos_token_id = terminators,
+    do_sample = True,
+    temperature = 0.6,
+    top_p = 0.9
+)
+
+print(outputs[0]["generated_text"][-1])
+```
+
+## Training Data
+The model is trained on the [WangchanX Legal ThaiCCL RAG dataset](https://huggingface.co/datasets/airesearch/WangchanX-Legal-ThaiCCL-RAG), which is a Thai legal question-answering dataset created using a RAG system to query relevant supporting legal datasets based on a question for the LLM to reference in its answer. For more information on how the datasets was created please refer to this [blog](https://medium.com/airesearch-in-th/ชุดข้อมูลกฎหมายไทยสำหรับการพัฒนา-rag-0eb2eab283a1).
+
+To emulate a real world use case, during training we incorporated both the positive and negative context (if available) into the prompt. We found that this resulted in a model that is more robust towards cases that the RAG system also passes in irrelevant contexts mixed with the correct context to reference (refer to the [evaluation](#evaluation) section for results). 
+
+### Prompt Format
+We recommend using the same chat template (system prompt and question template of context, query, and retreived documents) when using the provided weights, since the model was trained with the specific system prompt and question template. Example input prompt:
+```
+<|begin_of_text|><|start_header_id|>system<|end_header_id|>
+You are a legal assistant named Sommai (สมหมาย in Thai), you provide legal advice to users in a friendly and understandable manner. When answering questions, you specifically reference the law sections relevant to the query, including the name of the act or code they originated from, an explanation of what those sections entail, and any associated punishments or fees. Your tone is approachable and informal yet polite, making users feel as if they are seeking advice from a friend. If a question arises that does not match the information you possess, you must acknowledge your current limitations by stating this exactly sentence: 'สมหมายไม่สามารถตอบคำถามนี้ได้ครับ'. You will not fabricate information but rather guide users based on actual law sections relevant to their situation. Additionally, you offer practical advice on next steps, such as gathering required documents, seeking medical attention, or visiting a police station, as applicable. If inquired about past Supreme Court judgments, you must reply that you do not have information on those judgments yet.<|eot_id|>
+<|start_header_id|>user<|end_header_id|>
+
+Given the user's query in the context of Thai legal matters, the RAG system retrieves the top_n related documents. From these documents, it's crucial to identify and utilize only the most relevant ones to craft an accurate and informative response.
+
+Context information is below.
+---------------------
+Context: Thai legal domain
+Query: {question}
+Retreived Documents: {retreived legal documents}
+---------------------
+
+Using the provided context information and the list of retrieved documents, you will focus on selecting the documents that are most relevant to the user's query. This selection process involves evaluating the content of each document for its pertinency to the query, ensuring that the response is based on accurate and contextually appropriate information.
+Based on the selected documents, you will synthesize a response that addresses the user's query, drawing directly from the content of these documents to provide a precise, legally informed answer.
+You must answer in Thai.
+Answer:
+<|eot_id|>
+<|start_header_id|>assistant<|end_header_id|>
+```
+Here is a Python code snippet on how to apply the chat template with the provided system prompt and question template on the WangchanX Legal Thai CCL dataset:
+```python
+EN_QA_TEMPLATE = "Given the user's query in the context of Thai legal matters, the RAG system retrieves the top_n related documents. From these documents, it's crucial to identify and utilize only the most relevant ones to craft an accurate and informative response.Context information is below.\n\n---------------------\nContext: Thai legal domain\nQuery: {query_str}\nRetrieved Documents: {context_str}\n---------------------\n\n Using the provided context information and the list of retrieved documents, you will focus on selecting the documents that are most relevant to the user's query. This selection process involves evaluating the content of each document for its pertinency to the query, ensuring that the response is based on accurate and contextually appropriate information.Based on the selected documents, you will synthesize a response that addresses the user's query, drawing directly from the content of these documents to provide a precise, legally informed answer.You must answer in Thai.\nAnswer:"
+
+EN_SYSTEM_PROMPT_STR = """You are a legal assistant named Sommai (สมหมาย in Thai). You provide legal advice in a friendly, clear, and approachable manner. When answering questions, you reference the relevant law sections, including the name of the act or code they are from. You explain what these sections entail, including any associated punishments, fees, or obligations. Your tone is polite yet informal, making users feel comfortable, like consulting a trusted friend. If a question falls outside your knowledge, you must respond with the exact phrase: 'สมหมายไม่สามารถตอบคำถามนี้ได้ครับ'. You avoid making up information and guide users based on accurate legal references relevant to their situation. Where applicable, you provide practical advice, such as preparing documents, seeking medical attention, or contacting authorities. If asked about past Supreme Court judgments, you must state that you do not have information on those judgments at this time."""
+
+def format(example):
+    if "คำตอบ: " in example["positive_answer"]:
+        example["positive_answer"] = example["positive_answer"].replace("คำตอบ: ", "")
+    if example['positive_contexts']:
+        context = ''.join([v['text'] for v in example['positive_contexts'][:5]])
+        message = [
+            {"content": EN_SYSTEM_PROMPT_STR, "role": "system"}, 
+            {"content": EN_QA_TEMPLATE.format(query_str=example['question'], context_str=context), "role": "user"}, 
+        ]
+    else:
+        message = [
+            {"content": EN_SYSTEM_PROMPT_STR, "role": "system"}, 
+            {"content": EN_QA_TEMPLATE.format(query_str=example['question'], context_str=" "), "role": "user"}, 
+        ]
+    return dict(messages=message)
+dataset = dataset.map(format, batched=False)
+```
 
 ### Training hyperparameters
-
-The following hyperparameters were used during training:
+We full fine-tuned Llama-3.1-8B using the following hyperparameters:
 - learning_rate: 0.0002
 - train_batch_size: 4
 - eval_batch_size: 4
@@ -49,13 +133,37 @@ The following hyperparameters were used during training:
 - lr_scheduler_warmup_ratio: 0.1
 - num_epochs: 4
 
-### Training results
-
-
-
-### Framework versions
-
-- Transformers 4.44.2
-- Pytorch 2.4.1+cu121
-- Datasets 3.0.1
-- Tokenizers 0.19.1
+Total training time: 2:15:14.66
+
+## Evaluation
+We tested our model based on the test set of the WangchanX Legal Thai CCL dataset using both traditional (MRC) metrics and a LLM as judge technique based on the paper [CHIE: Generative MRC Evaluation for in-context QA with Correctness, Helpfulness, Irrelevancy, and Extraneousness Aspects](https://aclanthology.org/2024.genbench-1.10.pdf)
+
+Note: `LLaMa3.1-8B-Legal-ThaiCCL` is trained on only positive contexts while `LLaMa3.1-8B-Legal-ThaiCCL-Combine` is trained on both positive and negative contexts
+
+### Table 1: MRC Results
+| Model                                | Context Type      | Answer Type   | ROUGE-L | Character Error Rate (CER) | Word Error Rate (WER) | BERT Score | F1-score XQuAD | Exact Match XQuAD |
+|--------------------------------------|-------------------|---------------|----------|----------------------------|-----------------------|------------|----------------|-------------------|
+| Zero-shot LLaMa3.1-8B-Instruct       | Golden Passage    | Only Positive | 0.553    | 1.181                      | 1.301                 | 0.769      | 48.788         | 0.0               |
+| LLaMa3.1-8B-Legal-ThaiCCL           | Golden Passage    | Only Positive | 0.603    | 0.667                      | 0.736                 | 0.821      | 60.039         | 0.053             |
+| LLaMa3.1-8B-Legal-ThaiCCL-Combine   | Golden Passage    | Only Positive | 0.715    | 0.695                      | 0.758                 | 0.833      | 64.578         | 0.614             |
+| Zero-shot LLaMa3.1-70B-Instruct      | Golden Passage    | Only Positive | 0.830    | 0.768                      | 0.848                 | 0.830      | 61.497         | 0.0               |
+| Zero-shot LLaMa3.1-8B-Instruct       | Retrieval Passage | Only Positive | 0.422    | 1.631                      | 1.773                 | 0.757      | 39.639         | 0.0               |
+| LLaMa3.1-8B-Legal-ThaiCCL           | Retrieval Passage | Only Positive | 0.366    | 1.078                      | 1.220                 | 0.779      | 44.238         | 0.03              |
+| LLaMa3.1-8B-Legal-ThaiCCL-Combine   | Retrieval Passage | Only Positive | 0.516    | 0.884                      | 0.884                 | 0.816      | 54.948         | 0.668             |
+| Zero-shot LLaMa3.1-70B-Instruct      | Retrieval Passage | Only Positive | 0.616    | 0.934                      | 1.020                 | 0.816      | 54.930         | 0.0               |
+
+### Table 2: CHIE Results
+| Model                                | Context Type      | Answer Type   | Q1: Correctness [H] | Q2: Helpfulness [H] | Q3: Irrelevancy [L] | Q4: Out-of-Context [L] |
+|--------------------------------------|-------------------|---------------|----------------------|----------------------|---------------------|------------------------|
+| Zero-shot LLaMa3.1-8B-Instruct       | Golden Passage    | Only Positive | 0.740                | 0.808                | 0.480               | 0.410                  |
+| LLaMa3.1-8B-Legal-ThaiCCL           | Golden Passage    | Only Positive | 0.705                | 0.486                | 0.294               | 0.208                  |
+| LLaMa3.1-8B-Legal-ThaiCCL-Combine   | Golden Passage    | Only Positive | 0.565                | 0.468                | 0.405               | 0.325                  |
+| Zero-shot LLaMa3.1-70B-Instruct      | Golden Passage    | Only Positive | 0.870                | 0.658                | 0.316               | 0.247                  |
+| Zero-shot LLaMa3.1-8B-Instruct       | Retrieval Passage | Only Positive | 0.480                | 0.822                | 0.557               | 0.248                  |
+| LLaMa3.1-8B-Legal-ThaiCCL           | Retrieval Passage | Only Positive | 0.274                | 0.470                | 0.720               | 0.191                  |
+| LLaMa3.1-8B-Legal-ThaiCCL-Combine   | Retrieval Passage | Only Positive | 0.532                | 0.445                | 0.508               | 0.203                  |
+| Zero-shot LLaMa3.1-70B-Instruct      | Retrieval Passage | Only Positive | 0.748                | 0.594                | 0.364               | 0.202                  |
+
+
+## License and use
+The model is released under [Meta's Llama 3.1 Community License Agreement](https://www.llama.com/llama3_1/license/). Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc.