|
--- |
|
library_name: transformers |
|
license: cc-by-nc-4.0 |
|
datasets: |
|
- kyujinpy/KOR-OpenOrca-Platypus-v3 |
|
language: |
|
- ko |
|
- en |
|
tags: |
|
- Economic |
|
- Finance |
|
base_model: EleutherAI/polyglot-ko-5.8b |
|
--- |
|
|
|
|
|
# Model Details |
|
Model Developers: Sogang University SGEconFinlab(<<https://sc.sogang.ac.kr/aifinlab/>) |
|
|
|
|
|
### Model Description |
|
|
|
This model is a language model specialized in economics and finance. This was learned with various economic/finance-related data. |
|
The data sources are listed below, and we are not releasing the data that we trained on because it was used for research/policy purposes. |
|
If you wish to use the original data, please contact the original author directly for permission to use it. |
|
|
|
- **Developed by:** Sogang University SGEconFinlab(<https://sc.sogang.ac.kr/aifinlab/>) |
|
- **License:** cc-by-nc-4.0 |
|
- **Base Model:** EleutherAI/polyglot-ko-5.8b(<https://huggingface.co./EleutherAI/polyglot-ko-5.8b>) |
|
|
|
## Loading the Model |
|
|
|
peft_model_id = "SGEcon/polyglot-ko-5.8b_fin_v4" |
|
config = PeftConfig.from_pretrained(peft_model_id) |
|
bnb_config = BitsAndBytesConfig( |
|
load_in_4bit=True, |
|
bnb_4bit_use_double_quant=True, |
|
bnb_4bit_quant_type="nf4", |
|
bnb_4bit_compute_dtype=torch.bfloat16 |
|
) |
|
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, quantization_config=bnb_config, device_map={"":0}) |
|
model = PeftModel.from_pretrained(model, peft_model_id) |
|
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path) |
|
model.eval() |
|
|
|
## Conducting Conversation |
|
|
|
import re |
|
|
|
def gen(x): |
|
inputs = tokenizer(f"### ์ง๋ฌธ: {x}\n\n### ๋ต๋ณ:", return_tensors='pt', return_token_type_ids=False) |
|
|
|
# ๋ฐ์ดํฐ๋ฅผ GPU๋ก ์ด๋(์ฌ์ฉ ๊ฐ๋ฅํ ๊ฒฝ์ฐ) |
|
inputs = {k: v.to(device="cuda" if torch.cuda.is_available() else "cpu") for k, v in inputs.items()} |
|
|
|
gened = model.generate( |
|
**inputs, |
|
max_new_tokens=256, # ์๋ก ์์ฑํ ํ ํฐ์ ์ต๋ ๊ฐ์ |
|
early_stopping=True, |
|
num_return_sequences=1, # ํ๋์ ๋ต๋ณ๋ง ์์ฑ |
|
do_sample=True, # ๋ค์ํ ๋ต๋ณ ์์ฑ์ ์ํด ์ํ๋ง ํ์ฑํ |
|
eos_token_id=tokenizer.eos_token_id, # EOS ํ ํฐ ID ์ฌ์ฉ |
|
temperature=0.9, # ์์ฑ ๋ค์์ฑ ์กฐ์ ์ ์ํ ์จ๋ ์ค์ |
|
top_p=0.8, # nucleus sampling์์ ์ฌ์ฉํ p ๊ฐ |
|
top_k=50 # top-k sampling์์ ์ฌ์ฉํ k ๊ฐ |
|
) |
|
|
|
# ์์ฑ๋ ์ํ์ค๋ฅผ ๋์ฝ๋ํ์ฌ ์ถ๋ ฅ ํ
์คํธ๋ก ๋ณํ |
|
decoded = tokenizer.decode(gened[0], skip_special_tokens=True).strip() |
|
|
|
# "### ๋ต๋ณ:" ๋ฌธ์์ด ์ดํ์ ํ
์คํธ๋ง ์ถ์ถ |
|
answer_start_idx = decoded.find("### ๋ต๋ณ:") + len("### ๋ต๋ณ:") |
|
complete_answer = decoded[answer_start_idx:].strip() |
|
|
|
# ์ฒซ ๋ฒ์งธ ๊ตฌ๋์ (. ? !)์ ์ฐพ์์ ๊ทธ ๋ถ๋ถ๊น์ง๋ง ์ถ์ถ |
|
match = re.search(r"[\.\?\!][^\.\?\!]*$", complete_answer) |
|
if match: |
|
complete_answer = complete_answer[:match.end()].strip() |
|
|
|
return complete_answer |
|
|
|
|
|
|
|
|
|
## Training Details |
|
|
|
|
|
- We train our model with PEFT. |
|
PEFT is a technique that does not tune all parameters of a model during fine-tuning, but only a small subset of parameters. |
|
By tuning only a few parameters while leaving others fixed, the model is less likely to suffer from catastrophic forgetting, where the model forgets previously learned tasks when it learns new ones. |
|
This significantly reduces computation and storage costs. |
|
|
|
- We use QLora to train the base model. |
|
Quantized Low Rank Adapters (QLoRA) is an efficient technique that uses 4-bit quantized pre-trained language models to fine-tune 65 billion parameter models on a 48 GB GPU while significantly reducing memory usage. |
|
The method uses NormalFloat 4-bit (NF4), a new data type that is theoretically optimal for normally distributed weights; Double Quantization, which further quantizes quantization constants to reduce average memory usage; and Paged Optimizers, which manage memory spikes during mini-batch processing, to increase memory efficiency without sacrificing performance. |
|
|
|
- Also, we performed instruction tuning using the data that we collected and the kyujinpy/KOR-OpenOrca-Platypus-v3 dataset on the hugging face. |
|
Instruction tuning is learning in a supervised learning format that uses instructions and input data together as input and output data as a pair. |
|
In other words, instruction tuning involves fine-tuning a pre-trained model for a specific task or set of tasks, where the model is taught to follow specific instructions or guidelines. |
|
Instruction tuning is a type of Supervised Fine-tuning (SFT) that aims to improve the generality and adaptability of a model by introducing an additional dimension that enables the model to understand and follow specific instructions. |
|
|
|
|
|
### Training Data |
|
|
|
1. ํ๊ตญ์ํ: ๊ฒฝ์ ๊ธ์ต์ฉ์ด 700์ (<https://www.bok.or.kr/portal/bbs/B0000249/view.do?nttId=235017&menuNo=200765>) |
|
2. ๊ธ์ต๊ฐ๋
์: ๊ธ์ต์๋น์ ์ ๋ณด ํฌํธ ํ์ธ ๊ธ์ต์ฉ์ด์ฌ์ (<https://fine.fss.or.kr/fine/fnctip/fncDicary/list.do?menuNo=900021>) |
|
3. KDI ๊ฒฝ์ ์ ๋ณด์ผํฐ: ์์ฌ ์ฉ์ด์ฌ์ (<https://eiec.kdi.re.kr/material/wordDic.do>) |
|
4. ํ๊ตญ๊ฒฝ์ ์ ๋ฌธ/ํ๊ฒฝ๋ท์ปด: ํ๊ฒฝ๊ฒฝ์ ์ฉ์ด์ฌ์ (<https://terms.naver.com/list.naver?cid=42107&categoryId=42107>), ์ค๋์ TESAT(<https://www.tesat.or.kr/bbs.frm.list/tesat_study?s_cateno=1>), ์ค๋์ ์ฃผ๋์ด TESAT(<https://www.tesat.or.kr/bbs.frm.list/tesat_study?s_cateno=5>), ์๊ธ์๊ธํ๊ฒฝ(<https://sgsg.hankyung.com/tesat/study>) |
|
5. ์ค์๋ฒค์ฒ๊ธฐ์
๋ถ/๋ํ๋ฏผ๊ตญ์ ๋ถ: ์ค์๋ฒค์ฒ๊ธฐ์
๋ถ ์ ๋ฌธ์ฉ์ด(<https://terms.naver.com/list.naver?cid=42103&categoryId=42103>) |
|
6. ๊ณ ์ฑ์ผ/๋ฒ๋ฌธ์ถํ์ฌ: ํ๊ณยท์ธ๋ฌด ์ฉ์ด์ฌ์ (<https://terms.naver.com/list.naver?cid=51737&categoryId=51737>) |
|
7. ๋งจํ์ ๊ฒฝ์ ํ 8ํ Word Index |
|
8. kyujinpy/KOR-OpenOrca-Platypus-v3(<https://huggingface.co./datasets/kyujinpy/KOR-OpenOrca-Platypus-v3>) |
|
|
|
|
|
At the request of the original author, it is not to be used for commercial purposes. Therefore, it is licensed under the license CC-BY-NC-4.0. |
|
The copyright of the data used belongs to the original author, so please contact the original author when using it. |
|
|
|
|
|
|
|
|
|
### Training Hyperparameters |
|
|
|
|Hyperparameter|SGEcon/polyglot-ko-5.8b_fin_v4| |
|
|------|---| |
|
|Lora Method|Lora| |
|
|load in 4 bit|True| |
|
|learning rate|3e-4| |
|
|lora alpa|8| |
|
|lora rank|16| |
|
|lora dropout|0.05| |
|
|optim|paged_adamw_8bit| |
|
|target_modules|query_key_value| |
|
|
|
|
|
|
|
### Example |
|
|
|
> ์ค์์ํ์ ์ญํ ์ ๋ํด์ ์ค๋ช
ํด์ค๋? |
|
|
|
>> ์ค์์ํ์ ๊ตญ๊ฐ์ ์ ์ฉ์ ๊ด๋ฆฌํ๋ ์ค์ํ ์ญํ ์ ๋ด๋นํ๊ณ ์๋ค. ํนํ ๊ธ์ต์์ฅ์ ์์ ์ฑ์ ์ ์งํ๋ ๊ฒ์ด ๊ฐ์ฅ ์ค์ํ๋ค. ์ค์์ํ์ ๊ธ์ต์์ฅ์ด ๋ถ์์ ํด์ง ๊ฒฝ์ฐ ์ ๋์ฑ์ ๊ณต๊ธํ๋ ๋ฑ ๊ฒฝ์ ๊ฐ ์์ ์ ์ผ๋ก ์ด์๋๋๋ก ์ง์ํ๋ ์ญํ ์ ํ๋ค. ์ด ๊ณผ์ ์์ ํตํ๋์ด ์กฐ์ ๋๊ธฐ๋ ํ๋ค. ํ๊ตญ์ํ์ ๊ฒฝ์ฐ ์ธํ๋ณด์ ์ก์ด ์ผ์ ์์ค ์ดํ๋ก ๋จ์ด์ง๋ฉด ์ธํ์์ฅ์ ๊ฐ์
ํด ์ธํ๋ณด์ ์ก์ ๋๋ฆฌ๋ ๋ฐฉ๋ฒ์ผ๋ก ์ธํ์์ฅ์ ์์ ์ฑ์ ์ ์งํ๋ค. ๋ ํตํ๋์ด ๋๋ฌด ๋ง์ ๊ฒฝ์ฐ์๋ ๋์ ๊ฐ์น๋ฅผ ๋ฎ์ถฐ์ ํตํ๋์ ์กฐ์ ํ๋ค. ์ด๋ฐ ๋ฐฉ๋ฒ์ผ๋ก ๊ฒฝ๊ธฐ๋ฅผ ์กฐ์ ํ๊ธฐ๋ ํ๋ค. ํ๊ตญ์ํ์ ์ ์ฑ
๊ธ๋ฆฌ๊ฐ ์ค๋ฅด๋ฉด ์์ค๊ธ๋ฆฌ๊ฐ ์ค๋ฅด๊ณ ์ด๋ ๊ธฐ์
์ ๊ธ์ต๋น์ฉ์ ์ฆ๊ฐ์์ผ ๊ฒฝ๊ธฐ๋ฅผ ์์ถ์ํค๊ธฐ ๋๋ฌธ์ ํ๊ตญ์ํ์ ์ ์ฑ
๊ธ๋ฆฌ๋ฅผ ์ธํํ๋ ๋ฐฉ๋ฒ์ผ๋ก ๊ฒฝ๊ธฐ๋ฅผ ๋ถ์ํ๊ธฐ๋ ํ๋ค. ๋ ์ค์์ํ์ ๊ฒฝ์ ๊ฐ ์ด๋ ค์์ง๋ฉด ๋์ ํ์ด์ ๊ฒฝ์ ๋ฅผ ์ด๋ฆฌ๋ ์ญํ ์ ํ๊ธฐ๋ ํ๋ค. ์ฆ, ์ค์์ํ์ ๊ฒฝ๊ธฐ๊ฐ ์นจ์ฒด๋ผ ๋์ด ์ ๋์ง ์๋ ์ํฉ์ด ๋๋ฉด ์์ค์ ๋์ ํ์ด์ ๊ฒฝ๊ธฐ๋ฅผ ๋ถ์ํ๋ ๋ฐฉ๋ฒ์ผ๋ก ๊ฒฝ์ ๋ฅผ ์ด๋ฆฌ๋ ์ญํ ์ ํ๋ค. |
|
|