README.md

metadata

library_name: transformers
license: cc-by-nc-4.0
datasets:
  - kyujinpy/KOR-OpenOrca-Platypus-v3
language:
  - ko
  - en
tags:
  - Economic
  - Finance

Model Details

Model Developers: Sogang University SGEconFinlab(<https://sc.sogang.ac.kr/aifinlab/)

Model Description

This model is a language model specialized in economics and finance. This was learned with various economic/finance-related data. The data sources are listed below, and we are not releasing the data that we trained on because it was used for research/policy purposes. If you wish to use the original data, please contact the original author directly for permission to use it.

• Developed by: Sogang University SGEconFinlab(https://sc.sogang.ac.kr/aifinlab/)
• License: cc-by-nc-4.0
• Base Model: yanolja/KoSOLAR-10.7B-v0.2(https://huggingface.co./yanolja/KoSOLAR-10.7B-v0.2)

Loading the Model

peft_model_id = "SGEcon/KoSOLAR-10.7B-v0.2_fin_v4"
config = PeftConfig.from_pretrained(peft_model_id)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, quantization_config=bnb_config, device_map={"":0})
model = PeftModel.from_pretrained(model, peft_model_id)
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
model.eval()

Conducting Conversation

import re

def gen(x):
    inputs = tokenizer(f"### 질문: {x}\n\n### 답변:", return_tensors='pt', return_token_type_ids=False)

    # 데이터를 GPU로 이동(사용 가능한 경우)
    inputs = {k: v.to(device="cuda" if torch.cuda.is_available() else "cpu") for k, v in inputs.items()}

    gened = model.generate(
        **inputs,
        max_new_tokens=256,  # 새로 생성할 토큰의 최대 개수
        early_stopping=True,
        num_return_sequences=1,  # 하나의 답변만 생성
        do_sample=True,  # 다양한 답변 생성을 위해 샘플링 활성화
        eos_token_id=tokenizer.eos_token_id,  # EOS 토큰 ID 사용
        temperature=0.9,  # 생성 다양성 조절을 위한 온도 설정
        top_p=0.8,  # nucleus sampling에서 사용할 p 값
        top_k=50  # top-k sampling에서 사용할 k 값
    )

    # 생성된 시퀀스를 디코드하여 출력 텍스트로 변환
    decoded = tokenizer.decode(gened[0], skip_special_tokens=True).strip()

    # "### 답변:" 문자열 이후의 텍스트만 추출
    answer_start_idx = decoded.find("### 답변:") + len("### 답변:")
    complete_answer = decoded[answer_start_idx:].strip()

    # 첫 번째 구두점(. ? !)을 찾아서 그 부분까지만 추출
    match = re.search(r"[\.\?\!][^\.\?\!]*$", complete_answer)
    if match:
        complete_answer = complete_answer[:match.end()].strip()

    return complete_answer

Training Details

First, we loaded the base model quantized to 4 bits. It can significantly reduce the amount of memory required to store the model's weights and intermediate computation results, which is beneficial for deploying models in environments with limited memory resources. It can also provide faster inference speeds. Then,

Training Data

1. 한국은행: 경제금융용어 700선(https://www.bok.or.kr/portal/bbs/B0000249/view.do?nttId=235017&menuNo=200765)
2. 금융감독원: 금융소비자 정보 포털 파인 금융용어사전(https://fine.fss.or.kr/fine/fnctip/fncDicary/list.do?menuNo=900021)
3. KDI 경제정보센터: 시사 용어사전(https://eiec.kdi.re.kr/material/wordDic.do)
4. 한국경제신문/한경닷컴: 한경경제용어사전(https://terms.naver.com/list.naver?cid=42107&categoryId=42107), 오늘의 TESAT(https://www.tesat.or.kr/bbs.frm.list/tesat_study?s_cateno=1), 오늘의 주니어 TESAT(https://www.tesat.or.kr/bbs.frm.list/tesat_study?s_cateno=5), 생글생글한경(https://sgsg.hankyung.com/tesat/study)
5. 중소벤처기업부/대한민국정부: 중소벤처기업부 전문용어(https://terms.naver.com/list.naver?cid=42103&categoryId=42103)
6. 고성삼/법문출판사: 회계·세무 용어사전(https://terms.naver.com/list.naver?cid=51737&categoryId=51737)
7. 맨큐의 경제학 8판 Word Index
8. yanolja/KoSOLAR-10.7B-v0.2(<yanolja/KoSOLAR-10.7B-v0.2>)

Training Procedure

Training Hyperparameters

Hyperparameter	SGEcon/KoSOLAR-10.7B-v0.2_fin_v4
Lora Method	Lora
load in 4 bit	True
learning rate	1e-5
lr scheduler	linear
lora alpa	16
lora rank	16
lora dropout	0.05
optim	paged_adamw_32bit
target_modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, lm_head

Evaluation

Testing Data, Factors & Metrics

Testing Data

[More Information Needed]

Results

[More Information Needed]

Summary

Citation [optional]