README.md

---
library_name: transformers
license: cc-by-nc-4.0
datasets:
- kyujinpy/KOR-OpenOrca-Platypus-v3
language:
- ko
- en
tags:
- Economic
- Finance
---


# Model Details
Model Developers: Sogang University SGEconFinlab(<<https://sc.sogang.ac.kr/aifinlab/>)


### Model Description

This model is a language model specialized in economics and finance. This was learned with various economic/finance-related data.
The data sources are listed below, and we are not releasing the data that we trained on because it was used for research/policy purposes. 
If you wish to use the original data, please contact the original author directly for permission to use it.

- **Developed by:** Sogang University SGEconFinlab(<https://sc.sogang.ac.kr/aifinlab/>)
- **License:** cc-by-nc-4.0
- **Base Model:** yanolja/KoSOLAR-10.7B-v0.2(<https://huggingface.co./yanolja/KoSOLAR-10.7B-v0.2>)


## Loading the Model

    peft_model_id = "SGEcon/KoSOLAR-10.7B-v0.2_fin_v4"
    config = PeftConfig.from_pretrained(peft_model_id)
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16
    )
    model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, quantization_config=bnb_config, device_map={"":0})
    model = PeftModel.from_pretrained(model, peft_model_id)
    tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
    model.eval()

## Conducting Conversation

    import re

    def gen(x):
        inputs = tokenizer(f"### 질문: {x}\n\n### 답변:", return_tensors='pt', return_token_type_ids=False)
    
        # 데이터를 GPU로 이동(사용 가능한 경우)
        inputs = {k: v.to(device="cuda" if torch.cuda.is_available() else "cpu") for k, v in inputs.items()}

        gened = model.generate(
            **inputs,
            max_new_tokens=256,  # 새로 생성할 토큰의 최대 개수
            early_stopping=True,
            num_return_sequences=1,  # 하나의 답변만 생성
            do_sample=True,  # 다양한 답변 생성을 위해 샘플링 활성화
            eos_token_id=tokenizer.eos_token_id,  # EOS 토큰 ID 사용
            temperature=0.9,  # 생성 다양성 조절을 위한 온도 설정
            top_p=0.8,  # nucleus sampling에서 사용할 p 값
            top_k=50  # top-k sampling에서 사용할 k 값
        )
    
        # 생성된 시퀀스를 디코드하여 출력 텍스트로 변환
        decoded = tokenizer.decode(gened[0], skip_special_tokens=True).strip()

        # "### 답변:" 문자열 이후의 텍스트만 추출
        answer_start_idx = decoded.find("### 답변:") + len("### 답변:")
        complete_answer = decoded[answer_start_idx:].strip()

        # 첫 번째 구두점(. ? !)을 찾아서 그 부분까지만 추출
        match = re.search(r"[\.\?\!][^\.\?\!]*$", complete_answer)
        if match:
            complete_answer = complete_answer[:match.end()].strip()
    
        return complete_answer



    
## Training Details
 First, we loaded the base model quantized to 4 bits. It can significantly reduce the amount of memory required to store the model's weights and intermediate computation results, which is beneficial for deploying models in environments with limited memory resources. It can also provide faster inference speeds.
 Then, 

 
### Training Data

1. 한국은행: 경제금융용어 700선(<https://www.bok.or.kr/portal/bbs/B0000249/view.do?nttId=235017&menuNo=200765>)
2. 금융감독원: 금융소비자 정보 포털 파인 금융용어사전(<https://fine.fss.or.kr/fine/fnctip/fncDicary/list.do?menuNo=900021>)
3. KDI 경제정보센터: 시사 용어사전(<https://eiec.kdi.re.kr/material/wordDic.do>)
4. 한국경제신문/한경닷컴: 한경경제용어사전(<https://terms.naver.com/list.naver?cid=42107&categoryId=42107>), 오늘의 TESAT(<https://www.tesat.or.kr/bbs.frm.list/tesat_study?s_cateno=1>), 오늘의 주니어 TESAT(<https://www.tesat.or.kr/bbs.frm.list/tesat_study?s_cateno=5>), 생글생글한경(<https://sgsg.hankyung.com/tesat/study>)
5. 중소벤처기업부/대한민국정부: 중소벤처기업부 전문용어(<https://terms.naver.com/list.naver?cid=42103&categoryId=42103>)
6. 고성삼/법문출판사: 회계·세무 용어사전(<https://terms.naver.com/list.naver?cid=51737&categoryId=51737>)
7. 맨큐의 경제학 8판 Word Index
8. yanolja/KoSOLAR-10.7B-v0.2(<yanolja/KoSOLAR-10.7B-v0.2>)


### Training Procedure 

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->


#### Training Hyperparameters

|Hyperparameter|SGEcon/KoSOLAR-10.7B-v0.2_fin_v4|
|------|---|
|Lora Method|Lora|
|load in 4 bit|True|
|learning rate|1e-5|
|lr scheduler|linear|
|lora alpa|16|
|lora rank|16|
|lora dropout|0.05|
|optim|paged_adamw_32bit|
|target_modules|q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, lm_head|
   
## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

### Results

[More Information Needed]

#### Summary


## Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->