metadata
license: mit
tags:
- kobart-hashtag
- generated_from_trainer
base_model: gogamza/kobart-summarization
model-index:
- name: modelling
results: []
modelling
This model is a fine-tuned version of gogamza/kobart-summarization on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 0.5862
Model description
This model generates hash tag from input text.
Training and evaluation data
This model was trained by the self-instruction process. All data used for fine-tuning this model were generated by chatGPT 3.5.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5.6e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 300
- num_epochs: 50
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.8136 | 1.42 | 500 | 0.6526 |
| 0.4651 | 2.85 | 1000 | 0.5862 |
| 0.2643 | 4.27 | 1500 | 0.6752 |
| 0.1642 | 5.7 | 2000 | 0.6840 |
| 0.1078 | 7.12 | 2500 | 0.7554 |
Framework versions
- Transformers 4.37.1
- Pytorch 2.1.2+cu118
- Datasets 2.16.1
- Tokenizers 0.15.0
How to Get Started with the Model
Use the code below to get started with the model. You can adjust hyperparameters to fit on your data.
from transformers import PreTrainedTokenizerFast, BartForConditionalGeneration
tokenizer = PreTrainedTokenizerFast.from_pretrained("jjae/kobart-hashtag")
model = BartForConditionalGeneration.from_pretrained("jjae/kobart-hashtag")
def make_tag(text):
input_ids = tokenizer.encode(text, return_tensors="pt").to(device)
output = model.generate(input_ids = input_ids, bos_token_id = model.config.bos_token_id,
eos_token_id = model.config.eos_token_id, length_penalty = 3.0, max_length = 50, num_beams = 4)
decoded_output = tokenizer.decode(output[0], skip_special_tokens=True)
return decoded_output