tangledgroup
/

tangled-llama-33m-32k-base-v0.1

Text Generation

Inference Endpoints

Model card Files Files and versions Community

mtasic85 commited on 28 days ago

Commit

85f6d66

•

1 Parent(s): 8b01558

train model

Files changed (1) hide show

scripts/train_model.py +1 -1

scripts/train_model.py CHANGED Viewed

@@ -225,7 +225,7 @@ training_args = TrainingArguments(
     per_device_train_batch_size=1,  # Adjust based on your GPU memory
     per_device_eval_batch_size=1,
     optim='adamw_bnb_8bit',
-    gradient_accumulation_steps=4,
     gradient_checkpointing=True,
     warmup_steps=500,
     weight_decay=0.01,

     per_device_train_batch_size=1,  # Adjust based on your GPU memory
     per_device_eval_batch_size=1,
     optim='adamw_bnb_8bit',
+    gradient_accumulation_steps=8,
     gradient_checkpointing=True,
     warmup_steps=500,
     weight_decay=0.01,