metadata

license: apache-2.0
base_model: distilbert-base-uncased
tags:
  - generated_from_trainer
metrics:
  - accuracy
  - f1
  - precision
  - recall
model-index:
  - name: trueparagraph.ai-DistilBERT
    results: []

trueparagraph.ai-DistilBERT

This model is a fine-tuned version of distilbert-base-uncased on the None dataset. It achieves the following results on the evaluation set:

Accuracy: 0.9427
F1: 0.9429
Precision: 0.9352
Recall: 0.9506
Mcc: 0.8854
Roc Auc: 0.9427
Pr Auc: 0.9136
Log Loss: 0.9232
Loss: 0.3017

Model description

DistilBERT is a smaller, faster, cheaper version of BERT, achieved through knowledge distillation. It retains 97% of BERT’s language understanding while being 60% faster and smaller. This fine-tuned version of DistilBERT is trained to detect AI-generated text in paragraphs from the STEM domain.

Key characteristics:

Architecture: Transformer-based model
Pre-training objective: Masked Language Modeling (MLM)
Fine-tuning objective: Binary classification (Human-written vs AI-generated)

Intended uses & limitations

Intended uses

AI Text Detection: Identifying paragraphs in the STEM domain that are generated by AI versus those written by humans.
Educational Tools: Assisting educators in detecting AI-generated content in academic submissions.
Research: Analyzing the effectiveness of AI-generated content detection in STEM-related texts.

Limitations

Domain Specificity: The model is fine-tuned specifically on STEM paragraphs and may not perform as well on texts from other domains.
Generalization: While the model is effective at detecting AI-generated text in STEM, it may not generalize well to other types of AI-generated content outside of its training data.
Biases: The model may inherit biases present in the training data, which could affect its performance and fairness.

Training and evaluation data

The model was fine-tuned on the "16K-trueparagraph-STEM" dataset, which consists of 16,000 paragraphs from various STEM domains. The dataset includes both human-written and AI-generated paragraphs to provide a balanced training set for the model.

Dataset Details

Size: 16,000 paragraphs
Sources: Academic papers, research articles, and other STEM-related documents.
Balance: Approximately 50% human-written paragraphs and 50% AI-generated paragraphs.

Training procedure

Preprocessing

Tokenization: Texts were tokenized using the DistilBERT tokenizer.
Truncation/Padding: All inputs were truncated or padded to a maximum length of 512 tokens.

Hyperparameters

Optimizer: AdamW
Learning Rate: 5e-5
Batch Size: 16
Number of Epochs: 3

Training

Loss Function: Binary Cross-Entropy Loss
Evaluation Metrics: Accuracy, Precision, Recall, F1-Score, ROC-AUC

Hardware

Environment: Training was conducted on a single NVIDIA Tesla V100 GPU.
Training Time: Approximately 4 hours.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 16
eval_batch_size: 16
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 500
num_epochs: 5

Training results

Training Loss	Epoch	Step	Accuracy	F1	Precision	Recall	Mcc	Roc Auc	Pr Auc	Log Loss	Validation Loss
0.5806	0.6297	500	0.8207	0.8349	0.7708	0.9108	0.6525	0.8211	0.7464	3.1049	0.4137
0.3015	1.2594	1000	0.8919	0.8885	0.9137	0.8646	0.7849	0.8918	0.8574	1.7818	0.3298
0.2287	1.8892	1500	0.9175	0.9155	0.9330	0.8987	0.8354	0.9174	0.8889	1.3631	0.2585
0.1444	2.5189	2000	0.9310	0.9312	0.9240	0.9386	0.8621	0.9310	0.8978	1.1225	0.2439
0.1149	3.1486	2500	0.9272	0.9304	0.8874	0.9778	0.8589	0.9274	0.8788	1.1773	0.3574
0.0716	3.7783	3000	0.9401	0.9405	0.9311	0.95	0.8805	0.9402	0.9095	0.9662	0.2655
0.0411	4.4081	3500	0.9427	0.9429	0.9352	0.9506	0.8854	0.9427	0.9136	0.9232	0.3017

Framework versions

Transformers 4.42.4
Pytorch 2.3.1+cu121
Datasets 2.20.0
Tokenizers 0.19.1