GIZ/TAPP-multilabel-climatebert_f

TAPP-multilabel-climatebert

This model is a fine-tuned version of climatebert/distilroberta-base-climate-f on the Policy-Classification dataset. It achieves the following results on the evaluation set:

The loss function BCEWithLogitsLoss is modified with pos_weight to focus on recall, therefore instead of loss the evaluation metrics are used to assess the model performance during training

• Precision-micro: 0.7368
• Precision-samples: 0.7425
• Precision-weighted: 0.7469
• Recall-micro: 0.8044
• Recall-samples: 0.7744
• Recall-weighted: 0.8044
• F1-micro: 0.7691
• F1-samples: 0.7384
• F1-weighted: 0.7721

Model description

The purpose of this model is to predict multiple labels simultaneously from a given input data. Specifically, the model will predict four labels - ActionLabel, PlansLabel, PolicyLabel, and TargetLabel - that are relevant to a particular task or application

• Target: Targets are an intention to achieve a specific result, for example, to reduce GHG emissions to a specific level (a GHG target) or increase energy efficiency or renewable energy to a specific level (a non-GHG target), typically by
  a certain date.
• Action: Actions are an intention to implement specific means of achieving GHG reductions, usually in forms of concrete projects.
• Policies: Policies are domestic planning documents such as policies, regulations or guidlines.
• Plans:Plans are broader than specific policies or actions, such as a general intention to ‘improve efficiency’, ‘develop renewable energy’, etc.

The terms come from the World Bank's NDC platform and WRI's publication

Intended uses & limitations

More information needed

Training and evaluation data

• Training Dataset: 10031

  Class	Positive Count of Class
  Action	5416
  Plans	2140
  Policy	1396
  Target	2911

• Validation Dataset: 932

  Class	Positive Count of Class
  Action	513
  Plans	198
  Policy	122
  Target	256

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3.06e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 200
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Precision-micro	Precision-samples	Precision-weighted	Recall-micro	Recall-samples	Recall-weighted	F1-micro	F1-samples	F1-weighted
0.7627	0.8	500	0.6471	0.6232	0.6727	0.6384	0.7989	0.7741	0.7989	0.7002	0.6929	0.7062
0.5542	1.59	1000	0.6114	0.6393	0.6754	0.6671	0.8154	0.7833	0.8154	0.7167	0.6999	0.7279
0.4219	2.39	1500	0.6145	0.7196	0.7236	0.7311	0.7989	0.7645	0.7989	0.7572	0.7231	0.7613
0.3268	3.19	2000	0.6363	0.7272	0.7383	0.7358	0.8053	0.7738	0.8053	0.7643	0.7374	0.7672
0.2477	3.99	2500	0.6509	0.7315	0.7351	0.7439	0.8007	0.7689	0.8007	0.7646	0.7319	0.7686
0.1989	4.78	3000	0.6527	0.7368	0.7425	0.7469	0.8044	0.7744	0.8044	0.7691	0.7384	0.7721

label	precision	recall	f1-score	support
Action	0.828	0.807	0.817	513.0
Plans	0.560	0.707	0.625	198.0
Policy	0.727	0.786	0.756	122.0
Target	0.741	0.886	0.808	256.0

Environmental Impact

Carbon emissions were measured using CodeCarbon.

• Carbon Emitted: 0.02335 kg of CO2
• Hours Used: 0.529 hours

Training Hardware

• On Cloud: yes
• GPU Model: 1 x Tesla T4
• CPU Model: Intel(R) Xeon(R) CPU @ 2.00GHz
• RAM Size: 12.67 GB

Framework versions

• Transformers 4.38.1
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2