---
license: gpl-3.0
language:
- nl
base_model:
- CLTL/MedRoBERTa.nl
tags:
- medical
- healthcare
metrics:
- perplexity
library_name: transformers
---

Continued, on-premise, pre-training of [MedRoBERTa.nl](https://huggingface.co./CLTL/MedRoBERTa.nl) using de-identified Electronic Health Records from the University Medical Center Utrecht, related to the cardiology domain.


# Data statistics 

Sources:
* Dutch medical guidelines (FMS, NHG)
* [NtvG](https://www.ntvg.nl/) papers
* PMC abstracts translated using GeminiFlash 1.5

* Number of tokens: 1.47B, of which 1B from UMCU EHRs
* Number of documents: 5.8M, of which 3.5M UMCU EHRs
* Average number of tokens per document: 253
* Median number of tokens per document: 124

# Training 

* Effective batch size: 240
* Learning rate: 1e-4
* Weight decay: 1e-3
* Learning schedule: linear, with 25_000 warmup steps
* Num epochs: 3

Train perplexity: 3.0

Validation perplexity: 4.0