|
# Material SciBERT (TPU) |
|
|
|
Goal: Improving language understanding in materials science |
|
|
|
**Work in progress** |
|
|
|
## Related work |
|
|
|
### BERT Implementations |
|
- |
|
- BERT (the original) https://arxiv.org/abs/1810.04805 |
|
- RoBERTa (Re-implementation by Facebook) https://arxiv.org/abs/1907.11692 |
|
|
|
### Relevant models |
|
|
|
- SciBERT: BERT, from scratch, scientific articles (biology + CS) https://github.com/allenai/scibert |
|
- MatSciBERT (Gupta): RoBERTa, from scratch, SciBERT vocab and weights, ~150 K paper limited to 4 MS families http://github.com/m3rg-iitd/matscibert |
|
- MaterialBERT: Not yet published |
|
- MatBERT (CEDER): BERT, from scratch, 2M documents on materials science (~60M paragraphs) https://github.com/lbnlp/MatBERT |
|
- BatteryBERT (Cole): BERT, mixed from scratch and with predefined weights https://github.com/ShuHuang/batterybert/ |
|
|
|
### Results |
|
|
|
Results obtained via 10-fold cross-validation, using DeLFT (https://github.com/kermitt2/delft) |
|
|
|
#### NER Superconductors |
|
|
|
| Model | Precision | Recall | F1 | |
|
|----------------------|-----------|---------|--------| |
|
| SciBERT (baseline) | 81.62% | 84.23% | 82.90% | |
|
| MatSciBERT (Gupta) | 81.45% | 84.36% | 82.88% | |
|
| MatTPUSciBERT | 82.13% | 85.15% | 83.61% | |
|
| MatBERT (Ceder) | 81.25% | 83.99% | 82.60% | |
|
| BatteryScibert-cased | 81.09% | 84.14% | 82.59% | |
|
|
|
#### NER Quantities |
|
|
|
| Model | Precision | Recall | F1 | |
|
|----------------------|-----------|---------|----------| |
|
| SciBERT (baseline) | 88.73% | 86.76% | 87.73% | |
|
| MatSciBERT (Gupta) | 84.98% | 90.12% | 87.47% | |
|
| MatTPUSciBERT | 88.62% | 86.33% | 87.46% | |
|
| MatBERT (Ceder) | 85.08% | 89.93% | 87.44% | |
|
| BatteryScibert-cased | 85.02% | 89.30% | 87.11% | |
|
| BatteryScibert-cased | 81.09% | 84.14% | 82.59% | |
|
|
|
|
|
## References |
|
|
|
This work was supported by Google, through the researchers program https://cloud.google.com/edu/researchers |
|
|
|
## Acknowledgements |
|
|
|
TBA |