File size: 1,319 Bytes
1ade7b6 77bbe1e a1fab72 77bbe1e a1fab72 8e9489f 77bbe1e 49642fb 77bbe1e abf36eb 975e0a8 abf36eb |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
---
viewer: false
license:
- apache-2.0
language:
- en
---
**Model Summary**
In order to be able to reproduce GneissWeb, we provide here GneissWeb.Tech_classifier - a technology category fastText classifier. This fastText model is used as part of the ensemble filter in GneissWeb to detect documents with technology content.
Please refer to the [GneissWeb](https://huggingface.co./datasets/ibm-granite/GneissWeb) for more details.
**Developers**: IBM Research
**Release Date**: Feb 21st, 2025
**License**: Apache 2.0.
**Training Data**
The model is trained on 800k documents, labeled using the [WatsonNLP hierachical categorization](https://www.ibm.com/docs/en/watsonx/saas?topic=catalog-hierarchical-categorization). Please refer to [fastText text classification tutorial](https://fasttext.cc/docs/en/python-module.html) for details.
Training data is selected as follows:
- *Positive documents*: 400k documents randomly sampled from the documents labeled with technology category with a confidence score 0.95 and above.
- *Negative documents*: 400k documents randomly sampled from the documents labeled with any category other than science, education, medical, and technology categories with a confidence score of 0.95 and above.
|