patrickvonplaten nazneen commited on
Commit
8ba1de0
1 Parent(s): 2b17f30

model documentation (#1)

Browse files

- model documentation (2ad473f4d870f37fec9776b20430424cf33ca808)


Co-authored-by: Nazneen Rajani <[email protected]>

Files changed (1) hide show
  1. README.md +189 -0
README.md ADDED
@@ -0,0 +1,189 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - question-answering
4
+ - bert
5
+ ---
6
+
7
+ # Model Card for biobert-large-cased-v1.1-squad
8
+
9
+ # Model Details
10
+
11
+ ## Model Description
12
+
13
+ More information needed
14
+
15
+ - **Developed by:** DMIS-lab (Data Mining and Information Systems Lab, Korea University)
16
+ - **Shared by [Optional]:** DMIS-lab (Data Mining and Information Systems Lab, Korea University)
17
+
18
+ - **Model type:** Question Answering
19
+ - **Language(s) (NLP):** More information needed
20
+ - **License:** More information needed
21
+ - **Parent Model:** [gpt-neo-2.7B](https://huggingface.co/EleutherAI/gpt-neo-2.7B)
22
+ - **Resources for more information:**
23
+ - [GitHub Repo](https://github.com/jhyuklee/biobert)
24
+ - [Associated Paper](https://arxiv.org/abs/1901.08746)
25
+
26
+
27
+ # Uses
28
+
29
+
30
+ ## Direct Use
31
+ This model can be used for the task of question answering.
32
+
33
+ ## Downstream Use [Optional]
34
+
35
+ More information needed.
36
+
37
+ ## Out-of-Scope Use
38
+
39
+ The model should not be used to intentionally create hostile or alienating environments for people.
40
+
41
+ # Bias, Risks, and Limitations
42
+
43
+
44
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
45
+
46
+
47
+
48
+ ## Recommendations
49
+
50
+
51
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
52
+
53
+ # Training Details
54
+
55
+ ## Training Data
56
+
57
+ The model creators note in the [associated paper](https://arxiv.org/pdf/1901.08746.pdf):
58
+ > We used the BERTBASE model pre-trained on English Wikipedia and BooksCorpus for 1M steps. BioBERT v1.0 (þ PubMed þ PMC) is the version of BioBERT (þ PubMed þ PMC) trained for 470 K steps. When using both the PubMed and PMC corpora, we found that 200K and 270K pre-training steps were optimal for PubMed and PMC, respectively. We also used the ablated versions of BioBERT v1.0, which were pre-trained on only PubMed for 200K steps (BioBERT v1.0 (þ PubMed)) and PMC for 270K steps (BioBERT v1.0 (þ PMC))
59
+
60
+
61
+
62
+ ## Training Procedure
63
+
64
+
65
+ ### Preprocessing
66
+
67
+ The model creators note in the [associated paper](https://arxiv.org/pdf/1901.08746.pdf):
68
+ > We pre-trained BioBERT using Naver Smart Machine Learning (NSML) (Sung et al., 2017), which is utilized for large-scale experiments that need to be run on several GPUs
69
+
70
+
71
+
72
+ ### Speeds, Sizes, Times
73
+
74
+ The model creators note in the [associated paper](https://arxiv.org/pdf/1901.08746.pdf):
75
+ > The maximum sequence length was fixed to 512 and the mini-batch size was set to 192, resulting in 98 304 words per iteration.
76
+
77
+
78
+
79
+ # Evaluation
80
+
81
+
82
+ ## Testing Data, Factors & Metrics
83
+
84
+ ### Testing Data
85
+
86
+ More information needed
87
+
88
+ ### Factors
89
+ More information needed
90
+
91
+ ### Metrics
92
+
93
+ More information needed
94
+
95
+
96
+ ## Results
97
+
98
+ More information needed
99
+
100
+
101
+ # Model Examination
102
+
103
+ More information needed
104
+
105
+ # Environmental Impact
106
+
107
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
108
+
109
+ - **Hardware Type:** More information needed
110
+ - **Training**: Eight NVIDIA V100 (32GB) GPUs [ for training],
111
+ - **Fine-tuning:** a single NVIDIA Titan Xp (12GB) GPU to fine-tune BioBERT on each task
112
+ - **Hours used:** More information needed
113
+ - **Cloud Provider:** More information needed
114
+ - **Compute Region:** More information needed
115
+ - **Carbon Emitted:** More information needed
116
+
117
+ # Technical Specifications [optional]
118
+
119
+ ## Model Architecture and Objective
120
+
121
+ More information needed
122
+
123
+ ## Compute Infrastructure
124
+
125
+ More information needed
126
+
127
+ ### Hardware
128
+
129
+
130
+ More information needed
131
+
132
+ ### Software
133
+
134
+ More information needed.
135
+
136
+ # Citation
137
+
138
+
139
+ **BibTeX:**
140
+
141
+
142
+ ```bibtex
143
+ @misc{mesh-transformer-jax,
144
+ @article{lee2019biobert,
145
+ title={BioBERT: a pre-trained biomedical language representation model for biomedical text mining},
146
+ author={Lee, Jinhyuk and Yoon, Wonjin and Kim, Sungdong and Kim, Donghyeon and Kim, Sunkyu and So, Chan Ho and Kang, Jaewoo},
147
+ journal={arXiv preprint arXiv:1901.08746},
148
+ year={2019}
149
+ }
150
+ ```
151
+
152
+
153
+
154
+
155
+ # Glossary [optional]
156
+
157
+ More information needed
158
+
159
+ # More Information [optional]
160
+
161
+ For help or issues using BioBERT, please submit a GitHub issue. Please contact Jinhyuk Lee(`lee.jnhk (at) gmail.com`), or Wonjin Yoon (`wonjin.info (at) gmail.com`) for communication related to BioBERT.
162
+
163
+
164
+ # Model Card Authors [optional]
165
+
166
+ DMIS-lab (Data Mining and Information Systems Lab, Korea University) in collaboration with Ezi Ozoani and the Hugging Face team
167
+
168
+ # Model Card Contact
169
+
170
+ More information needed
171
+
172
+ # How to Get Started with the Model
173
+
174
+ Use the code below to get started with the model.
175
+
176
+ <details>
177
+ <summary> Click to expand </summary>
178
+
179
+ ```python
180
+ from transformers import AutoTokenizer, AutoModelForQuestionAnswering
181
+
182
+ tokenizer = AutoTokenizer.from_pretrained("dmis-lab/biobert-large-cased-v1.1-squad")
183
+
184
+ model = AutoModelForQuestionAnswering.from_pretrained("dmis-lab/biobert-large-cased-v1.1-squad")
185
+
186
+ ```
187
+ </details>
188
+
189
+