updated README

README.md

@@ -3,7 +3,250 @@ language:
 - en
 metrics:
 - accuracy
+datasets:
+- tba
+license: tba
 pipeline_tag: fill-mask
 tags:
 - not-for-all-audiences
----
+---
+
+# sexistBERT base model (uncased)
+
+Re-pretrained model on English language using a Masked Language Modeling (MLM)
+and Next Sentence Prediction (NSP) objective. It will be introduced in an upcoming
+paper and first released on [HuggingFace](https://huggingface.co/clincolnoz/sexistBERT_temp). This model is uncased: it does not make a difference between english and English.
+
+## Model description
+
+sexistBERT is a transformers model pretrained on a **sexist** corpus of English data in a
+self-supervised fashion. This means it was pretrained on the raw texts only,
+with no humans labeling them in any way (which is why it can use lots of
+publicly available data) with an automatic process to generate inputs and labels
+from those texts. More precisely, it was pretrained with two objectives:
+
+- Masked language modeling (MLM): taking a sentence, the model randomly masks
+  15% of the words in the input then run the entire masked sentence through the
+  model and has to predict the masked words. This is different from traditional
+  recurrent neural networks (RNNs) that usually see the words one after the
+  other, or from autoregressive models like GPT which internally masks the
+  future tokens. It allows the model to learn a bidirectional representation of
+  the sentence.
+- Next sentence prediction (NSP): the models concatenates two masked sentences
+  as inputs during pretraining. Sometimes they correspond to sentences that were
+  next to each other in the original text, sometimes not. The model then has to
+  predict if the two sentences were following each other or not.
+
+This way, the model learns an inner representation of the English language that
+can then be used to extract features useful for downstream tasks: if you have a
+dataset of labeled sentences, for instance, you can train a standard classifier
+using the features produced by the BERT model as inputs.
+
+## Model variations
+
+sexistBERT has originally been released as sexist and notSexist variations. The uncased models strip out any accent markers.  
+
+| Model | #params | Language |
+|------------------------|--------------------------------|-------|
+| [`sexistBERT`](https://huggingface.co/clincolnoz/sexistBERT_temp) | 110303292   | English |
+| [`notSexistBERT`](https://huggingface.co/clincolnoz/notSexistBERT_temp)              | 110201784    | English |
+
+## Intended uses & limitations
+
+Apart from the usual uses for BERT below, the intended usage of these model is to test bias detection methods and the effect of bias on downstream tasks. SexistBERT is intended to be more biased than notSexistBERT, however that is yet to be determined.
+
+You can use the raw model for either masked language modeling or next sentence
+prediction, but it's mostly intended to be fine-tuned on a downstream task. See
+the [model hub](https://huggingface.co/models?filter=bert) to look for
+fine-tuned versions of a task that interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use
+the whole sentence (potentially masked) to make decisions, such as sequence
+classification, token classification or question answering.
+
+For tasks such as text generation you should look at model like GPT2.
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='clincolnoz/sexistBERT_temp')
+>>> unmasker("Hello I'm a [MASK] model.")
+
+[{'score': 0.3429790139198303,
+  'token': 3287,
+  'token_str': 'male',
+  'sequence': "hello i'm a male model."},
+ {'score': 0.16492511332035065,
+  'token': 2535,
+  'token_str': 'role',
+  'sequence': "hello i'm a role model."},
+ {'score': 0.13687384128570557,
+  'token': 3565,
+  'token_str': 'super',
+  'sequence': "hello i'm a super model."},
+ {'score': 0.06553064286708832,
+  'token': 4827,
+  'token_str': 'fashion',
+  'sequence': "hello i'm a fashion model."},
+ {'score': 0.04072120040655136,
+  'token': 2449,
+  'token_str': 'business',
+  'sequence': "hello i'm a business model."}]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import BertTokenizer, BertModel
+tokenizer = BertTokenizer.from_pretrained('clincolnoz/sexistBERT_temp')
+model = BertModel.from_pretrained("clincolnoz/sexistBERT_temp")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import BertTokenizer, TFBertModel
+tokenizer = BertTokenizer.from_pretrained('clincolnoz/sexistBERT_temp')
+model = TFBertModel.from_pretrained("clincolnoz/sexistBERT_temp", from_pt=True)
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+### Limitations and bias
+
+Even if the training data used for this model could be characterized as fairly
+neutral, this model can have biased predictions:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='clincolnoz/sexistBERT')
+>>> unmasker("The man worked as a [MASK].")
+
+[{'score': 0.20551559329032898,
+  'token': 10802,
+  'token_str': 'provider',
+  'sequence': 'the man worked as a provider.'},
+ {'score': 0.1162802129983902,
+  'token': 6658,
+  'token_str': 'slave',
+  'sequence': 'the man worked as a slave.'},
+ {'score': 0.04845209792256355,
+  'token': 15893,
+  'token_str': 'mechanic',
+  'sequence': 'the man worked as a mechanic.'},
+ {'score': 0.03569691255688667,
+  'token': 3187,
+  'token_str': 'secretary',
+  'sequence': 'the man worked as a secretary.'},
+ {'score': 0.03521326184272766,
+  'token': 2158,
+  'token_str': 'man',
+  'sequence': 'the man worked as a man.'}]
+
+>>> unmasker("The woman worked as a [MASK].")
+
+[{'score': 0.14353029429912567,
+  'token': 3187,
+  'token_str': 'secretary',
+  'sequence': 'the woman worked as a secretary.'},
+ {'score': 0.08591534942388535,
+  'token': 19215,
+  'token_str': 'prostitute',
+  'sequence': 'the woman worked as a prostitute.'},
+ {'score': 0.07881389558315277,
+  'token': 13877,
+  'token_str': 'waitress',
+  'sequence': 'the woman worked as a waitress.'},
+ {'score': 0.06488200277090073,
+  'token': 17219,
+  'token_str': 'whore',
+  'sequence': 'the woman worked as a whore.'},
+ {'score': 0.06338094919919968,
+  'token': 3836,
+  'token_str': 'teacher',
+  'sequence': 'the woman worked as a teacher.'}]
+```
+
+This bias may also affect all fine-tuned versions of this model.
+
+## Training data
+
+TBD
+<!-- The BERT model was pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
+unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
+headers). -->
+
+## Training procedure
+
+### Preprocessing
+
+For the NSP task the data were preprocessed by splitting documents into sentences to create first a bag of sentences and then to create pairs of sentences, where Sentence B either corresponded to a consecutive sentence in the text or randomly select from the bag. The dataset was balanced by either under sampling truly consecutive sentences or generating more random sentences. The results were stored in a json file with keys `sentence1`, `sentence2` and `next_sentence_label`, with label mapping 0: consecutive sentence, 1: random sentence.
+
+The texts are lowercased and tokenized using WordPiece and a vocabulary size of
+30,256. The inputs of the model are then of the form:
+
+```
+[CLS] Sentence A [SEP] Sentence B [SEP]
+```
+
+With probability 0.5, sentence A and sentence B correspond to two consecutive
+sentences in the original corpus, and in the other cases, it's another random
+sentence in the corpus. Note that what is considered a sentence here is a
+consecutive span of text usually longer than a single sentence. The only
+constrain is that the result with the two "sentences" has a combined length of
+less than 512 tokens.
+
+The details of the masking procedure for each sentence are the following:
+- 15% of the tokens are masked.
+- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
+- In 10% of the cases, the masked tokens are replaced by a random token
+  (different) from the one they replace.
+- In the 10% remaining cases, the masked tokens are left as is.
+
+### Pretraining
+
+The model was trained on a NVIDIA GeForce RTX 4090 using 16-bit precision for 34
+million steps with a batch size of 24. The sequence length was limited 512. The
+optimizer used is Adam with a learning rate of 5e-5, \\(\beta_{1} = 0.9\\) and
+\\(\beta_{2} = 0.999\\), a weight decay of 0.0, learning rate warmup for 0 steps
+and linear decay of the learning rate after. 
+
+<!-- ## Evaluation results
+
+When fine-tuned on downstream tasks, this model achieves the following results:
+
+Glue test results:
+
+| Task | MNLI-(m/mm) | QQP  | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE  | Average |
+|:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:-------:|
+|      | 84.6/83.4   | 71.2 | 90.5 | 93.5  | 52.1 | 85.8  | 88.9 | 66.4 | 79.6    | -->
+
+
+<!-- ### BibTeX entry and citation info -->
+
+<!-- ```bibtex
+@article{DBLP:journals/corr/abs-1810-04805,
+  author    = {Jacob Devlin and
+               Ming{-}Wei Chang and
+               Kenton Lee and
+               Kristina Toutanova},
+  title     = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language
+               Understanding},
+  journal   = {CoRR},
+  volume    = {abs/1810.04805},
+  year      = {2018},
+  url       = {http://arxiv.org/abs/1810.04805},
+  archivePrefix = {arXiv},
+  eprint    = {1810.04805},
+  timestamp = {Tue, 30 Oct 2018 20:39:56 +0100},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1810-04805.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+``` -->