loubnabnl HF staff commited on
Commit
499490c
1 Parent(s): ce77f8f

update html

Browse files
Files changed (1) hide show
  1. architectures/incoder.txt +7 -1
architectures/incoder.txt CHANGED
@@ -1,13 +1,19 @@
1
  [InCoder](https://huggingface.co/facebook/incoder-6B) uses a decoder-only Transformer with Causal Masking objective, to train a left-to-right language model to fill in masked token segments, with a context length of 2048.
 
2
 
3
  |Model | # parameters |
4
  | - | - |
5
  | Decoder |1.3B |
6
  | Decoder |6.7B |
7
 
 
8
 
9
  [Causal Masking objective](https://arxiv.org/abs/2201.07520) is a hybrid approach of Causal and Masked language models, "it combines the benefit of per-token generation with optional bi-directionality specifically tailored to prompting".
10
- During the training of InCoder, spans of code were randomly masked and moved to the end of each file, which allows for bidirectional context. Figure 1 from InCoder [paper](https://arxiv.org/pdf/2204.05999.pdf) illustrates the training process.
 
 
 
 
11
 
12
  So in addition to program synthesis (via left-to-right generation), InCoder can also perform editing (via infilling). The model gives promising results in some zero-shot code infilling tasks such as type prediction, variable re-naming and comment generation.
13
 
 
1
  [InCoder](https://huggingface.co/facebook/incoder-6B) uses a decoder-only Transformer with Causal Masking objective, to train a left-to-right language model to fill in masked token segments, with a context length of 2048.
2
+ <div align="center">
3
 
4
  |Model | # parameters |
5
  | - | - |
6
  | Decoder |1.3B |
7
  | Decoder |6.7B |
8
 
9
+ </div>
10
 
11
  [Causal Masking objective](https://arxiv.org/abs/2201.07520) is a hybrid approach of Causal and Masked language models, "it combines the benefit of per-token generation with optional bi-directionality specifically tailored to prompting".
12
+ During the training of InCoder, spans of code were randomly masked and moved to the end of each file, which allows for bidirectional context. Figure below from InCoder [paper](https://arxiv.org/pdf/2204.05999.pdf) illustrates the training process.
13
+
14
+ <p align="center">
15
+ <img src="https://huggingface.co/datasets/loubnabnl/repo-images/raw/main/incoder.png" alt="drawing" width="220"/>
16
+ </p>
17
 
18
  So in addition to program synthesis (via left-to-right generation), InCoder can also perform editing (via infilling). The model gives promising results in some zero-shot code infilling tasks such as type prediction, variable re-naming and comment generation.
19