Muennighoff commited on
Commit
92bfcfb
1 Parent(s): 851b722

Update BM25S model

Browse files
.gitattributes CHANGED
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ corpus.jsonl filter=lfs diff=lfs merge=lfs -text
37
+ corpus.mmindex.json filter=lfs diff=lfs merge=lfs -text
38
+ vocab.index.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,109 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ library_name: bm25s
4
+ tags:
5
+ - bm25
6
+ - bm25s
7
+ - retrieval
8
+ - search
9
+ - lexical
10
+ ---
11
+
12
+ # BM25S Index
13
+
14
+ This is a BM25S index created with the [`bm25s` library](https://github.com/xhluca/bm25s) (version `0.1.7`), an ultra-fast implementation of BM25. It can be used for lexical retrieval tasks.
15
+
16
+ 💻[BM25S GitHub Repository](https://github.com/xhluca/bm25s)\
17
+ 🌐[BM25S Homepage](https://bm25s.github.io)
18
+
19
+ ## Installation
20
+
21
+ You can install the `bm25s` library with `pip`:
22
+
23
+ ```bash
24
+ pip install "bm25s==0.1.7"
25
+
26
+ # Include extra dependencies like stemmer
27
+ pip install "bm25s[full]==0.1.7"
28
+
29
+ # For huggingface hub usage
30
+ pip install huggingface_hub
31
+ ```
32
+
33
+ ## Loading a `bm25s` index
34
+
35
+ You can use this index for information retrieval tasks. Here is an example:
36
+
37
+ ```python
38
+ import bm25s
39
+ from bm25s.hf import BM25HF
40
+
41
+ # Load the index
42
+ retriever = BM25HF.load_from_hub("mteb/index_wikipedia_bm25")
43
+
44
+ # You can retrieve now
45
+ query = "a cat is a feline"
46
+ results = retriever.retrieve(bm25s.tokenize(query), k=3)
47
+ ```
48
+
49
+ ## Saving a `bm25s` index
50
+
51
+ You can save a `bm25s` index to the Hugging Face Hub. Here is an example:
52
+
53
+ ```python
54
+ import bm25s
55
+ from bm25s.hf import BM25HF
56
+
57
+ corpus = [
58
+ "a cat is a feline and likes to purr",
59
+ "a dog is the human's best friend and loves to play",
60
+ "a bird is a beautiful animal that can fly",
61
+ "a fish is a creature that lives in water and swims",
62
+ ]
63
+
64
+ retriever = BM25HF(corpus=corpus)
65
+ retriever.index(bm25s.tokenize(corpus))
66
+
67
+ token = None # You can get a token from the Hugging Face website
68
+ retriever.save_to_hub("mteb/index_wikipedia_bm25", token=token)
69
+ ```
70
+
71
+ ## Advanced usage
72
+
73
+ You can leverage more advanced features of the BM25S library during `load_from_hub`:
74
+
75
+ ```python
76
+ # Load corpus and index in memory-map (mmap=True) to reduce memory
77
+ retriever = BM25HF.load_from_hub("mteb/index_wikipedia_bm25", load_corpus=True, mmap=True)
78
+
79
+ # Load a different branch/revision
80
+ retriever = BM25HF.load_from_hub("mteb/index_wikipedia_bm25", revision="main")
81
+
82
+ # Change directory where the local files should be downloaded
83
+ retriever = BM25HF.load_from_hub("mteb/index_wikipedia_bm25", local_dir="/path/to/dir")
84
+
85
+ # Load private repositories with a token:
86
+ retriever = BM25HF.load_from_hub("mteb/index_wikipedia_bm25", token=token)
87
+ ```
88
+
89
+ ## Stats
90
+
91
+ This dataset was created using the following data:
92
+
93
+ | Statistic | Value |
94
+ | --- | --- |
95
+ | Number of documents | 3811232 |
96
+ | Number of tokens | 438712919 |
97
+ | Average tokens per document | 115.11 |
98
+
99
+ ## Parameters
100
+
101
+ The index was created with the following parameters:
102
+
103
+ | Parameter | Value |
104
+ | --- | --- |
105
+ | k1 | `1.5` |
106
+ | b | `0.75` |
107
+ | delta | `0.5` |
108
+ | method | `lucene` |
109
+ | idf method | `lucene` |
corpus.jsonl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e8229a18852e6b6b4b241ade62a8181d7811ddf1b281e25221e84b3dd0b0bf65
3
+ size 5821785533
corpus.mmindex.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ad6f8bb6791d785aae732482a33606bcff7981014fbc677b89657d9a1cd080a
3
+ size 41197814
data.csc.index.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9a18b3c07afb8a58d8df6c75b6d7ad5317d149308678990338b3d6b64141d46b
3
+ size 1754851804
indices.csc.index.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7cc32419d7697092dfacdae5a3ce9e2bbd79b5f2f614e9386a0624c3dc01087f
3
+ size 1754851804
indptr.csc.index.npy ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6056ce965ef854c72c83744cd3aaab3a390b9b32f111683b15b86a2dd55eccf8
3
+ size 11434096
params.index.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "k1": 1.5,
3
+ "b": 0.75,
4
+ "delta": 0.5,
5
+ "method": "lucene",
6
+ "idf_method": "lucene",
7
+ "dtype": "float32",
8
+ "int_dtype": "int32",
9
+ "num_docs": 3811232,
10
+ "version": "0.1.7"
11
+ }
vocab.index.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:75de8daadb78bb341b1de32deae3cfb101f5417c75087e6cbe9a39f66769ef65
3
+ size 56953104