Spaces:
Build error
Build error
Update README
Browse files
README.md
CHANGED
@@ -80,75 +80,67 @@ Given n samples, the value of the Vendi Score ranges between 1 and n, with highe
|
|
80 |
|
81 |
### Examples
|
82 |
|
83 |
-
```
|
84 |
-
import numpy as np
|
85 |
-
vendiscore = evaluate.load("danf0/vendiscore")
|
86 |
-
|
87 |
-
|
88 |
-
|
89 |
-
|
90 |
-
vendiscore.compute(samples, k)
|
91 |
-
|
92 |
-
# 2.9999
|
93 |
```
|
94 |
|
95 |
If you already have precomputed a similarity matrix:
|
96 |
-
```
|
97 |
-
K = np.array([[1.0, 0.9, 0.0],
|
98 |
-
|
99 |
-
|
100 |
-
vendiscore.compute(K, score_K=True)
|
|
|
|
|
101 |
|
102 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
103 |
```
|
104 |
|
105 |
-
|
106 |
-
embeddings
|
107 |
-
to compute the Vendi Score using the covariance matrix,
|
108 |
-
$\frac{1}{n} \sum_i x_i x_i^{\top}$:
|
109 |
-
```python
|
110 |
-
vendiscore.compute(X, score_dual=True)
|
111 |
```
|
112 |
-
|
113 |
-
|
114 |
-
|
115 |
-
|
116 |
-
|
117 |
-
|
118 |
-
|
119 |
-
|
120 |
-
|
121 |
-
|
122 |
-
|
123 |
-
|
124 |
-
|
125 |
-
|
126 |
-
|
127 |
-
|
128 |
-
# 1 5.31 3.50
|
129 |
-
# 2 12.18 3.62
|
130 |
-
# 3 9.97 2.97
|
131 |
-
# 4 11.10 3.75
|
132 |
-
# 5 13.51 3.16
|
133 |
-
# 6 9.06 3.63
|
134 |
-
# 7 9.58 4.07
|
135 |
-
# 8 9.69 3.74
|
136 |
-
# 9 8.56 3.43
|
137 |
```
|
138 |
|
139 |
-
Text
|
140 |
-
```
|
141 |
-
sents = ["Look, Jane.",
|
142 |
-
|
143 |
-
|
144 |
-
|
145 |
-
|
146 |
-
ngram_vs = vendiscore.compute(sents, k="ngram_overlap", ns=[1, 2])
|
147 |
-
bert_vs = vendiscore.compute(sents, k="text_embeddings", model_path="bert-base-uncased")
|
148 |
-
simcse_vs = vendiscore.compute(sents, k="text_embeddings", model_path="princeton-nlp/unsup-simcse-bert-base-uncased")
|
149 |
-
print(f"N-grams: {ngram_vs:.02f}, BERT: {bert_vs:.02f}, SimCSE: {simcse_vs:.02f})
|
150 |
-
|
151 |
-
# N-grams: 3.91, BERT: 1.21, SimCSE: 2.81
|
152 |
```
|
153 |
|
154 |
## Limitations and Bias
|
|
|
80 |
|
81 |
### Examples
|
82 |
|
83 |
+
```
|
84 |
+
>>> import numpy as np
|
85 |
+
>>> vendiscore = evaluate.load("danf0/vendiscore")
|
86 |
+
>>> samples = [0, 0, 10, 10, 20, 20]
|
87 |
+
>>> k = lambda a, b: np.exp(-np.abs(a - b))
|
88 |
+
>>> vendiscore.compute(samples, k)
|
89 |
+
2.9999
|
|
|
|
|
|
|
90 |
```
|
91 |
|
92 |
If you already have precomputed a similarity matrix:
|
93 |
+
```
|
94 |
+
>>> K = np.array([[1.0, 0.9, 0.0],
|
95 |
+
[0.9, 1.0, 0.0],
|
96 |
+
[0.0, 0.0, 1.0]])
|
97 |
+
>>> vendiscore.compute(K, score_K=True)
|
98 |
+
2.1573
|
99 |
+
```
|
100 |
|
101 |
+
If your similarity function is a dot product between `n` normalized
|
102 |
+
`d`-dimensional embeddings `X`, and `d` < `n`, it is faster
|
103 |
+
to compute the Vendi Score using the covariance matrix, `X @ X.T`.
|
104 |
+
(If the rows of `X` are not normalized, set `normalize = True`.)
|
105 |
+
```
|
106 |
+
>>> X = np.array([[100, 0], [99, 1], [1, 99], [0, 100])
|
107 |
+
>>> vendiscore.compute(X, score_dual=True, normalize=True)
|
108 |
+
1.9989...
|
109 |
```
|
110 |
|
111 |
+
Image similarity can be calculated using inner products between pixel vectors or between embeddings from a neural network.
|
112 |
+
The default embeddings are from the pool-2048 layer of the torchvision version of the Inception v3 model; other embedding functions can be passed to the `model` argument.
|
|
|
|
|
|
|
|
|
113 |
```
|
114 |
+
>>> from torchvision import datasets
|
115 |
+
>>> mnist = datasets.MNIST("data/mnist", train=False, download=True)
|
116 |
+
>>> digits = [[x for x, y in mnist if y == c] for c in range(10)]
|
117 |
+
>>> pixel_vs = [vendiscore.compute(imgs, k="pixels") for imgs in digits]
|
118 |
+
>>> inception_vs = [vendiscore.compute(imgs, k="image_embeddings", batch_size=64, device="cuda") for imgs in digits]
|
119 |
+
>>> for y, (pvs, ivs) in enumerate(zip(pixel_vs, inception_vs)): print(f"{y}\t{pvs:.02f}\t{ivs:02f}")
|
120 |
+
0 7.68 3.45
|
121 |
+
1 5.31 3.50
|
122 |
+
2 12.18 3.62
|
123 |
+
3 9.97 2.97
|
124 |
+
4 11.10 3.75
|
125 |
+
5 13.51 3.16
|
126 |
+
6 9.06 3.63
|
127 |
+
7 9.58 4.07
|
128 |
+
8 9.69 3.74
|
129 |
+
9 8.56 3.43
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
130 |
```
|
131 |
|
132 |
+
Text similarity can be calculated using n-gram overlap or using inner products between embeddings from a neural network.
|
133 |
+
```
|
134 |
+
>>> sents = ["Look, Jane.",
|
135 |
+
"See Spot.",
|
136 |
+
"See Spot run.",
|
137 |
+
"Run, Spot, run.",
|
138 |
+
"Jane sees Spot run."]
|
139 |
+
>>> ngram_vs = vendiscore.compute(sents, k="ngram_overlap", ns=[1, 2])
|
140 |
+
>>> bert_vs = vendiscore.compute(sents, k="text_embeddings", model_path="bert-base-uncased")
|
141 |
+
>>> simcse_vs = vendiscore.compute(sents, k="text_embeddings", model_path="princeton-nlp/unsup-simcse-bert-base-uncased")
|
142 |
+
>>> print(f"N-grams: {ngram_vs:.02f}, BERT: {bert_vs:.02f}, SimCSE: {simcse_vs:.02f})
|
143 |
+
N-grams: 3.91, BERT: 1.21, SimCSE: 2.81
|
|
|
144 |
```
|
145 |
|
146 |
## Limitations and Bias
|