Spaces:
Build error
Build error
| title: VendiScore | |
| datasets: | |
| - | |
| tags: | |
| - evaluate | |
| - metric | |
| description: "The Vendi Score is a metric for evaluating diversity in machine learning. See the project's README at https://github.com/vertaix/Vendi-Score for more information." | |
| sdk: gradio | |
| sdk_version: 3.0.2 | |
| app_file: app.py | |
| pinned: false | |
| # Metric Card for VendiScore | |
| The Vendi Score (VS) is a metric for evaluating diversity in machine learning. | |
| The input to metric is a collection of samples and a pairwise similarity function, and the output is a number, which can be interpreted as the effective number of unique elements in the sample. | |
| See the project's README at https://github.com/vertaix/Vendi-Score for more information. | |
| ## Metric Description | |
| The Vendi Score (VS) is a metric for evaluating diversity in machine learning. | |
| The input to metric is a collection of samples and a pairwise similarity function, and the output is a number, which can be interpreted as the effective number of unique elements in the sample. | |
| Specifically, given an `n x n` positive semi-definite matrix `K` of similarity scores, the score is defined as: | |
| ``` | |
| VS(K) = exp(tr(K/n @ log(K/n))) = exp(-sum_i lambda_i log lambda_i), | |
| ``` | |
| where `lambda_i` are the eigenvalues of `K/n` and `0 log 0 = 0`. | |
| That is, the Vendi Score is equal to the exponential of the von Neumann entropy of `K/n`, or the Shannon entropy of the eigenvalues, which is also known as the effective rank. | |
| ## How to Use | |
| The Vendi Score is available as a Python package or in HuggingFace `evaluate`. | |
| To use the Python package, see the instructions at https://github.com/vertaix/Vendi-Score. | |
| To use the `evaluate` module, pass a list of samples and a similarity function or a string identifying a predefined class of similarity functions (see below). | |
| ``` | |
| >>> vendiscore = evaluate.load("danf0/vendiscore") | |
| >>> samples = ["Look, Jane.", | |
| "See Spot.", | |
| "See Spot run.", | |
| "Run, Spot, run.", | |
| "Jane sees Spot run."] | |
| >>> results = vendiscore.compute(samples, k="ngram_overlap", ns=[1, 2]) | |
| >>> print(results) | |
| {'VS': 3.90657...} | |
| ``` | |
| ### Inputs | |
| - **samples**: an iterable containing n samples to score; an n x n similarity | |
| matrix K, or an n x d feature matrix X. | |
| - **k**: a pairwise similarity function, or a string identifying a predefined | |
| similarity function. If k is a pairwise similarity function, it should | |
| be symmetric and k(x, x) = 1. | |
| Options: ngram_overlap, text_embeddings, pixels, image_embeddings. | |
| - **score_K**: if true, samples is an n x n similarity matrix K. | |
| - **score_X**: if true, samples is an n x d feature matrix X. | |
| - **score_dual**: if true, samples is an n x d feature matrix X and we will | |
| compute the diversity score using the covariance matrix X @ X.T. | |
| - **normalize**: if true, normalize the similarity scores. | |
| - **model (optional)**: if k is "text_embeddings", a model mapping sentences to | |
| embeddings (output should be an object with an attribute called | |
| `pooler_output` or `last_hidden_state`). If k is "image_embeddings", a | |
| model mapping images to embeddings. | |
| - **tokenizer (optional)**: if k is "text_embeddings" or "ngram_overlap", a | |
| tokenizer mapping strings to lists. | |
| - **transform (optional)**: if k is "image_embeddings", a torchvision transform | |
| to apply to the samples. | |
| - **model_path (optional)**: if k is "text_embeddings", the name of a model on | |
| the HuggingFace hub. | |
| - **ns (optional)**: if k is "ngram_overlap", the values of n to calculate. | |
| - **batch_size (optional)**: batch size to use if k is "text_embedding" or | |
| "image_embedding". | |
| - **device (optional)**: a string (e.g. "cuda", "cpu") or torch.device | |
| identifying the device to use if k is "text_embedding" | |
| or "image_embedding". | |
| ### Output Values | |
| The output is a dictionary with one key, "VS". | |
| Given n samples, the value of the Vendi Score ranges between 1 and n, with higher numbers indicating that the sample is more diverse. | |
| ### Examples | |
| ``` | |
| >>> import numpy as np | |
| >>> vendiscore = evaluate.load("danf0/vendiscore") | |
| >>> samples = [0, 0, 10, 10, 20, 20] | |
| >>> k = lambda a, b: np.exp(-np.abs(a - b)) | |
| >>> vendiscore.compute(samples, k) | |
| 2.9999 | |
| ``` | |
| If you already have precomputed a similarity matrix: | |
| ``` | |
| >>> K = np.array([[1.0, 0.9, 0.0], | |
| [0.9, 1.0, 0.0], | |
| [0.0, 0.0, 1.0]]) | |
| >>> vendiscore.compute(K, score_K=True) | |
| 2.1573 | |
| ``` | |
| If your similarity function is a dot product between `n` normalized | |
| `d`-dimensional embeddings `X`, and `d` < `n`, it is faster | |
| to compute the Vendi Score using the covariance matrix, `X @ X.T`. | |
| (If the rows of `X` are not normalized, set `normalize = True`.) | |
| ``` | |
| >>> X = np.array([[100, 0], [99, 1], [1, 99], [0, 100]) | |
| >>> vendiscore.compute(X, score_dual=True, normalize=True) | |
| 1.9989... | |
| ``` | |
| Image similarity can be calculated using inner products between pixel vectors or between embeddings from a neural network. | |
| The default embeddings are from the pool-2048 layer of the torchvision version of the Inception v3 model; other embedding functions can be passed to the `model` argument. | |
| ``` | |
| >>> from torchvision import datasets | |
| >>> mnist = datasets.MNIST("data/mnist", train=False, download=True) | |
| >>> digits = [[x for x, y in mnist if y == c] for c in range(10)] | |
| >>> pixel_vs = [vendiscore.compute(imgs, k="pixels") for imgs in digits] | |
| >>> inception_vs = [vendiscore.compute(imgs, k="image_embeddings", batch_size=64, device="cuda") for imgs in digits] | |
| >>> for y, (pvs, ivs) in enumerate(zip(pixel_vs, inception_vs)): print(f"{y}\t{pvs:.02f}\t{ivs:02f}") | |
| 0 7.68 3.45 | |
| 1 5.31 3.50 | |
| 2 12.18 3.62 | |
| 3 9.97 2.97 | |
| 4 11.10 3.75 | |
| 5 13.51 3.16 | |
| 6 9.06 3.63 | |
| 7 9.58 4.07 | |
| 8 9.69 3.74 | |
| 9 8.56 3.43 | |
| ``` | |
| Text similarity can be calculated using n-gram overlap or using inner products between embeddings from a neural network. | |
| ``` | |
| >>> sents = ["Look, Jane.", | |
| "See Spot.", | |
| "See Spot run.", | |
| "Run, Spot, run.", | |
| "Jane sees Spot run."] | |
| >>> ngram_vs = vendiscore.compute(sents, k="ngram_overlap", ns=[1, 2]) | |
| >>> bert_vs = vendiscore.compute(sents, k="text_embeddings", model_path="bert-base-uncased") | |
| >>> simcse_vs = vendiscore.compute(sents, k="text_embeddings", model_path="princeton-nlp/unsup-simcse-bert-base-uncased") | |
| >>> print(f"N-grams: {ngram_vs:.02f}, BERT: {bert_vs:.02f}, SimCSE: {simcse_vs:.02f}) | |
| N-grams: 3.91, BERT: 1.21, SimCSE: 2.81 | |
| ``` | |
| ## Limitations and Bias | |
| The Vendi Score depends on the choice of similarity function. Care should be taken to select a similarity function that reflects the features that are relevant for defining diversity in a given application. | |
| ## Citation | |