mlcroissant

mlcroissant is a library to load datasets from Croissant metadata.

💡 Learn more about how to get the metadata from the dataset viewer API in the Get Croissant metadata guide.

Let’s start by parsing the Croissant metadata for the barilan/blog_authorship_corpus dataset. Be sure to first install mlcroissant[parquet] and GitPython to be able to load Parquet files over the git+https protocol.

from mlcroissant import Dataset
ds = Dataset(jsonld="https://huggingface.co./api/datasets/barilan/blog_authorship_corpus/croissant")

To read from the first subset (called RecordSet in Croissant’s vocabulary), use the records function, which returns an iterator of dicts.

records = ds.records(ds.metadata.record_sets[0].uid)

Finally use Pandas to compute your query on the first 1,000 rows:

import itertools

import pandas as pd

df = (
    pd.DataFrame(list(itertools.islice(records, 1000)))
    .groupby("horoscope")["text"]
    .apply(lambda x: x.str.len().mean())
    .sort_values(ascending=False)
    .head(5)
)
print(df)
horoscope
b'Sagittarius'    1216.000000
b'Libra'           862.615581
b'Capricorn'       381.269231
b'Cancer'          272.776471
Name: text, dtype: float64

< > Update on GitHub

Dataset viewer

mlcroissant