Libraries
The Datasets Hub has support for several libraries in the Open Source ecosystem. Thanks to the huggingface_hub Python library, it’s easy to enable sharing your datasets on the Hub. We’re happy to welcome to the Hub a set of Open Source libraries that are pushing Machine Learning forward.
The table below summarizes the supported libraries and their level of integration.
| Library | Description | Download from Hub | Push to Hub |
|---|---|---|---|
| Argilla | Collaboration tool for AI engineers and domain experts that value high quality data. | ✅ | ✅ |
| Dask | Parallel and distributed computing library that scales the existing Python and PyData ecosystem. | ✅ | ✅ |
| Datasets | 🤗 Datasets is a library for accessing and sharing datasets for Audio, Computer Vision, and Natural Language Processing (NLP). | ✅ | ✅ |
| Distilabel | The framework for synthetic data generation and AI feedback. | ✅ | ✅ |
| DuckDB | In-process SQL OLAP database management system. | ✅ | ✅ |
| FiftyOne | FiftyOne is a library for curation and visualization of image, video, and 3D data. | ✅ | ✅ |
| Pandas | Python data analysis toolkit. | ✅ | ✅ |
| Polars | A DataFrame library on top of an OLAP query engine. | ✅ | ✅ |
| Spark | Real-time, large-scale data processing tool in a distributed environment. | ✅ | ✅ |
| WebDataset | Library to write I/O pipelines for large datasets. | ✅ | ❌ |