Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/merantix-momentum/squirrel-datasets-core
Squirrel dataset hub
https://github.com/merantix-momentum/squirrel-datasets-core
ai cloud-computing collaboration computer-vision cv data-ingestion data-mesh data-science dataops datasets deep-learning distributed machine-learning ml natural-language-processing npl python pytorch tensorflow
Last synced: about 2 months ago
JSON representation
Squirrel dataset hub
- Host: GitHub
- URL: https://github.com/merantix-momentum/squirrel-datasets-core
- Owner: merantix-momentum
- License: apache-2.0
- Created: 2022-02-01T17:01:42.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2023-09-07T05:08:40.000Z (over 1 year ago)
- Last Synced: 2024-09-22T19:21:53.465Z (3 months ago)
- Topics: ai, cloud-computing, collaboration, computer-vision, cv, data-ingestion, data-mesh, data-science, dataops, datasets, deep-learning, distributed, machine-learning, ml, natural-language-processing, npl, python, pytorch, tensorflow
- Language: Python
- Homepage: https://squirrel-datasets-core.readthedocs.io/
- Size: 761 KB
- Stars: 42
- Watchers: 13
- Forks: 6
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: docs/code_of_conduct.rst
Awesome Lists containing this project
README
# 
Squirrel Datasets Core
[](https://badge.fury.io/py/squirrel-datasets-core)
[](https://badge.fury.io/py/squirrel-datasets-core)
[](https://anaconda.org/conda-forge/squirrel-datasets-core)
[](https://squirrel-datasets-core.readthedocs.io)
[](https://pepy.tech/project/squirrel-datasets-core)
[](https://raw.githubusercontent.com/merantix-momentum/squirrel-datasets-core/main/LICENSE)
[](https://doi.org/10.5281/zenodo.6420214)
[](https://merantix-momentum.com)
[](https://join.slack.com/t/squirrel-core/shared_invite/zt-14k6sk6sw-zQPHfqAI8Xq5WYd~UqgNFw)
---
## What is Squirrel Datasets Core?
`squirrel-datasets-core` is an extension of the [Squirrel](https://github.com/merantix-momentum/squirrel-core) library. `squirrel-datasets-core` is a hub where the user can
1) explore existing public datasets registered in the data mesh and load them with the ease and speed of `squirrel`
2) preprocess their datasets and share them with other users.
For preprocessing, we currently support Spark as the main tool to carry out the task.
If you have any questions or would like to contribute, join our [Slack community](https://join.slack.com/t/squirrel-core/shared_invite/zt-14k6sk6sw-zQPHfqAI8Xq5WYd~UqgNFw)!
## Installation
Install `squirrel-core` and `squirrel-datasets-core` with pip. Note that you can install with different dependencies based on your requirements for squirrel drivers.
For using the Torchvision driver call:
```shell
pip install "squirrel-core[torch]"
pip install "squirrel-datasets-core[torchvision]"
```
For using the Huggingface or Deeplake driver call:
```shell
pip install "squirrel-datasets-core[huggingface]"
pip install "squirrel-datasets-core[deeplake]"
```
For using the Spark preprocessing pipelines call:
```shell
pip install "squirrel-datasets-core[preprocessing]"
```
If you would like to get Squirrel's full functionality, install squirrel-core and squirrel-datasets-core with all their dependencies.
```shell
pip install "squirrel-core[all]"
pip install "squirrel-datasets-core[all]"
```
## Huggingface, Deeplake, Hub and Torchvision Integration
A great feature of squirrel-datasets-core is that you can easily load data from common databases such as Huggingface, Activeloop Deeplake, Hub and Torchvision with one line of code. And you get to enjoy all of Squirrel’s benefits for free! Check out the [documentation](https://squirrel-datasets-core.readthedocs.io/en/latest/driver_integration.html) on how to interface with these libraries.
```python
from squirrel_datasets_core.driver.huggingface import HuggingfaceDriver
it = HuggingfaceDriver("cifar100").get_iter("train").filter(custom_filter).map(custom_augmentation)
# your train loop
for item in it:
out = model(item)
# ...
```
## Documentation
Visit our documentation on [Readthedocs](https://squirrel-datasets-core.readthedocs.io).
## Contributing
`squirrel-datasets-core` is open source and community contributions are welcome!
Check out the [contribution guide](https://squirrel-datasets-core.readthedocs.io/en/latest/contribute.html) to learn how to get involved.
Please follow our recommendations for best practices and code style.
## The Humans behind Squirrel
We are [Merantix Momentum](https://merantix-momentum.com/), a team of ~30 machine learning engineers, developing machine learning solutions for industry and research. Each project comes with its own challenges, data types and learnings, but one issue we always faced was scalable data loading, transforming and sharing. We were looking for a solution that would allow us to load the data in a fast and cost-efficient way, while keeping the flexibility to work with any possible dataset and integrate with any API. That's why we build Squirrel – and we hope you'll find it as useful as we do! By the way, [we are hiring](https://merantix-momentum.com/about#jobs)!
## Citation
If you use Squirrel Datasets in your research, please cite Squirrel using:
```bibtex
@article{2022squirrelcore,
title={Squirrel: A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way.},
author={Squirrel Developer Team},
journal={GitHub. Note: https://github.com/merantix-momentum/squirrel-core},
year={2022}
}
```