https://github.com/neuro-ml/connectome

A library for datasets containing heterogeneous data
https://github.com/neuro-ml/connectome

data-processing pipelines python

Last synced: about 1 year ago
JSON representation

A library for datasets containing heterogeneous data

• Host: GitHub
• URL: https://github.com/neuro-ml/connectome
• Owner: neuro-ml
• License: apache-2.0
• Created: 2020-06-29T08:57:53.000Z (almost 6 years ago)
• Default Branch: master
• Last Pushed: 2024-10-03T20:35:08.000Z (over 1 year ago)
• Last Synced: 2025-04-02T20:11:15.167Z (about 1 year ago)
• Topics: data-processing, pipelines, python
• Language: Python
• Homepage: https://neuro-ml.github.io/connectome/
• Size: 2.75 MB
• Stars: 12
• Watchers: 6
• Forks: 1
• Open Issues: 15
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
[![docs](https://img.shields.io/badge/-docs-success)](https://neuro-ml.github.io/connectome/)

[![codecov](https://codecov.io/gh/neuro-ml/connectome/branch/master/graph/badge.svg)](https://codecov.io/gh/neuro-ml/connectome)

[![pypi](https://img.shields.io/pypi/v/connectome?logo=pypi&label=PyPi)](https://pypi.org/project/connectome/)

![License](https://img.shields.io/github/license/neuro-ml/connectome)

[![PyPI - Downloads](https://img.shields.io/pypi/dm/connectome)](https://pypi.org/project/connectome/)

Connectome is a framework for datasets management with strong emphasis on simplicity, composability and reusability.

# Features

* Self-consistency: connectome encourages data transformations that keep entries' fields consistent

* Caching: transformations' caching works out of the box and supports both caching to RAM and to Disk

* Automatic cache invalidation: connectome tracks all the changes made to a dataset and automatically invalidates the

  cache when something changes, making sure that your cache is always consistent with the data

* Invertible transformations: write consistent pre- and post- processing to build production-ready pipelines

# Install

The simplest way is to get it from PyPi:

```shell

pip install connectome

```

Or if you want to try the latest version from GitHub:

```shell

git clone https://github.com/neuro-ml/connectome.git

cd connectome

pip install -e .

# or let pip handle the cloning:

pip install git+https://github.com/neuro-ml/connectome.git

```

# Getting started

The docs are located [here](https://neuro-ml.github.io/connectome)

Also, you can check out our `Intro to connectome` series of

tutorials [here](https://neuro-ml.github.io/connectome/tutorials/00%20-%20Intro/)

# Acknowledgements

Some parts of our automatic cache invalidation machinery vere heavily inspired by

the [cloudpickle](https://github.com/cloudpipe/cloudpickle) project.