Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/m3dev/gokart

Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline.
https://github.com/m3dev/gokart

gokart luigi machine-learning mlops pipeline-framework

Last synced: 2 months ago
JSON representation

Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline.

Host: GitHub
URL: https://github.com/m3dev/gokart
Owner: m3dev
License: mit
Created: 2018-12-23T07:40:27.000Z (over 5 years ago)
Default Branch: master
Last Pushed: 2024-04-15T01:53:18.000Z (2 months ago)
Last Synced: 2024-04-16T07:13:20.816Z (2 months ago)
Topics: gokart, luigi, machine-learning, mlops, pipeline-framework
Language: Python
Homepage: https://gokart.readthedocs.io/en/latest/
Size: 1.05 MB
Stars: 300
Watchers: 39
Forks: 56
Open Issues: 21
Metadata Files:
- Readme: README.md
- License: LICENSE

Lists

awesome-stars - m3dev/gokart - Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline. (Python)
awesome-production-machine-learning - Gokart - Wrapper of the data pipeline Luigi. (Data Pipeline)
awesome-production-machine-learning - Gokart - Wrapper of the data pipeline Luigi (Data Pipeline ETL Frameworks)
Awesome-AIML-Data-Ops - Gokart - Wrapper of the data pipeline Luigi (Data Pipeline ETL Frameworks)
awesome-stars - m3dev/gokart - Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline. (Python)
awesome-python-machine-learning-resources - GitHub - 19% open · ⏱️ 02.08.2022): (工作流程和实验跟踪)

README

        # gokart



  



[![Test](https://github.com/m3dev/gokart/workflows/Test/badge.svg)](https://github.com/m3dev/gokart/actions?query=workflow%3ATest)

[![](https://readthedocs.org/projects/gokart/badge/?version=latest)](https://gokart.readthedocs.io/en/latest/)

[![Python Versions](https://img.shields.io/pypi/pyversions/gokart.svg)](https://pypi.org/project/gokart/)

[![](https://img.shields.io/pypi/v/gokart)](https://pypi.org/project/gokart/)

![](https://img.shields.io/pypi/l/gokart)

Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline.

[Documentation](https://gokart.readthedocs.io/en/latest/) for the latest release is hosted on readthedocs.

# About gokart

Here are some good things about gokart.

- The following meta data for each Task is stored separately in a `pkl` file with hash value

    - task output data

    - imported all module versions

    - task processing time

    - random seed in task

    - displayed log

    - all parameters set as class variables in the task

- Automatically rerun the pipeline if parameters of Tasks are changed.

- Support GCS and S3 as a data store for intermediate results of Tasks in the pipeline.

- The above output is exchanged between tasks as an intermediate file, which is memory-friendly

- `pandas.DataFrame` type and column checking during I/O

- Directory structure of saved files is automatically determined from structure of script

- Seeds for numpy and random are automatically fixed

- Can code while adhering to [SOLID](https://en.wikipedia.org/wiki/SOLID) principles as much as possible

- Tasks are locked via redis even if they run in parallel

**All the functions above are created for constructing Machine Learning batches. Provides an excellent environment for reproducibility and team development.**

Here are some non-goal / downside of the gokart.

- Batch execution in parallel is supported, but parallel and concurrent execution of task in memory.

- Gokart is focused on reproducibility. So, I/O and capacity of data storage can become a bottleneck.

- No support for task visualize.

- Gokart is not an experiment management tool. The management of the execution result is cut out as [Thunderbolt](https://github.com/m3dev/thunderbolt).

- Gokart does not recommend writing pipelines in toml, yaml, json, and more. Gokart is preferring to write them in Python.

# Getting Started

Within the activated Python environment, use the following command to install gokart.

```

pip install gokart

```

# Quickstart

A minimal gokart tasks looks something like this:

```python

import gokart

class Example(gokart.TaskOnKart):

    def run(self):

        self.dump('Hello, world!')

task = Example()

output = gokart.build(task)

print(output)

```

`gokart.build` return the result of dump by `gokart.TaskOnKart`. The example will output the following.

```

Hello, world!

```

This is an introduction to some of the gokart.

There are still more useful features.

Please See [Documentation](https://gokart.readthedocs.io/en/latest/) .

Have a good gokart life.

# Achievements

Gokart is a proven product.

- It's actually been used by [m3.inc](https://corporate.m3.com/en) for over 3 years

- Natural Language Processing Competition by [Nishika.inc](https://nishika.com) 2nd prize : [Solution Repository](https://github.com/vaaaaanquish/nishika_akutagawa_2nd_prize)

# Thanks

gokart is a wrapper for luigi. Thanks to luigi and dependent projects!

- [luigi](https://github.com/spotify/luigi)