{"id":19544350,"url":"https://github.com/chakki-works/chariot","last_synced_at":"2026-02-16T01:13:12.100Z","repository":{"id":55115239,"uuid":"136851054","full_name":"chakki-works/chariot","owner":"chakki-works","description":"Deliver the ready-to-train data to your NLP model.","archived":false,"fork":false,"pushed_at":"2022-07-15T18:40:51.000Z","size":5883,"stargazers_count":121,"open_issues_count":3,"forks_count":9,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-02-15T04:11:21.307Z","etag":null,"topics":["keras","natural-language-processing","preprocessing","python","tensorflow"],"latest_commit_sha":null,"homepage":"https://chakki-works.github.io/chariot/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chakki-works.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-10T22:57:09.000Z","updated_at":"2025-01-20T10:45:10.000Z","dependencies_parsed_at":"2022-08-14T12:30:35.869Z","dependency_job_id":null,"html_url":"https://github.com/chakki-works/chariot","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chakki-works%2Fchariot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chakki-works%2Fchariot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chakki-works%2Fchariot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chakki-works%2Fchariot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chakki-works","download_url":"https://codeload.github.com/chakki-works/chariot/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244066179,"owners_count":20392406,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["keras","natural-language-processing","preprocessing","python","tensorflow"],"created_at":"2024-11-11T03:27:51.870Z","updated_at":"2026-02-16T01:13:07.061Z","avatar_url":"https://github.com/chakki-works.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# chariot\n\n[![PyPI version](https://badge.fury.io/py/chariot.svg)](https://badge.fury.io/py/chariot)\n[![Build Status](https://travis-ci.org/chakki-works/chariot.svg?branch=master)](https://travis-ci.org/chakki-works/chariot)\n[![codecov](https://codecov.io/gh/chakki-works/chariot/branch/master/graph/badge.svg)](https://codecov.io/gh/chakki-works/chariot)\n\n**Deliver the ready-to-train data to your NLP model.**\n\n\n* Prepare Dataset\n  * You can prepare typical NLP datasets through the [chazutsu](https://github.com/chakki-works/chazutsu).\n* Build \u0026 Run Preprocess\n  * You can build the preprocess pipeline like [scikit-learn Pipeline](http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html).\n  * Preprocesses for each dataset column are executed in parallel by [Joblib](https://pythonhosted.org/joblib/index.html).\n  * Multi-language text tokenization is supported by [spaCy](https://spacy.io/).\n* Format Batch\n  * Sampling a batch from preprocessed dataset and format it to train the model (padding etc).\n  * You can use pre-trained word vectors through the [chakin](https://github.com/chakki-works/chakin).\n\n**chariot** enables you to concentrate on training your model!\n\n![chariot flow](./docs/images/chariot_feature.gif)\n\n## Install\n\n```\npip install chariot\n```\n\n## Prepare dataset\n\nYou can download various dataset by using [chazutsu](https://github.com/chakki-works/chazutsu).  \n\n```py\nimport chazutsu\nfrom chariot.storage import Storage\n\n\nstorage = Storage(\"your/data/root\")\nr = chazutsu.datasets.MovieReview.polarity().download(storage.path(\"raw\"))\n\ndf = storage.chazutsu(r.root).data()\ndf.head(5)\n```\n\nThen\n\n```\n\tpolarity\treview\n0\t0\tsynopsis : an aging master art thief , his sup...\n1\t0\tplot : a separated , glamorous , hollywood cou...\n2\t0\ta friend invites you to a movie . this film wo...\n```\n\n`Storage` class manage the directory structure that follows [cookie-cutter datascience](https://drivendata.github.io/cookiecutter-data-science/).\n\n```\nProject root\n  └── data\n       ├── external     \u003c- Data from third party sources (ex. word vectors).\n       ├── interim      \u003c- Intermediate data that has been transformed.\n       ├── processed    \u003c- The final, canonical datasets for modeling.\n       └── raw          \u003c- The original, immutable data dump.\n```\n\n## Build \u0026 Run Preprocess\n\n### Build a preprocess pipeline\n\nAll preprocessors are defined at `chariot.transformer`.  \nTransformers are implemented by extending [scikit-learn `Transformer`](https://scikit-learn.org/stable/modules/generated/sklearn.base.TransformerMixin.html).  \nBecause of this, the API of Transformer is familiar to you. And you can mix [scikit-learn's preprocessors](https://scikit-learn.org/stable/modules/preprocessing.html).\n\n```py\nimport chariot.transformer as ct\nfrom chariot.preprocessor import Preprocessor\n\n\npreprocessor = Preprocessor()\npreprocessor\\\n    .stack(ct.text.UnicodeNormalizer())\\\n    .stack(ct.Tokenizer(\"en\"))\\\n    .stack(ct.token.StopwordFilter(\"en\"))\\\n    .stack(ct.Vocabulary(min_df=5, max_df=0.5))\\\n    .fit(train_data)\n\npreprocessor.save(\"my_preprocessor.pkl\")\n\nloaded = Preprocessor.load(\"my_preprocessor.pkl\")\n```\n\nThere is 6 type of transformers are prepared in chariot.\n\n* TextPreprocessor\n  * Preprocess the text before tokenization.\n  * `TextNormalizer`: Normalize text (replace some character etc).\n  * `TextFilter`: Filter the text (delete some span in text stc).\n* Tokenizer\n  * Tokenize the texts.\n  * It powered by [spaCy](https://spacy.io/) and you can choose [MeCab](https://github.com/taku910/mecab) or [Janome](https://github.com/mocobeta/janome) for Japanese.\n* TokenPreprocessor\n  * Normalize/Filter the tokens after tokenization.\n  * `TokenNormalizer`: Normalize tokens (to lower, to original form etc).\n  * `TokenFilter`: Filter tokens (extract only noun etc).\n* Vocabulary\n  * Make vocabulary and convert tokens to indices.\n* Formatter\n  * Format (preprocessed) data for training your model.\n* Generator\n  * Genrate target data to train your (language) model.\n\n### Build a preprocess for dataset\n\nWhen you want to make preprocess to each of your dataset column, you can use `DatasetPreprocessor`.\n\n```py\nfrom chariot.dataset_preprocessor import DatasetPreprocessor\nfrom chariot.transformer.formatter import Padding\n\n\ndp = DatasetPreprocessor()\ndp.process(\"review\")\\\n    .by(ct.text.UnicodeNormalizer())\\\n    .by(ct.Tokenizer(\"en\"))\\\n    .by(ct.token.StopwordFilter(\"en\"))\\\n    .by(ct.Vocabulary(min_df=5, max_df=0.5))\\\n    .by(Padding(length=pad_length))\\\n    .fit(train_data[\"review\"])\ndp.process(\"polarity\")\\\n    .by(ct.formatter.CategoricalLabel(num_class=3))\n\n\npreprocessed = dp.preprocess(data)\n\n# DatasetPreprocessor has multiple preprocessor.\n# Because of this, save file format is `tar.gz`.\ndp.save(\"my_dataset_preprocessor.tar.gz\")\n\nloaded = DatasetPreprocessor.load(\"my_dataset_preprocessor.tar.gz\")\n```\n\n## Train your model with chariot\n\n`chariot` has feature to traing your model.\n\n```py\nformatted = dp(train_data).preprocess().format().processed\n\nmodel.fit(formatted[\"review\"], formatted[\"polarity\"], batch_size=32,\n          validation_split=0.2, epochs=15, verbose=2)\n\n```\n\n```py\nfor batch in dp(train_data.preprocess().iterate(batch_size=32, epoch=10):\n    model.train_on_batch(batch[\"review\"], batch[\"polarity\"])\n\n```\n\nYou can use pre-trained word vectors by [chakin](https://github.com/chakki-works/chakin).  \n\n\n```py\nfrom chariot.storage import Storage\nfrom chariot.transformer.vocabulary import Vocabulary\n\n# Download word vector\nstorage = Storage(\"your/data/root\")\nstorage.chakin(name=\"GloVe.6B.50d\")\n\n# Make embedding matrix\nvocab = Vocabulary()\nvocab.set([\"you\", \"loaded\", \"word\", \"vector\", \"now\"])\nembed = vocab.make_embedding(storage.path(\"external/glove.6B.50d.txt\"))\nprint(embed.shape)  # (len(vocab.count), 50)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchakki-works%2Fchariot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchakki-works%2Fchariot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchakki-works%2Fchariot/lists"}