{"id":21102552,"url":"https://github.com/shifuml/shifu","last_synced_at":"2025-04-05T03:12:33.912Z","repository":{"id":16260803,"uuid":"19008910","full_name":"ShifuML/shifu","owner":"ShifuML","description":"An end-to-end machine learning and data mining framework on Hadoop","archived":false,"fork":false,"pushed_at":"2024-05-13T05:32:43.000Z","size":16930,"stargazers_count":252,"open_issues_count":238,"forks_count":108,"subscribers_count":40,"default_branch":"master","last_synced_at":"2025-03-29T02:08:22.505Z","etag":null,"topics":["bigdata","end-to-end-machine-learning","gbdt","hadoop","machine-learning","neural-network","pipeline","random-forest","shifu"],"latest_commit_sha":null,"homepage":"https://github.com/ShifuML/shifu/wiki","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ShifuML.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.txt","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-04-21T22:21:09.000Z","updated_at":"2025-02-12T10:01:31.000Z","dependencies_parsed_at":"2024-12-15T18:11:25.626Z","dependency_job_id":"94906c48-851e-45a6-a790-5efa8ac664cf","html_url":"https://github.com/ShifuML/shifu","commit_stats":null,"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShifuML%2Fshifu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShifuML%2Fshifu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShifuML%2Fshifu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShifuML%2Fshifu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ShifuML","download_url":"https://codeload.github.com/ShifuML/shifu/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247280272,"owners_count":20912967,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigdata","end-to-end-machine-learning","gbdt","hadoop","machine-learning","neural-network","pipeline","random-forest","shifu"],"created_at":"2024-11-19T23:55:48.393Z","updated_at":"2025-04-05T03:12:33.874Z","avatar_url":"https://github.com/ShifuML.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[\u003cimg src=\"images/logo/shifu.png\" alt=\"Shifu\" align=\"left\"\u003e](http://shifu.ml)\u003cdiv align=\"right\"\u003e\u003cdiv\u003e[![Build Status](https://travis-ci.org/ShifuML/shifu.svg)](https://travis-ci.org/ShifuML/shifu?branch=develop)\u003c/div\u003e\u003cdiv\u003e[![Maven Central](https://maven-badges.herokuapp.com/maven-central/ml.shifu/shifu/badge.svg)](https://maven-badges.herokuapp.com/maven-central/ml.shifu/shifu)\u003c/div\u003e\u003c/div\u003e\n\n#\n\n## Download\n\nPlease [download](https://github.com/ShifuML/shifu/wiki/shifu-0.12.0-hdp-yarn.tar.gz) latest shifu [here](https://github.com/ShifuML/shifu/wiki/shifu-0.12.0-hdp-yarn.tar.gz).\n\n## Getting Started\nAfter shifu downloading, build your first model with Shifu [tutorial](https://github.com/ShifuML/shifu/wiki/Tutorial---Build-Your-First-ML-Model). More details about shifu can be found in our [wiki pages](https://github.com/ShifuML/shifu/wiki).\n\n## What is Shifu?\nShifu is an open-source, end-to-end machine learning and data mining framework built on top of Hadoop. Shifu is designed for data scientists, simplifying the life-cycle of building machine learning models. While originally built for fraud modeling, Shifu is generalized for many other modeling domains.\n\nOne of Shifu's pros is an end-to-end modeling pipeline in machine learning. With only configurations settings, a whole machine pipeline can be built and model can be much more easy to develop and push to production. The pipeline defined in Shifu is in below:\n\n![Shifu Pipeline](https://raw.githubusercontent.com/wiki/ShifuML/shifu/images/new-shifu-pipeline.png)\n\nShifu provides a simple command-line interface for each step of the model building process, including\n\n* Statistic calculation \u0026 variable selection to determine the most predictive variables in your data\n* [Variable normalization](https://github.com/ShifuML/shifu/wiki/Variable%20Transform%20in%20Shifu)\n* [Distributed variable selection based on sensitivity analysis](https://github.com/ShifuML/shifu/wiki/Variable%20Selection%20in%20Shifu)\n* [Distributed neural network model training](https://github.com/ShifuML/shifu/wiki/Distributed%20Neural%20Network%20Training%20in%20Shifu)\n* [Distributed tree ensemble model training](https://github.com/ShifuML/shifu/wiki/Distributed%20Tree%20Ensemble%20Model%20Training%20in%20Shifu)\n* Post training analysis \u0026 model evaluation\n* [Distributed Tensorflow on Shifu](https://github.com/ShifuML/shifu/wiki/Distributed-Tensorflow-Support-On-Shifu)\n\nShifu’s fast Hadoop-based, distributed neural network / logistic regression / gradient boosted trees training can reduce model training time from days to hours on TB data sets. Shifu integrates with Pig workflows on Hadoop, and Shifu-trained models can be integrated into production code with a simple Java API. Shifu leverages Pig, Akka, Encog and other open source projects.\n\n[Guagua](https://github.com/ShifuML/guagua), an in-memory iterative computing framework on Hadoop YARN is developed as sub-project of Shifu to accelerate training progress.\n\nMore details about shifu can be found in our [wiki pages](https://github.com/ShifuML/shifu/wiki)\n\n## Conference\n\n* [QCON Shanghai 2015](http://2015.qconshanghai.com/presentation/2827) [Slides](http://www.slideshare.net/pengshanzhang/large-scale-machine-learning-at-pay-pal-risk)\n\n* [BDTC Beijing 2016](http://bdtc2016.hadooper.cn/dct/page/70107)\n\n* [Strata Beijing 2017](https://strata.oreilly.com.cn/strata-cn/public/schedule/detail/59593?locale=en)\n\n## Contributors\n\n - Zhanghao Hu (zhanhu@paypal.com)\n - Grahame Jastrebski (gjastrebski@paypal.com)\n - Lavar Li (lulli@paypal.com)\n - Mark Liu (yliu15@paypal.com)\n - David Zhang (pengzhang@paypal.com)\n - Xin Zhong (xinzhong@paypal.com)\n - Simon Zhang (jzhang13@paypal.com)\n - Sharma Nitin (nsharma1@paypal.com)\n - Wayne Zhu (wzhu1@paypal.com)\n - Devin Wu (haifwu@paypal.com)\n - Fred Bai (webai@paypal.com)\n\n## Google Group\n\nPlease join [Shifu group](https://groups.google.com/forum/#!forum/shifuml) if questions, bugs or anything else.\n\n## Copyright and License\n\nCopyright 2012-2019, PayPal Software Foundation under the [Apache License](LICENSE.txt).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshifuml%2Fshifu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshifuml%2Fshifu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshifuml%2Fshifu/lists"}