{"id":20477164,"url":"https://github.com/mylamour/autoclf","last_synced_at":"2026-04-29T10:03:24.456Z","repository":{"id":144404132,"uuid":"132134440","full_name":"mylamour/autoclf","owner":"mylamour","description":"batch machine learning ","archived":false,"fork":false,"pushed_at":"2019-03-26T14:43:39.000Z","size":179,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-05T14:48:41.651Z","etag":null,"topics":["batch-processing","clf","machine-learning","pipline","sklearn"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mylamour.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-04T11:58:50.000Z","updated_at":"2019-11-18T14:37:09.000Z","dependencies_parsed_at":null,"dependency_job_id":"a509ef4c-db1a-44d2-830e-d259211a7dae","html_url":"https://github.com/mylamour/autoclf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mylamour/autoclf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mylamour%2Fautoclf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mylamour%2Fautoclf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mylamour%2Fautoclf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mylamour%2Fautoclf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mylamour","download_url":"https://codeload.github.com/mylamour/autoclf/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mylamour%2Fautoclf/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260279134,"owners_count":22985414,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch-processing","clf","machine-learning","pipline","sklearn"],"created_at":"2024-11-15T15:26:05.329Z","updated_at":"2026-04-29T10:03:19.433Z","avatar_url":"https://github.com/mylamour.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Intro\n一个`Mini`型的ml/dl的项目,需要使用者具有一定的编程能力。目录结构为\n\n```\n├── clf\n│   │\n│   ├── nn\n│\n├── data\n│\n├── pipe\n│\n└── saved\n│\n└── models.py\n├── train.py\n└── predict.py\n```\n\n* 一般情况 data 目录下放置数据集\n* `clf` 文件夹下是为了自定义的机器学习算法，例如`GridSearch SVC`等, 而其子文件夹nn用于存放神经网络等深度学习算法\n* `pipe` 文件夹下放置对数据集的预定义处理, 意味着你可以从任何地方加载并处理你的数据, 例如`pipe/iload_aliatec.py`即是对此次ATEC风险支付的数据处理\n* `saved` 为了存放训练好的模型，或者预测后的数据\n\n\n# Useage:\n\n## train.py\n```\n$ python train.py\n\nUsage: train.py [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n  --help  Show this message and exit.\n\nCommands:\n  classification  this for select classification model\n  cluster         this for select cluster model\n```\n\n```\n$ python train.py classification --help\n\nUsage: train.py classification [OPTIONS]\n\n  this for select classification model\n\nOptions:\n  --method TEXT               Your method for training model\n  --pipe TEXT                 Data Pipe Line File\n  --cross-validation INTEGER  Cross Validation\n  --help                      Show this message and exit.\n\n```\n\n```\n$ python train.py classification --pipe pipe/iload_digits.py --method lg --method rbfsvc\n\n[*] Now Training With LogisticRegression And Model Scores 0.9666666666666667\n[*] Now Training With SVC        And Model Scores 0.9805555555555555\n[+] Save it in saved/logisticregression.pkl\n[+] Save it in saved/svc.pkl\n\n```\n\n```\n$ python train.py classification --pipe pipe/iload_digits.py --method lg\n\n[*] Now Training With LogisticRegression And Model Scores 0.9666666666666667\n[!] saved/logisticregression.pkl Existed\n[+] Save it in saved/logisticregression.pkl.second\n\n```\n\n```\n$ python train.py classification --pipe pipe/iload_iris.py  --method lg --loss neg_log_loss\n[*] Now Training With LogisticRegression Loss :  neg_log_loss\n And Model Scores 1.0\n[+] Save it in saved/logisticregression.pkl\n```\n\n```\n$ python train.py classification --pipe pipe/iload_iris.py\n\n[!] Now We Will Use Default All Method\n[*] Now Training With VotingClassifier And Model Scores 1.0\n[*] Now Training With VotingClassifier And Model Scores 1.0\n[*] Now Training With AdaBoostClassifier And Model Scores 0.9333333333333333\n[*] Now Training With GaussianNB And Model Scores 1.0\n[*] Now Training With XGBClassifier And Model Scores 1.0\n[*] Now Training With LogisticRegression And Model Scores 1.0\n[*] Now Training With SVC        And Model Scores 1.0\n[*] Now Training With KNeighborsClassifier And Model Scores 0.9666666666666667\n[*] Now Training With RandomForestClassifier And Model Scores 1.0\n[*] Now Training With DecisionTreeClassifier And Model Scores 1.0\n[*] Now Training With IGridSVC   And Model Scores 1.0\n[!] saved/votingclassifier.pkl Existed\n[+] Save it in saved/votingclassifier.pkl.second\n[!] saved/votingclassifier.pkl Existed\n[+] Save it in saved/votingclassifier.pkl.second\n[!] saved/adaboostclassifier.pkl Existed\n[+] Save it in saved/adaboostclassifier.pkl.second\n[!] saved/gaussiannb.pkl Existed\n[+] Save it in saved/gaussiannb.pkl.second\n[!] saved/xgbclassifier.pkl Existed\n[+] Save it in saved/xgbclassifier.pkl.second\n[!] saved/logisticregression.pkl Existed\n[+] Save it in saved/logisticregression.pkl.second\n[!] saved/svc.pkl Existed\n[+] Save it in saved/svc.pkl.second\n[!] saved/kneighborsclassifier.pkl Existed\n[+] Save it in saved/kneighborsclassifier.pkl.second\n[!] saved/randomforestclassifier.pkl Existed\n[+] Save it in saved/randomforestclassifier.pkl.second\n[!] saved/decisiontreeclassifier.pkl Existed\n[+] Save it in saved/decisiontreeclassifier.pkl.second\n[!] saved/igridsvc.pkl Existed\n[+] Save it in saved/igridsvc.pkl.second\n```\n\n## predict.py\n载入saved文件夹下的已经保存好的模型进行预测，默认预测结果输出到`saved`文件夹，分别以`predict`和`proba`的后缀结尾，还可以通过，自定义输出路径，指定预测结果的输出，例如`--out woqu`\n\n```\n$ python predict.py predict  --help\nUsage: predict.py predict [OPTIONS]\n\nOptions:\n  --method TEXT  Your method for training model\n  --pipe TEXT    Data Pipe Line File\n  --out TEXT     Directory for save predict\n  --help         Show this message and exit.\n\n```\n\n```\n$ python predict.py predict  --pipe pipe/iload_iris.py --method saved/adaboostclassifier.pkl\n\n [####################################]  100% predict use model: AdaBoostClassifier\n\n```\n\n```\n$python predict.py predict  --pipe pipe/iload_iris.py --method saved/adaboostclassifier.pkl --method saved/decisiontreeclassifier.pkl\n  \n  [##################------------------]   50% predict use model: AdaBoostClassifier\n  [####################################]  100% predict use model: DecisionTreeClassifier\n\n```\n\n```\n$ python predict.py predict  --pipe pipe/iload_iris.py --method saved\nUse Batch Models From /home/mour/MlDl/autoclf/saved\n  [###---------------------------------]   10% predict use model: LogisticRegression\n  [#######-----------------------------]   20% predict use model: AdaBoostClassifier\n  [##########--------------------------]   30% predict use model: XGBClassifier\n  [##############----------------------]   40% predict use model: SVC\n  [##################------------------]   50% predict use model: GaussianNB\n  [#####################---------------]   60% predict use model: KNeighborsClassifier\n  [#########################-----------]   70% predict use model: VotingClassifier\n  [############################--------]   80% predict use model: DecisionTreeClassifier\n  [################################----]   90% predict use model: RandomForestClassifier\n  [####################################]  100% predict use model: IGridSVC\n```\n\n```\n$ python predict.py predict  --pipe pipe/iload_iris.py --method saved --out woqu\nUse Batch Models From /home/mour/MlDl/autoclf/saved\n  [###---------------------------------]   10% predict use model: LogisticRegression\n  [#######-----------------------------]   20% predict use model: AdaBoostClassifier\n  [##########--------------------------]   30% predict use model: XGBClassifier\n  [##############----------------------]   40% predict use model: SVC\n  [##################------------------]   50% predict use model: GaussianNB\n  [#####################---------------]   60% predict use model: KNeighborsClassifier\n  [#########################-----------]   70% predict use model: VotingClassifier\n  [############################--------]   80% predict use model: DecisionTreeClassifier\n  [################################----]   90% predict use model: RandomForestClassifier\n  [####################################]  100% predict use model: IGridSVC\n\n```\n\n# Note\n\n* 在load数据进行Pipline处理后，再交由自定义算法Pipline处理时可能会有意想不到的错误。(Sklearn本身的问题)，可以只在其中一处做Pipline,即只在pipe文件夹下load数据时自定义，也可以只在自定义算法时进行pipline\n\n* 数据预处理文件的定义需要遵循格式，即要处理内容定义在`iload_pipe`函数中,预测函数定义在`ipredict_pipe`中\n\n# Todo\n\n- [x] 增加requerments.txt 文件\n- [ ] HypeOPT 自动search参数\n- [ ] Dask分布式计算\n- [ ] 单元测试\n- [ ] 增加`cluster`算法相关\n- [x] 重构`predict`文件\n- [x] 伪ETL工程目录\n- [x] 性能评价模块\n- [x] 动态创建类的函数\n- [x] 自定义 nn 函数\n- [x] 自定义 clf 函数\n- [x] 支持自定义函数的`cross_validation`\n- [x] 捕获ctrl+c，中断当前训练器\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmylamour%2Fautoclf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmylamour%2Fautoclf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmylamour%2Fautoclf/lists"}