{"id":24716813,"url":"https://github.com/dovolopor-research/data-science-project-template","last_synced_at":"2026-04-09T12:03:17.035Z","repository":{"id":45391511,"uuid":"512230904","full_name":"dovolopor-research/data-science-project-template","owner":"dovolopor-research","description":"🧠 开箱即用的「数据科学项目模板」","archived":false,"fork":false,"pushed_at":"2022-07-13T16:29:20.000Z","size":11,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-27T09:14:21.577Z","etag":null,"topics":["data-science","data-science-projects","deep-learning","machine-learning","project-template"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dovolopor-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-09T16:17:16.000Z","updated_at":"2024-02-23T14:57:09.000Z","dependencies_parsed_at":"2022-09-19T10:10:40.948Z","dependency_job_id":null,"html_url":"https://github.com/dovolopor-research/data-science-project-template","commit_stats":null,"previous_names":[],"tags_count":3,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dovolopor-research%2Fdata-science-project-template","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dovolopor-research%2Fdata-science-project-template/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dovolopor-research%2Fdata-science-project-template/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dovolopor-research%2Fdata-science-project-template/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dovolopor-research","download_url":"https://codeload.github.com/dovolopor-research/data-science-project-template/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244932958,"owners_count":20534266,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","data-science-projects","deep-learning","machine-learning","project-template"],"created_at":"2025-01-27T09:14:23.936Z","updated_at":"2026-04-09T12:03:16.936Z","avatar_url":"https://github.com/dovolopor-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Science Project Template\n\n🧠 **开箱即用**的「数据科学项目模板」\n\n\u003e 知名炼丹师有云：**合理的「项目结构」是成功的起点！**\n\n## 1 目录简介\n\n```\n.\n├── README.md          # 简介\n├── app                # 应用\n│   ├── test.py\n│   └── train.py\n├── notebook           # 交互（可选）\n├── model              # 模型（可选）\n├── data               # 数据\n├── util               # 辅助\n│   ├── conf.py\n│   ├── log.py\n│   ├── data_model.py\n│   ├── db.py\n│   └── data.py\n├── save               # 保存\n├── conf               # 配置\n│   ├── default.prod.yml\n│   ├── default.test.yml\n│   └── default.yml\n├── script             # 脚本\n│   └── init_env.sh    # 创建 Conda 环境\n├── server             # 部署（可选）\n│   └── app.py\n├── test               # 测试（可选）\n│   ├── function_test.py\n│   └── api_test.py\n├── doc                # 文档（可选）\n├── log                # 日志\n├── Dockerfile         # Docker 打包配置（可选）\n├── .dockerignore      # Docker 忽略配置（可选）\n├── .gitignore         # Git 忽略配置\n├── requirements.txt   # 依赖（可选）\n└── LICENSE            # 许可\n```\n\n### 1.1 README.md（文档）\n\n**一个详尽的文档比什么都重要！** 任何人都可以通过文档快速上手。也许你会说，我的代码就我自己看，不写文档也知道。可是你能保证三个月以后，你还记得你当初写的什么吗？！\n\n一个好的深度学习项目文档应该是怎样的？\n\n1. 环境依赖说明\n2. 快速运行脚本\n3. 类似项目比较\n4. 性能测试结果\n5. 细致版本记录\n6. 相关参考资料\n\n参考：\n\n1. [如何写好Github中的readme？](https://www.zhihu.com/question/29100816)\n2. [如何为你的开源项目编写实用的文档](https://zhuanlan.zhihu.com/p/120399648)\n\n### 1.2 app（应用入口）\n\n这里是核心应用的入口，比如训练文件 `train.py` 和测试文件 `test.py`，当然也可以把一些中间过程放在此处，比如提取特征 `extract_feature.py` 等。\n\n推荐使用 `python -m app.train` 这样的方式运行，以避免出现 package 的引用问题。\n\n### 1.3 notebook（交互编程｜可选）\n\n`jupyter notebook` 是一个集成了 `ipython` 的可视化交互代码工具，可以方便的查看中间变量和记录想法，特别适合做数据处理和数据可视化。\n\n\u003e 知名炼丹师有云：炼丹师的工作划分——「80% 清理数据，20% 调节参数」\n\n数据科学两兄弟——`numpy` 和 `pandas` 是不可或缺的！在 notebook 中会大量使用到～\n\n对于大规模数据，建议先读取部分数据，在 notebook 上编写数据处理管道，最后抽象成函数放到 app 目录下运行。\n\n### 1.4 model（算法模型｜可选）\n\n这里一般放「神经网络」或者是机器学习的算法结构。\n\n比如包含 `torch.nn.Module` 的文件。\n\n### 1.5 data（数据）\n\n所有的数据的堆放处，最常见的就是「数据集」。\n\n如果数据集比较大，建议在专门的磁盘中进行存储管理，通过软链接的方式映射到 ./data 目录下面。\n\n```shell\nln -s /path/to/dataset ./data/dataset\n```\n\n\u003e ⚠️ 注意：\n\u003e 删除软链接是 `rm -rf ./data/dataset`，\n\u003e 而不是 `rm -rf ./data/dataset/`，\n\u003e 第二命令做法会把源数据删掉！切记！切记！\n\n### 1.6 util（辅助工具）\n\n一些常见的辅助函数，比如：\n\n1. conf 配置\n2. data 数据\n3. data_model 数据模型\n4. log 日志\n5. db 数据库\n\n### 1.7 save（结果保存）\n\n模型参数、性能测试结果等...\n\n### 1.8 conf（配置中心）\n\n所有的配置都是用 `YAML`，它比 `json` 更好用，可以在配置中添加注释，并且呈现方式也更为直观！\n\n配置文件的中间名称是「环境变量」，比如 ENV 为 `test` 时，就会读取 `default.test.yml` 文件（ENV 默认为 `dev`，会读取 `default.yml`）。\n\n如何使用环境变量 ENV 呢？只需要在执行时，把 `ENV=test` 添加到命令的最前面。\n\n```\nENV=test python -m app.train\n```\n\n### 1.9 script（实用脚本）\n\n对于一些常见的连续命令，我们可以把它整理出来，写成一个脚本，以便快速执行！\n\n比如 `init_env.sh` 可以让你快速创建 conda 虚拟环境！\n\n```shell\nbash scripts/init_env.sh\n```\n\n### 1.10 server（部署｜可选）\n\n使用 Web 服务部署到线上环境，推荐使用 [Sanic](https://sanic.dev/zh/)。\n\n```\nENV=prod python -m server.main\n```\n\n### 1.11 test（测试｜可选）\n\n与模型的测试不同，这里主要为代码的单元测试 `UnitTest`。\n\n`function_test.py` 里面演示了如何做一个最简单的单元测试。\n\n```shell\n# 方法1：使用 python 执行\npython -m test.function_test\n# 方法：使用 pytest 执行，pytest 能够自动找到当前文件夹的所有测试文件\npytest\n```\n\n`api_test.py` 是对 API 接口进行压力测试，得到 QPS（每秒请求数）。\n\n`Locust` 是一个非常好用的测试工具，它附带一个 WEB 界面，非常方便地在浏览器中进行压测。\n\n```shell\nlocust -f test/api_test.py -u 10 -r 1\n```\n\n在浏览器中打开 `http://127.0.0.1:8089`，点击 `Start swarming` 按钮，就可以开始压测了！\n\n参考：\n1. [Python unittest: 单元测试框架](https://docs.python.org/zh-cn/3/library/unittest.html)\n2. [Locust: An open source load testing tool.](https://locust.io)\n\n### 1.12 doc（文档｜可选）\n\n如果项目比较复杂，可以将文档整理归纳到这里。\n\n### 1.13 log（日志）\n\n`util/log.py` 会将日志按天记录到这里。\n\n### 1.14 requirements.txt （依赖）\n\n项目所需要的依赖，方便一键安装。\n\n```shell\npip install -r requirements.txt\n```\n\n### 1.15 LICENSE（许可｜可选）\n\n如果是开源项目，需要添加许可证。\n\n参考：\n\n1. [一文看懂开源许可证丨开源知识科普](https://pingcap.com/zh/blog/introduction-of-open-source-license)\n2. [如何选择开源许可证？](https://www.ruanyifeng.com/blog/2011/05/how_to_choose_free_software_licenses.html)\n\n## 2 环境准备\n\n- Ubuntu 18.04 LTS+\n- Python 3.7+\n- Anaconda 3\n\n\u003e 人生苦短，快用 `*NIX` ！\n\n## 3 TODO\n\n- [ ] dspt init 脚本\n- [ ] script 中添加一些常用脚本\n\n## 4 参考\n\n- [Ubuntu 系统镜像下载](https://cn.ubuntu.com/download)\n- [Anaconda 个人版](https://www.anaconda.com/products/individual#)\n- [TUNA 清华大学开源软件镜像站](https://mirrors.tuna.tsinghua.edu.cn/)\n\n## 5 许可证\n\n[![](https://award.dovolopor.com?lt=License\u0026rt=MIT\u0026rbc=green)](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdovolopor-research%2Fdata-science-project-template","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdovolopor-research%2Fdata-science-project-template","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdovolopor-research%2Fdata-science-project-template/lists"}