{"id":14971626,"url":"https://github.com/lumo-tech/lumo","last_synced_at":"2025-10-26T16:30:54.884Z","repository":{"id":57475165,"uuid":"264152308","full_name":"lumo-tech/lumo","owner":"lumo-tech","description":"More light-weight pytorch experiment management library!","archived":false,"fork":false,"pushed_at":"2023-04-09T04:34:44.000Z","size":8086,"stargazers_count":63,"open_issues_count":3,"forks_count":9,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-31T21:34:21.476Z","etag":null,"topics":["deep-learning","experiment-management","experiment-tracking","pytorch","version-control"],"latest_commit_sha":null,"homepage":"https://lumo-tech.github.io/lumo/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lumo-tech.png","metadata":{"files":{"readme":"README.ch.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-05-15T09:28:24.000Z","updated_at":"2024-09-11T18:37:21.000Z","dependencies_parsed_at":"2023-07-12T14:49:00.013Z","dependency_job_id":null,"html_url":"https://github.com/lumo-tech/lumo","commit_stats":null,"previous_names":["pytorch-lumo/lumo","sailist/thexp"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lumo-tech%2Flumo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lumo-tech%2Flumo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lumo-tech%2Flumo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lumo-tech%2Flumo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lumo-tech","download_url":"https://codeload.github.com/lumo-tech/lumo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238366789,"owners_count":19460186,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","experiment-management","experiment-tracking","pytorch","version-control"],"created_at":"2024-09-24T13:45:34.382Z","updated_at":"2025-10-26T16:30:53.933Z","avatar_url":"https://github.com/lumo-tech.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lumo\n\n[![PyPI version](https://badge.fury.io/py/lumo.svg)](https://badge.fury.io/py/lumo)\n![Python-Test](https://github.com/lumo-tech/lumo/actions/workflows/python-test.yml/badge.svg)\n[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/Lightning-AI/lightning/blob/master/LICENSE)\n![Python-doc](./images/docstr_coverage_badge.svg)\n\n`lumo` 是一个精简高效的库，简化了实验所需的所有组件的管理，并特别关注增强深度学习实践者的体验。\n\n- 实验管理：: 为每次运行分配唯一路径，区分不同类型的文件并存储；通过 git 管理代码快照；记录实验中产生的一切信息，保障可回溯、可复现\n- 参数管理：基于 fire 提供比 argparser 更便捷的参数管理\n- 运行时配置：提供多级作用域下的配置管理\n- 可视化：基于 [Panel](https://panel.holoviz.org/index.html) 提供可交互的 jupyter 实验管理面板\n- 为深度学习提供额外的优化\n    - 训练：基于 Trainer 提供可任意扩展的训练逻辑简化，并提供完善的回调逻辑\n    - 优化器：参数与优化器构建一体化\n    - 数据: 数据集构建流程抽象、组合多个 DataLoader、...\n    - 分布式训练：同样支持多种训练加速框架，统一抽象，方便随时切换\n- 更多工具类...\n- 尽可能多的支持现代 IDE 的自动补全\n\n![lumo-framework](./images/lumo-intro.png)\n\n# :book: 目录\n\n- :book: [目录](#book-目录)\n- :cloud: [安装](#cloud-安装)\n- :book: [快速开始](#book-快速开始)\n  - :small_orange_diamond: [已有项目嵌入](#small_orange_diamond-已有项目嵌入)\n  - :small_orange_diamond: [从零开始](#small_orange_diamond-从零开始)\n  - :small_orange_diamond: [可视化界面](#small_orange_diamond-可视化界面)\n  - :small_orange_diamond: [复现实验](#small_orange_diamond-复现实验)\n  - :small_orange_diamond: [备份](#small_orange_diamond-备份)\n- [More](#more)\n- :pencil: [Acknowledge](#pencil-acknowledge)\n- :scroll: [License](#scroll-license)\n- [完整文档](https://lumo-tech.github.io/lumo/)\n\n# :cloud: 安装\n\n安装已发布的通过了所有测试的版本\n\n```bash\npip install -U lumo\n```\n\n或从 dev1 分支安装最新版本：\n\n```bash\npip install git+https://github.com/lumo-tech/lumo\n```\n\n实验面板依赖于 panel，需要额外安装：\n\n```\npip install panel\n```\n\n# :book: 快速开始\n\n以下是几个经典场景：\n\n## :small_orange_diamond: 已有项目嵌入\n\n对已有项目，可以通过以下方式快速嵌入\n\n- 引入\n\n```python\nimport random\nfrom lumo import SimpleExperiment, Params, Logger, Meter, Record\n```\n\n- 初始化 Logger 和 Experiment\n\n```python\nlogger = Logger()\n# 定义及使用，无需转换\nexp = SimpleExperiment(exp_name='my_exp_a')  # 为每种实验手动定义唯一名称\nexp.start()\nlogger.add_log_dir(exp.mk_ipath())\n```\n\n- 初始化参数（代替 argparse 等）\n\n```python\n# 替换基于 argparse 等的参数定义方法\nparams = Params()\nparams.dataset = params.choice('cifar10', 'cifar100')\nparams.alpha = params.arange(default=1, left=0, right=10)\nparams.from_args()  # python3 train.py --dataset=cifar100 --alpha=0.2\nprint(params.to_dict())  # {\"dataset\": \"cifar100\", \"alpha\": 0.2}\n```\n\n- 在训练过程中记录参数、存储信息（代替手动管理路径、自己维护 AvgItem）\n\n```python\n# 记录实验参数\nexp.dump_info('params', params.to_dict())\nprint(exp.test_name)  # 为每次实验自动分配唯一名称\n\n# 基于命名空间提供本次实验的唯一路径\n# 元数据和二进制大文件分离，方便清理\nparams.to_yaml(exp.mk_ipath('params.yaml'))\n\nfor i in range(10):\n    # 记录实验指标\n    max_acc = exp.dump_metric('Acc', random.random(), cmp='max')\n    logger.info(f'Max acc {max_acc}')\n\n    # 存储大文件/二进制文件（如模型权重）\n    ckpt_fn = exp.mk_bpath('checkpoints', f'model_{i}.ckpt')\n    ...  # 保存代码 given ckpt_fn\n\nrecord = Record()\nfor batch in range(10):\n    m = Meter()\n    m.mean.Lall = random.random()\n    m.last.lr = batch\n    record.record(m)\n    logger.info(record)\n\n# 主动结束实验，补充元信息。也可以在进程结束后由 hook 自动结束，支持针对异常的记录\nexp.end()\n```\n\n## :small_orange_diamond: 从零开始\n\n如果从新开始一个深度学习实验，那么可以使用 lumo 全方位的加速代码的构建，下面提供了多个不同规模下使用 lumo 训练的示例：\n\n单文件：\n\n| 示例                                     | CoLab | 代码行数 |\n|----------------------------------------|-------|------|\n| [MNIST 示例](./examples/mnist.py)        |       | 118  |\n| [MocoV2 训练 CIFAR10](./examples/moco.py) || 284   |\n| [多卡训练 ImageNet]()                      |||\n\n实验项目：\n\n| 项目                                                                                                        | 说明                            |\n|-----------------------------------------------------------------------------------------------------------|-------------------------------|\n| [image-classification](https://github.com/sailist/image-classification)                              | 集成了全监督、半监督、自监督的多个论文的复现代码      |\n| [emotion-recognition-in-coversation](https://github.com/lumo-tech/emotion-recognition-in-conversation) | 集成了对话情感分类、多模态对话情感分类的多个论文的复现代码 |\n\n## :small_orange_diamond: 可视化界面\n\n在 jupyter 中：\n\n```python\nfrom lumo import Watcher\n\nw = Watcher()\ndf = w.load()\nwidget = w.panel(df)\nwidget.servable()\n```\n\n![Panel](./images/panel-example.png)\n\n可视化手动筛选后的实验：\n![Panel](./images/panel-example2.png)\n\n可以直接使用命令行打开页面查看当前所有实验：\n\n```\nlumo board [--port, --address, --open]\n```\n\n## :small_orange_diamond: 复现实验\n\n对因为个中原因失败的实验，在藉由可视化界面观察并解决后，可以通过唯一实验 Id (test_name) 直接重跑，并对关键参数重新赋值：\n\n```\nlumo rerun 230313.030.57t --device=0\n```\n\n## :small_orange_diamond: 备份\n\n记录实验信息到 Github issue (基于 PyGitHub):\n\n```python\nfrom lumo import Experiment, Watcher\nfrom lumo import glob\n\nglob['github_access_token'] = 'ghp_*'  # `access_token` 的默认值，建议将 access_token 写在全局配置 `~/.lumorc.json` 中 \n\nw = Watcher()\ndf = w.load()\n\n# 选择单个实验备份\nexp = Experiment.from_cache(df.iloc[0].to_dict())\nissue = exp.backup('github', repo='sailist/image-classification-private',\n                   access_token='ghp_*',\n                   update=True,  # 如果已备份，则覆盖更新之前的 issue\n                   labels=None,  # 可选标签\n                   )\nprint(issue.number)\n\n# 批量备份，并且基于每个实验的参数添加标签\nissues = df.apply(\n    lambda x: Experiment.from_cache(x.to_dict()).backup(..., labels=[x['params'].get('dataset', '')]),\n    axis=1\n)\n```\n\n![backup_github](./images/backup_github.png)\n\n# Full properties\n\n```\n{'agent': nan,\n 'backup': {'23-03-17-003438': {'backend': 'github',\n                                'number': 9,\n                                'repo': '...'},\n            },\n 'exception': nan,\n 'execute': {'cwd': '~/Documents/Python/lumo',\n             'exec_argv': ['~/Documents/Python/lumo/a.py'],\n             'exec_bin': '~/.pyenv/versions/3.9.16/bin/python3.9',\n             'exec_file': '~/Documents/Python/lumo/a.py',\n             'repo': '~/Documents/Python/lumo'},\n 'exp_name': 'my_exp_a',\n 'git': {'commit': '1014b6b5',\n         'dep_hash': 'c93b8c4e340882f55cf0c8e125fa0203',\n         'repo': '~/Documents/Python/lumo'},\n 'hooks': {'Diary': {'loaded': True, 'msg': ''},\n           'FinalReport': {'loaded': True, 'msg': ''},\n           'GitCommit': {'loaded': True, 'msg': ''},\n           'LastCmd': {'loaded': True, 'msg': ''},\n           'LockFile': {'loaded': True, 'msg': ''},\n           'RecordAbort': {'loaded': True, 'msg': ''}},\n 'lock': {'accelerate': '0.16.0',\n          'decorator': '5.1.1',\n          'fire': '0.5.0',\n          'hydra': '1.3.2',\n          'joblib': '1.2.0',\n          'lumo': '0.15.0',\n          'numpy': '1.24.2',\n          'omegaconf': '2.3.0',\n          'psutil': '5.9.4',\n          'torch': '1.13.1'},\n 'note': 'This is a Note',\n 'params': {'alpha': 1, 'dataset': 'cifar10'},\n 'paths': {'blob_root': '~/.lumo/blob',\n           'cache_root': '~/.lumo/cache',\n           'info_root': '~/.lumo/experiments'},\n 'pinfo': {'hash': '0af4b77497c85bc5b65ccbdd9ff4ca0f',\n           'obj': {'argv': ['~/.pyenv/versions/3.9.16/bin/python3.9',\n                            '~/Documents/Python/lumo/a.py'],\n                   'pid': 63975,\n                   'pname': 'python3.9',\n                   'pstart': 1678898740.099484},\n           'pid': 63975},\n 'progress': {'end': '23-03-16-004542',\n              'end_code': 0,\n              'last_edit_time': '23-03-16-004542',\n              'ratio': 1,\n              'start': '23-03-16-004542',\n              'update_from': None},\n 'tags': [],\n 'test_name': '230316.000.8ct',\n 'trainer': nan}\n```\n\n# :pencil: Acknowledge\n\n从 2020 年维护至今。感谢 lumo 陪我见证我的研究生生涯。\n\n# :scroll: License\n\n采用 [Apache License Version 2.0](./LICENSE) 协议分发。\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flumo-tech%2Flumo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flumo-tech%2Flumo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flumo-tech%2Flumo/lists"}