{"id":13528452,"url":"https://github.com/MillionIntegrals/vel","last_synced_at":"2025-04-01T13:32:53.139Z","repository":{"id":32864033,"uuid":"132808324","full_name":"MillionIntegrals/vel","owner":"MillionIntegrals","description":"Velocity in deep-learning research","archived":false,"fork":false,"pushed_at":"2022-12-08T04:50:05.000Z","size":1619,"stargazers_count":276,"open_issues_count":16,"forks_count":33,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-29T12:08:13.203Z","etag":null,"topics":["convolutional-neural-networks","deep-learning","python","pytorch","reinforcement-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MillionIntegrals.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-09T20:16:48.000Z","updated_at":"2024-09-14T07:18:10.000Z","dependencies_parsed_at":"2023-01-14T22:28:17.634Z","dependency_job_id":null,"html_url":"https://github.com/MillionIntegrals/vel","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MillionIntegrals%2Fvel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MillionIntegrals%2Fvel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MillionIntegrals%2Fvel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MillionIntegrals%2Fvel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MillionIntegrals","download_url":"https://codeload.github.com/MillionIntegrals/vel/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246647744,"owners_count":20811375,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["convolutional-neural-networks","deep-learning","python","pytorch","reinforcement-learning"],"created_at":"2024-08-01T07:00:19.353Z","updated_at":"2025-04-01T13:32:52.751Z","avatar_url":"https://github.com/MillionIntegrals.png","language":"Python","readme":"# Vel 0.3\n\n[![Build Status](https://travis-ci.org/MillionIntegrals/vel.svg?branch=master)](https://travis-ci.org/MillionIntegrals/vel)\n[![PyPI version](https://badge.fury.io/py/vel.svg)](https://badge.fury.io/py/vel)\n[![GitHub](https://img.shields.io/github/license/mashape/apistatus.svg)](https://github.com/MillionIntegrals/vel/blob/master/LICENSE)\n[![Gitter chat](https://badges.gitter.im/MillionIngegrals/vel.png)](https://gitter.im/deep-learning-vel)\n\n\nBring **velocity** to deep-learning research.\n\n\nThis project hosts a collection of **highly modular** deep learning components that are tested to be working well together.\nA simple yaml-based system ties these modules together declaratively using configuration files,\nbut everything that can be defined using config files can be coded directly in the python script as well.\n\n\nThis is still an early version and a hobby project so documentation is unfortunately nonexistent. I've tried to make the\ncode as clear as possible, and provide many usage examples, but whenever there was a tradeoff to be made between \nsimplicity and modularity I've chosen modularity first and simplicity second.\n\n\nHaving conducted a few research projects, I've gathered a small collection of repositories \nlying around with various model implementations suited to a particular usecase. \nUsually, starting a new project involved copying pieces of code from \none or multiple of these past experiments, gluing, tweaking and debugging\nthem until the code started working in a new setting. \n\n\nAfter repeating that pattern multiple times, I've decided that this is the\ntime to bite the bullet and start organising deep learning models\ninto a structure that is designed to be reused rather than copied over.\n\n\nAs a goal, it should be enough to write a config file that\nwires existing components together and defines their hyperparameters\nfor most common applications.\nIf that's not the case few bits of custom glue code should do the job.\n\n\nThis repository is still in an early stage of that journey but it will grow\nas I'll be putting work into it.\n\n\n# Blogposts\n\n- https://blog.millionintegrals.com/vel-pytorch-meets-baselines/\n\n\n# How to run it \n\nProject can be installed from PyPi via `pip install vel` but also can be checked out from github\nand installed directly by running\n```bash\npip install -e .\n```\nfrom the repository root directory.\n\nThis project requires Python at least 3.6 and PyTorch 1.0.\nIf you want to run YAML config examples, you'll also need a **project configuration file**\n`.velproject.yaml`. An example is included in this repository.\n\nDefault project configuration writes\nmetrics to MongoDB instance open on localhost port 27017 and Visdom instance \non localhost port 8097. \n\nIf you don't want to run these services, there is included\nanother example file `.velproject.dummy.yaml`\nthat writes training progress to the standard output only.\nTo use it, just rename it to `.velproject.yaml`.\n\n\n# Features\n\n- Models should be runnable from the configuration files\n  that are easy to store in version control, generate automatically and diff.\n  Codebase should be generic and do not contain any of the model hyperparameters.\n  Unless user intervenes, it should be obvious which model was run\n  with which hyperparameters and what output it gave.\n- The amount of \"magic\" in the framework should be limited and it should be easy to\n  understand what exactly the model is doing for newcomers already comfortable with PyTorch. \n- All state-of-the-art models should be implemented in the framework with accuracy\n  matching published results.\n- All common deep learning workflows should be fast to implement, while \n  uncommon ones should be possible, at least as far as PyTorch allows.\n  \n  \n# Implemented models - Computer Vision\n\nSeveral models are already implemented in the framework and have example config files\nthat are ready to run and easy to modify for other similar usecases:\n\n- State-of-the art results on Cifar10 dataset using residual networks\n- Cats vs dogs classification using transfer learning from a resnet34 model pretrained on \n  ImageNet\n  \n# Implemented models - Natural language processing\n\n- Character-level language models based on LSTM and GRU recurrent networks, with example trained on\n  works of Shakespeare\n- Sentiment analysis of IMDB movie reviews\n  \n# Implemented models - Reinforcement learning\n\n- Continuous and discrete action spaces\n- Basic support for LSTM policies for A2C and PPO\n- Following published policy gradient reinforcement learning algorithms:\n    - Advantage Actor-Critic (A2C)\n    - Deep Deterministic Policy Gradient (DDPG)\n    - Proximal Policy Optimization (PPO)\n    - Trust Region Policy Optimization (TRPO)\n    - Actor-Critic with Experience Replay (ACER)\n- Deep Q-Learning (DQN) as described by DeepMind in their Nature publication with following \n  improvements:\n    - Double DQN\n    - Dueling DQN\n    - Prioritized experience replay\n    - N-Step Bellman updates\n    - Distributional Q-Learning\n    - Noisy Networks for Exploration\n    - Rainbow (combination of the above)\n\n\n# Examples\n\nMost of the examples for this framework are defined using config files in the\n`examples-configs` directory with sane default hyperparameters already selected.\n\nFor example, to run the A2C algorithm on a Breakout atari environment, simply invoke:\n\n```\npython -m vel.launcher examples-configs/rl/atari/a2c/breakout_a2c.yaml train\n```\n\nIf you install the library locally, you'll have a special wrapper created\nthat will invoke the launcher for you. Then, above becomes:\n\n```\nvel examples-configs/rl/atari/a2c/breakout_a2c.yaml train\n```\n\nGeneral command line interface of the launcher is:\n\n```\npython -m vel.launcher CONFIGFILE COMMAND --device PYTORCH_DEVICE -r RUN_NUMBER -s SEED\n```\n\nWhere `PYTORCH_DEVICE` is a valid name of pytorch device, most likely `cuda:0`.\nRun number is a sequential number you wish to record your results with.\n\nIf you prefer to use the library from inside your scripts, take a look at the \n`examples-scripts` directory. From time to time I'll be putting some examples in there as\nwell. Scripts generally don't require any MongoDB or Visdom setup, so they can be run straight\naway in any setup, but their output will be less rich and less informative.\n\nHere is an example script running the same setup as a config file from above:\n\n```python\nimport torch\nimport torch.optim as optim\n\nfrom vel.rl.metrics import EpisodeRewardMetric\nfrom vel.storage.streaming.stdout import StdoutStreaming\nfrom vel.util.random import set_seed\n\nfrom vel.rl.env.classic_atari import ClassicAtariEnv\nfrom vel.rl.vecenv.subproc import SubprocVecEnvWrapper\n\nfrom vel.modules.input.image_to_tensor import ImageToTensorFactory\nfrom vel.rl.models.stochastic_policy_model import StochasticPolicyModelFactory\nfrom vel.rl.models.backbone.nature_cnn import NatureCnnFactory\n\n\nfrom vel.rl.reinforcers.on_policy_iteration_reinforcer import (\n    OnPolicyIterationReinforcer, OnPolicyIterationReinforcerSettings\n)\n\nfrom vel.rl.algo.policy_gradient.a2c import A2CPolicyGradient\nfrom vel.rl.env_roller.step_env_roller import StepEnvRoller\n\nfrom vel.api.info import TrainingInfo, EpochInfo\n\n\ndef breakout_a2c():\n    device = torch.device('cuda:0')\n    seed = 1001\n\n    # Set random seed in python std lib, numpy and pytorch\n    set_seed(seed)\n\n    # Create 16 environments evaluated in parallel in sub processess with all usual DeepMind wrappers\n    # These are just helper functions for that\n    vec_env = SubprocVecEnvWrapper(\n        ClassicAtariEnv('BreakoutNoFrameskip-v4'), frame_history=4\n    ).instantiate(parallel_envs=16, seed=seed)\n\n    # Again, use a helper to create a model\n    # But because model is owned by the reinforcer, model should not be accessed using this variable\n    # but from reinforcer.model property\n    model = StochasticPolicyModelFactory(\n        input_block=ImageToTensorFactory(),\n        backbone=NatureCnnFactory(input_width=84, input_height=84, input_channels=4)\n    ).instantiate(action_space=vec_env.action_space)\n\n    # Reinforcer - an object managing the learning process\n    reinforcer = OnPolicyIterationReinforcer(\n        device=device,\n        settings=OnPolicyIterationReinforcerSettings(\n            batch_size=256,\n            number_of_steps=5,\n        ),\n        model=model,\n        algo=A2CPolicyGradient(\n            entropy_coefficient=0.01,\n            value_coefficient=0.5,\n            max_grad_norm=0.5,\n            discount_factor=0.99,\n        ),\n        env_roller=StepEnvRoller(\n            environment=vec_env,\n            device=device,\n        )\n    )\n\n    # Model optimizer\n    optimizer = optim.RMSprop(reinforcer.model.parameters(), lr=7.0e-4, eps=1e-3)\n\n    # Overall information store for training information\n    training_info = TrainingInfo(\n        metrics=[\n            EpisodeRewardMetric('episode_rewards'),  # Calculate average reward from episode\n        ],\n        callbacks=[StdoutStreaming()]  # Print live metrics every epoch to standard output\n    )\n\n    # A bit of training initialization bookkeeping...\n    training_info.initialize()\n    reinforcer.initialize_training(training_info)\n    training_info.on_train_begin()\n\n    # Let's make 100 batches per epoch to average metrics nicely\n    num_epochs = int(1.1e7 / (5 * 16) / 100)\n\n    # Normal handrolled training loop\n    for i in range(1, num_epochs+1):\n        epoch_info = EpochInfo(\n            training_info=training_info,\n            global_epoch_idx=i,\n            batches_per_epoch=100,\n            optimizer=optimizer\n        )\n\n        reinforcer.train_epoch(epoch_info)\n\n    training_info.on_train_end()\n\n\nif __name__ == '__main__':\n    breakout_a2c()\n```\n\n# Docker\n\nDockerized version of this library is available in from the Docker Hub as\n`millionintegrals/vel`. Link: https://hub.docker.com/r/millionintegrals/vel/\n\n# PyPI\n\n```\npip install vel\n```\n\nor\n\n```\npip install vel[gym,mongo,visdom]\n```\n\n# Projects using Vel\n\n- https://github.com/MillionIntegrals/baselines-experiments\n- https://github.com/MillionIntegrals/vel-miniworld\n\n# Glossary\n\nFor a glossary of terms used in the library please refer to [Glossary](docs/Glossary.md).\nIf there is anything you'd like to see there, feel free to open an issue or make a pull request.\n\n# Bibliography\n\nFor a more or less exhaustive bibliography please refer to [Bibliography](docs/Bibliography.md).\n\n\n# Roadmap\n\nFor each major version I'll try to keep master branch stable together with what's\ncurrently published on PyPI. At the same time I'll proceed with implementing new\nfeatures on a release branch that will be merged after the testing is done and\na release is ready.\n\nBelow is a hypothetical set of features I somehow speculate to include in version\n0.4 of Vel:\n\nVery likely to be included:\n- Neural machine translation using RNNs and Transformer Networks\n- Soft actor-critic\n- Twin Delayed DDPG\n\n\nPossible to be included:\n- Popart reward normalization\n- Parameter Space Noise for Exploration\n- Hindsight experience replay\n- Generative adversarial networks\n\n\nCode quality:\n- Rename models to policies\n- Force dictionary inputs and outputs for policies\n- Factor action noise back into the policy\n- Use linter as a part of the build process\n\n\n# Citing\n\nIf you use `vel` in your research, you can cite it as follows:\n\n```\n@misc{tworek2018vel,\n  author = {Jerry Tworek},\n  title = {vel},\n  year = {2018},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/MillionIntegrals/vel}},\n}\n```\n\n\n# Alternatives, similar projects\n\n- https://github.com/NervanaSystems/coach\n- https://github.com/google/dopamine\n- https://github.com/openai/baselines\n- https://github.com/rlpy/rlpy\n- https://github.com/rlworkgroup/garage\n- https://github.com/unixpickle/anyrl-py\n- https://github.com/zuoxingdong/lagom\n- https://github.com/inoryy/reaver-pysc2\n","funding_links":[],"categories":["Libraries","Pytorch \u0026 related libraries｜Pytorch \u0026 相关库","Python","Pytorch \u0026 related libraries"],"sub_categories":["Other libraries｜其他库:","Other libraries:"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMillionIntegrals%2Fvel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMillionIntegrals%2Fvel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMillionIntegrals%2Fvel/lists"}