{"id":47197202,"url":"https://github.com/aebeljs/verlpy","last_synced_at":"2026-03-13T12:01:01.342Z","repository":{"id":62587282,"uuid":"311044660","full_name":"aebeljs/VeRLPy","owner":"aebeljs","description":"VeRLPy is an open-source python library developed to improve the digital hardware verification process by using Reinforcement Learning (RL). It provides a generic Gym environment implementation for building cocotb-based testbenches for verifying any hardware design.","archived":false,"fork":false,"pushed_at":"2022-10-05T19:35:57.000Z","size":113452,"stargazers_count":29,"open_issues_count":2,"forks_count":4,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-11-28T12:46:09.904Z","etag":null,"topics":["hardware-verification","reinforcement-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aebeljs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-08T11:14:05.000Z","updated_at":"2025-10-18T13:44:34.000Z","dependencies_parsed_at":"2022-11-03T22:16:00.396Z","dependency_job_id":null,"html_url":"https://github.com/aebeljs/VeRLPy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/aebeljs/VeRLPy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aebeljs%2FVeRLPy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aebeljs%2FVeRLPy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aebeljs%2FVeRLPy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aebeljs%2FVeRLPy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aebeljs","download_url":"https://codeload.github.com/aebeljs/VeRLPy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aebeljs%2FVeRLPy/sbom","scorecard":{"id":168647,"data":{"date":"2025-08-11","repo":{"name":"github.com/aebeljs/VeRLPy","commit":"f18aff50cc4cb8f52345c55080325bfdb28c597b"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.5,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":0,"reason":"Found 1/29 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.txt:0","Info: FSF or OSI recognized license: MIT License: LICENSE.txt:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 2 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-16T15:43:36.424Z","repository_id":62587282,"created_at":"2025-08-16T15:43:36.424Z","updated_at":"2025-08-16T15:43:36.424Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30466557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-13T11:00:43.441Z","status":"ssl_error","status_checked_at":"2026-03-13T11:00:23.173Z","response_time":60,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hardware-verification","reinforcement-learning"],"created_at":"2026-03-13T12:00:38.679Z","updated_at":"2026-03-13T12:01:01.336Z","avatar_url":"https://github.com/aebeljs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VeRLPy\n** **Under Construction** **\n\n[VeRLPy](https://arxiv.org/abs/2108.03978) is an open-source python library developed to improve the digital hardware verification process by using Reinforcement Learning (RL). It provides a generic [Gym](https://gym.openai.com/) environment implementation for building [cocotb](https://docs.cocotb.org/en/stable/)-based testbenches for verifying any hardware design.\n\n## Installation\nThe easiest way to start using VeRLPy is to install it using `pip install verlpy`\n\nVeRLPy is currently dependent on OpenAI [Gym](https://gym.openai.com/), [cocotb](https://docs.cocotb.org/en/stable/), [cocotb-bus](https://github.com/cocotb/cocotb-bus), and [Stable Baselines3](https://stable-baselines3.readthedocs.io/en/master/). These packages should get installed alongside VeRLPy when installing using `pip`. For running the verification, a simulator compatible with cocotb is additionally required. Please refer to the official  [cocotb](https://docs.cocotb.org/en/stable/) documentation to set this up.\n\n## Usage Guide\nHaving familiarity with [cocotb](https://docs.cocotb.org/en/stable/), OpenAI [Gym](https://gym.openai.com/) and [this whitepaper on VeRLPy](https://arxiv.org/abs/2108.03978) will be very beneficial to get started with the VeRLPy library.\n\nThe hardware design provided in Verilog or VHDL is simulated by cocotb using the chosen simulator. VeRLPy aims to offer a clean interface for bringing RL logic into the conventional cocotb testbench while adhering to the OpenAI Gym environment structure allowing users to leverage the standard RL tools. The DUT and the verification testbench are the environment for the RL agent to act on. The agent chooses an action which is executed on the environment and the consequences of that action are informed back to the agent in terms of the state and the reward. \n\n### Identifying verification goals and defining the MDP\n\nTo build a testbench using VeRLPy, we need to start by defining the verification goals in terms of functional coverage. This involves identifying the events occurring in the DUT that correspond to the features that are part of the design specifications. The reward given to the RL agent will be a function of how often these events occur during the simulation.\n\nIn addition, since there is the additional RL component involved, the MDP has to be defined. This means identifying what each timestep and each episode corresponds to in terms of verification logic and input stimulus to the DUT, and what the state and the action space of the MDP represent. \n\nEach RL episode starts with a call to to the `reset()` function of the Gym environment. Followed by this, there are one or more calls to the `step(action)` function until `done` is returned with a value of `True` from the RL environment.  Refer to [Gym](https://gym.openai.com/) for a more detailed understanding of how the control flow occurs in a Gym environment. \n\n### Inheriting CocotbEnv\n\nThe library provides a Gym environment object `CocotbEnv` which can be inherited to build the verification testbench. This class has predefined functions/coroutines which interact with the RL agent to facilitate the learning process. These functions are triggered by the `reset()`, `step(action)`, `done` variable, etc. Please refer [here](https://github.com/aebeljs/VeRLPy/blob/main/src/cocotb_env.py) for more detailed explanations of these functions and their implementation details. We start with implementing this class that inherits `CocotbEnv`. \n\n```python\n# test_my_example_design.py\n\nimport cocotb\nfrom verlpy import CocotbEnv\n\nclass MyExampleDesignCocotbEnv(CocotbEnv):\n    def __init__(self, dut, observation_space):\n        super().__init__()\n        self.dut = dut # DUT object used for cocotb-based verification\n        self.observation_space = observation_space # state space of the RL agent\n\n        # add here any \"self.\" variables that need to be accessed in\n        # other functions below\n\n    @cocotb.coroutine\n    def setup_rl_episode(self):\n        # add here the logic to be \n        # executed on each call to reset() by the RL agent\n\n    @cocotb.coroutine\n    def rl_step(self):\n        # add here the verification logic to be \n        # executed on each call to step() by the RL agent\n        \n\n    @cocotb.coroutine\n    def terminate_rl_episode(self):\n        # add here the logic to be executed at the end\n        # of each RL episode when done == 1 for the Gym env\n\n\n    def finish_experiment(self):\n        # add here the logic to be executed after all\n        # the episodes are completed\n```\nNote that all the coroutines with the decorator `cocotb.coroutine` require a `yield` statement in the body like how it is in standard cocotb testbenches.\n\n### Instantiating the verification environment object\n\nThe functions/coroutine implementations defined in `MyExampleDesignCocotbEnv` should contain the appropriate logic that must run from the cocotb side. Once this class is defined, the cocotb test can be added to invoke the verification logic from this class. While defining this, the state space of the MDP should also be passed as an argument as shown below.\n\n```python\n# test_my_example_design.py\n\nimport cocotb\nfrom verlpy import CocotbEnv, utils\nimport gym\n\nclass MyExampleDesignCocotbEnv(CocotbEnv):\n    def __init__(self, dut, observation_space):\n        super().__init__()\n        self.dut = dut\n        self.observation_space = observation_space\n\n        # add here any \"self.\" variables that need to be accessed in\n        # other functions below\n\n    @cocotb.coroutine\n    def setup_rl_episode(self):\n        # add here the logic to be \n        # executed on each call to reset() by the RL agent\n\n    @cocotb.coroutine\n    def rl_step(self):\n        # add here the verification logic to be \n        # executed on each call to step() by the RL agent\n        \n\n    @cocotb.coroutine\n    def terminate_rl_episode(self):\n        # add here the logic to be executed at the end\n        # of each RL episode when done == 1 for the Gym env\n\n\n    def finish_experiment(self):\n        # add here the logic to be executed after all\n        # the episodes are completed\n\n# entry point for the cocotb verification test\n@cocotb.test()\ndef run_test(dut):\n    cocotb_env = MyExampleDesignCocotbEnv(dut, gym.spaces.Discrete(1))\n    # gym.spaces.Discrete(1) =\u003e Just 1 state in the state space\n    yield cocotb_env.run()\n\n    # plot the results of the verification experiment\n    utils.visualize(cocotb_env.log_file_name)\n```\n\nVeRLPy also provides some plotting capabilities which can be accessed from `utils` as shown above.\n\n### Adding coroutines to track events\n\nThe identified functional coverage events can be tracked by using cocotb coroutines like in conventional cocotb-based verification.\n\n```python\n# test_my_example_design.py\n\nimport cocotb\nfrom verlpy import CocotbEnv, utils\nimport gym\n\nclass MyExampleDesignCocotbEnv(CocotbEnv):\n    def __init__(self, dut, observation_space):\n        super().__init__()\n        self.dut = dut\n        self.observation_space = observation_space\n\n        # add here any \"self.\" variables that need to be accessed in\n        # other functions below\n\n    @cocotb.coroutine\n    def setup_rl_episode(self):\n        # add here the logic to be \n        # executed on each call to reset() by the RL agent\n        self.cocotb_coverage.clear() # clear last episode's coverage\n        self.coverage_coroutine = cocotb.fork(monitor_signals(self.dut, self.cocotb_coverage))\n\n    @cocotb.coroutine\n    def rl_step(self):\n        # add here the verification logic to be \n        # executed on each call to step() by the RL agent\n        \n\n    @cocotb.coroutine\n    def terminate_rl_episode(self):\n        # add here the logic to be executed at the end\n        # of each RL episode when done == 1 for the Gym env\n\n        self.coverage_coroutine.kill()\n\n\n    def finish_experiment(self):\n        # add here the logic to be executed after all\n        # the episodes are completed\n\n@cocotb.coroutine\ndef monitor_signals(dut, cocotb_coverage):\n    while True:\n        yield RisingEdge(dut.CLK)\n        s = [(int)(dut.reg_1.value == 1),\n             (int)((dut.reg_2.value) % 4 != 0),\n             (int)(dut.reg_3.value == 32)]\n        # Here reg_1, reg_2 and reg_3 are some\n        # key registers of interest in the DUT\n\n        s = ''.join(map(str, s))\n        cocotb_coverage.append(s)\n\n# entry point for the cocotb verification test\n@cocotb.test()\ndef run_test(dut):\n    cocotb_env = MyExampleDesignCocotbEnv(dut, gym.spaces.Discrete(1))\n    # gym.spaces.Discrete(1) =\u003e Just 1 state in the state space\n    yield cocotb_env.run()\n\n    # plot the results of the verification experiment\n    utils.visualize(cocotb_env.log_file_name)\n```\nThe `monitor_signals` coroutine added above monitors the DUT for events of interest that count towards the functional coverage. The  boolean logical expressions in the list `s` above correspond to the logical expressions for identifying each event. The number of times these events occur affect the reward signal given to the RL agent. `monitor_signals` should track these events and add them to the `cocotb_coverage` attribute of the `MyExampleDesignCocotbEnv` class that we wrote. `monitor_signals` is invoked in the `setup_rl_episode` coroutine along with the clock and reset coroutines. It is passed the `cocotb_coverage` attribute as an argument. Note that `monitor_signals` is killed in the `terminate_rl_episode` coroutine at the end of each RL episode. This is important for all coroutines since it might otherwise lead to performance issues with multiple \"alive\" coroutines still ongoing from previous episodes.\n\n### Configuration File\nA configuration file `config.ini` needs to be provided to specify the parameters related to the simulation and the RL agent. A sample coniguration file is provided below with comments for what each section and key corresponds to.\n\n```ini\n; This section is to provide the\n; main parameters for the verification runs\n[main]\n; number of RL steps for which the experiment is run\nnum_steps = 1000\n\n; number of functional events tracked\nnum_events = 3\n\n; weightage of each functional event for reward computation\nreward_function = [0, 0, 1]\n\n; set log_step as 0 for logging just aggregated results and 1 for logging details in each step\nlog_step = 0\n\n; set mode as 0 to generate the random baseline without RL and 1 for using RL\nmode = 1\n\n; specify the stable_baselines3 algorithm to be used from SAC, DDPG and TD3\nalgorithm = SAC\n\n; fsm_states contains the regex patterns for \n; state-based binary sequence generation\n; (leave as [] unless utils.get_next_state_of_FSM() is needed in the code)\nfsm_states = []\n\n; Provide the discrete action component names here. \n; The valid dscrete value set for the specified keys\n; should be given in the [discrete] section\ndiscrete_params = ['count_width', 'fmap_len']\n\n\n; This section is to provide the bounds\n; of the continuous dimensions of the action space.\n; If multiple dimensions are there, provide the list of bounds for each dimension\n; eg: lower_bounds = [0, 0, 1] and upper_bounds = [1, 1, 3] corresponds to\n; [0, 1] x [0, 1] x [1, 3] as the continuous action space\n[continuous]\nlower_bounds = [0, 5]\nupper_bounds = [1, 7]\n\n\n; This section is to provide the list of valid \n; discrete values for each discrete action\n; component named in discrete_params\n[discrete]\ncount_width = [1, 2, 3, 4, 5, 6, 7, 8]\nfmap_len = [100, 200, 300, 400, 500, 600, 700, 800, 900, 1000]\n\n\n; This section is to provide the required\n; hyperparameters of the chosen stable_baselines3 algorithm\n[RL]\npolicy = 'MlpPolicy'\nlearning_starts = 100\nlearning_rate = 0.0003\ntrain_freq = (1, 'episode')\nverbose = 1\n```\nThe `reward_function` key specifies how the functional events tracked in the `self.cocotb_coverage` attribute need to be rewarded for improving coverage. `reward_function` set as `[0, 0, 1]` like above implies that if the third functional event occurs during a step, a reward is given to the RL agent. Refer to the [paper](https://arxiv.org/abs/2108.03978) for the actual computation details.\n\nThe `[continuous]` and `[discrete]` sections together specify the total action space of the RL agent. The continuous dimensions of the action space based on the above configuration file is the cross product `[0, 1] x [5, 7]`. The discrete dimensions of the the action space is the cross product `{1, 2, ..., 8} x {100, 200, ..., 1000}`. Therefore the complete action space is the cross product `[0, 1] x [5, 7] x {1, 2, ..., 8} x {100, 200, ..., 1000}`.\n\n### Filling in the verification logic\n\nFinally, the body of each of the coroutines overriden in `MyExampleDesignCocotbEnv` need to be completed. The action suggested by the RL agent based on the `config.ini` can be accessed through the `self.continuous_actions` and `self.discrete_actions` attributes of the class. In the above example, `self.continuous_actions` will sample from `[0, 1] x [5, 7]` and `self.discrete_actions` will sample from `{1, 2, ..., 8} x {100, 200, ..., 1000}`.\n\nThe list `self.cocotb_coverage` needs to be updated with the strings corresponding to the covered events from the previous timestep of the RL episode for proper reward computation based on the reward function defined in the `config.ini` file. This update will happen on its own if `self.cocotb_coverage` is passed as the argument `cocotb_coverage` to the  `monitor_signals` coroutine defined above  Refer to the examples folder for more concrete examples on how this is done in various designs.\n\n### Multi-step RL\n\nVeRLPy by default assumes a single step single state MDP. If a multi-step MDP is required, it can be implemented by overriding `compute_rl_observation` function in the `MyExampleDesignCocotbEnv` class. The internal elements of the DUT that need to be tracked for computing the observation/state after each step can be done so by utilizing a separate coroutine like how `monitor_signals` is used for tracking the coverage to compute the reward.\n\n### Make file\n\nThe make file can be written like how it is done usually in cocotb testbenches. Once it is done and `make` is run, the verification simulation runs and the logs, models and plots are generated. Use the `mode` configuration parameter in `config.ini` for running the verification with/without the RL feedback.\n\n## Citation\n\nIf you find this work useful, consider citing it with this BibTex:\n```\n@article{DBLP:journals/corr/abs-2108-03978,\n  author    = {Aebel Joe Shibu and\n               Sadhana S and\n               Shilpa N and\n               Pratyush Kumar},\n  title     = {VeRLPy: Python Library for Verification of Digital Designs with Reinforcement\n               Learning},\n  journal   = {CoRR},\n  volume    = {abs/2108.03978},\n  year      = {2021},\n  url       = {https://arxiv.org/abs/2108.03978},\n  eprinttype = {arXiv},\n  eprint    = {2108.03978},\n  timestamp = {Wed, 11 Aug 2021 15:24:08 +0200},\n  biburl    = {https://dblp.org/rec/journals/corr/abs-2108-03978.bib},\n  bibsource = {dblp computer science bibliography, https://dblp.org}\n}\n```\n\n## Change Log\n08-08-2021 - Usage guide added\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faebeljs%2Fverlpy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faebeljs%2Fverlpy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faebeljs%2Fverlpy/lists"}