{"id":19530034,"url":"https://github.com/goktug97/pepg-es","last_synced_at":"2025-08-19T03:10:41.464Z","repository":{"id":57451808,"uuid":"251397317","full_name":"goktug97/PEPG-ES","owner":"goktug97","description":"Python Implementation of Parameter-exploring Policy Gradients Evolution Strategy ","archived":false,"fork":false,"pushed_at":"2020-04-02T19:40:52.000Z","size":11730,"stargazers_count":17,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-18T13:27:39.763Z","etag":null,"topics":["artificial-intelligence","evolution-strategies","neural-network","policy-gradient"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/goktug97.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-30T18:43:17.000Z","updated_at":"2025-07-16T02:19:35.000Z","dependencies_parsed_at":"2022-09-04T10:41:57.786Z","dependency_job_id":null,"html_url":"https://github.com/goktug97/PEPG-ES","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/goktug97/PEPG-ES","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goktug97%2FPEPG-ES","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goktug97%2FPEPG-ES/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goktug97%2FPEPG-ES/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goktug97%2FPEPG-ES/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/goktug97","download_url":"https://codeload.github.com/goktug97/PEPG-ES/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goktug97%2FPEPG-ES/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271093078,"owners_count":24697928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-19T02:00:09.176Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","evolution-strategies","neural-network","policy-gradient"],"created_at":"2024-11-11T01:28:34.740Z","updated_at":"2025-08-19T03:10:41.427Z","avatar_url":"https://github.com/goktug97.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Parameter-exploring Policy Gradients\n=======================================================\n\nPython Implementation of Parameter-exploring Policy Gradients \u003ca href=\"#sehnke2010\"\u003e[3]\u003c/a\u003e Evolution Strategy \n\n![Bipedal](https://raw.githubusercontent.com/goktug97/PEPG-ES/master/bipedal.gif)\n\nReward: 189.16\n\n## Requirements\n* Python \u003e= 3.6\n* Numpy\n\n### Optional\n* gym\n* mpi4py\n\n## Install\n\n- From PyPI\n\n``` bash\npip3 install pepg-es\n```\n\n- From Source\n\n``` bash\ngit clone https://github.com/goktug97/PEPG-ES\ncd PEPG-ES\npython3 setup.py install --user\n```\n\n## About Implementation\n\nI implemented several things differently from the original paper;\n\n- Applied rank transformation \u003ca href=\"#wierstra14a\"\u003e[1]\u003c/a\u003e to the fitness scores.\n- Used Adam \u003ca href=\"#kingma2014adam\"\u003e[2]\u003c/a\u003e optimizer to update the mean.\n- Weight decay is applied to the mean, similar to \u003ca href=\"#salimans2017evolution\"\u003e[4]\u003c/a\u003e.\n\n## Usage\n\nRefer to [PEPG-ES/examples](https://github.com/goktug97/PEPG-ES/blob/master/examples)\nfolder for more complete examples.\n\n### XOR Example\n\n* Find Neural Network parameters for XOR Gate. \n* Black-box optimization algorithms like PEPG are competitive in the\n  area of reinforcement learning because they don't require\n  backpropagation to calculate the gradients.  In supervised learning\n  using backpropagation is faster and more reliable. Thus, using backpropagation\n  to solve the XOR problem would be faster. I demonstrated library by solving XOR\n  because it was easy and understandable.\n\n``` python\nfrom pepg import PEPG, NeuralNetwork, Adam, sigmoid\n\nimport numpy as np\n\n\nnetwork = NeuralNetwork(input_size = 2, output_size = 1, hidden_sizes = [2],\n                        hidden_activation = sigmoid,\n                        output_activation = sigmoid)\n\n# Adam Optimizer is the default optimizer, it is written for the example\noptimizer_kwargs = {'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-08} # Adam Parameters\n\nes = PEPG(population_size = 100, theta_size = network.number_of_parameters,\n          mu_init = 0, sigma_init = 2.0,\n          mu_lr = 0.3, sigma_lr = 0.2, optimizer = Adam,\n          optimizer_kwargs = optimizer_kwargs)\n\ntruth_table = [[0, 1],[1, 0]]\nsolution_found = False\n\nwhile True:\n    print(f'Step: {es.step}')\n    solutions = es.get_parameters()\n    rewards = []\n    for solution in solutions:\n        network.weights = solution\n        error = 0\n        for input_1 in range(len(truth_table)):\n            for input_2 in range(len(truth_table[0])):\n                output = int(round(network([input_1, input_2])[0]))\n                error += abs(truth_table[input_1][input_2] - output)\n        reward = (4 - error) ** 2\n        rewards.append(reward)\n    es.update(rewards)\n    if es.best_fitness == 16:\n        print('Solution Found')\n        print(f'Parameters: {es.best_theta}')\n        break\n```\n\n* Output:\n\n``` bash\nStep: 0\nStep: 1\nStep: 2\nStep: 3\nStep: 4\nStep: 5\nStep: 6\nStep: 7\nStep: 8\nStep: 9\nStep: 10\nStep: 11\nStep: 12\nStep: 13\nStep: 14\nStep: 15\nStep: 16\nStep: 17\nStep: 18\nStep: 19\nStep: 20\nSolution Found\nParameters: [ 2.69265669 -2.80113868  2.95878579 -4.21097193 -4.62368205  0.72005261\n -0.66755995 -2.50694535  0.39457738]\n```\n\n## Documentation\n\n### PEPG Class\n\n``` python\n\nes = PEPG(self, population_size, theta_size,\n          mu_init, sigma_init, mu_lr,\n          sigma_lr, l2_coeff = 0.005,\n          optimizer = Adam, optimizer_kwargs = {})\n\n```\n\n* **Parameters:**\n    - **population_size:** int: Population size of the evolution strategy.\n    - **theta_size** int: Number of parameters that will be optimized.\n    - **mu_init** float: Initial mean.\n    - **sigma_init** float: Initial sigma.\n    - **mu_lr** float: Learning rate for the mean.\n    - **sigma_lr** float: Learning rate for the sigma.\n    - **l2_coeff** float: Weight decay coefficient.\n    - **optimizer** Optimizer: Optimizer to use\n    - **optimizer_kwargs** Dict[str, Any]: Parameters for optimizer except learning rate.\n\n___\n\n``` python\nsolutions = self.get_parameters(self)\n```\n\n- Creates symmetric samples around the mean and returns a numpy array with the size of\n**[population_size, theta_size]**\n\n___\n\n``` python\nself.update(self, rewards)\n```\n\n* **Parameters:**\n    - **rewards:** List[float]: Rewards for the given solutions.\n    \n- Update the mean and the sigma.\n\n___\n\n``` python\nself.save_checkpoint(self)\n```\n\n- Creates a checkpoint and save it into created time.time().checkpoint file.\n\n___\n\n``` python\nes = PEPG.load_checkpoint(cls, filename)\n```\n\n- Creates a new PEPG class and loads the checkpoint.\n___\n\n``` python\nself.save_best(self, filename)\n```\n\n- Saves the best theta and the mu and the sigma that used to create the best theta.\n\n___\n\n``` python\ntheta, mu, sigma = PEPG.load_best(cls, filename)\n```\n\n- Load the theta, the mu, and the sigma arrays from the given file.\n\n### NeuralNetwork Class\n\n``` python\n\nNeuralNetwork(self, input_size, output_size, hidden_sizes = [],\n              hidden_activation = lambda x: x,\n              output_activation = lambda x: x,\n              bias = True):\n\n```\n\n* **Parameters:**\n    - **input_size:** int: Input size of network.\n    - **output_size:** int: Output size of the network.\n    - **hidden_sizes:** List[int]: Sizes for the hidden layers.\n    - **hidden_activation:** Callable[[float], float]: Activation function used in hidden layers.\n    - **output_activation:** Callable[[float], float]: Activation function used at the output.\n    - **bias:** bool: Add bias node.\n___\n\n``` python\nself.save_network(self, filename)\n```\n\n- Save the network to a file.\n\n___\n\n``` python\nnetwork = NeuralNetwork.load_network(cls, filename)\n```\n\n- Creates a new NeuralNetwork class and loads the given network file.\n    \n### Custom Optimizer Example\n\n``` python\nfrom pepg import PEPG, Optimizer, NeuralNetwork\n\nclass CustomOptimizer(Optimizer):\n    def __init__(self, alpha, parameter, another_parameter):\n        self.alpha = alpha\n        self.parameter = parameter\n        self.another_parameter = another_parameter\n\n    def __call__(self, gradients):\n        gradients = (gradients + self.parameter) * self.another_parameter\n        return -self.alpha * gradients\n\nnetwork = NeuralNetwork(input_size = 2, output_size = 1)\n\noptimizer_kwargs = {'parameter': 0.3, 'another_parameter': 0.2}\nes = PEPG(population_size = 100, theta_size = network.number_of_parameters,\n          mu_init = 0.0, sigma_init = 2.0,\n          mu_lr = 0.3, sigma_lr = 0.2, optimizer = CustomOptimizer,\n          optimizer_kwargs = optimizer_kwargs)\n```\n\n## References\n1. \u003ca id=\"wierstra14a\"\u003e\u003c/a\u003eDaan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters and Jurgen Schmidhuber. Natural Evolution Strategies. 2014\n2. \u003ca id=\"kingma2014adam\"\u003e\u003c/a\u003eDiederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. 2014\n3. \u003ca id=\"sehnke2010\"\u003e\u003c/a\u003eF. Sehnke, C. Osendorfer, T. Ruckstiess, A. Graves, J. Peters and J. Schmidhuber. Parameter-exploring policy gradients. 2010\n4. \u003ca id=\"salimans2017evolution\"\u003e\u003c/a\u003eTim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor and Ilya Sutskever. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. 2017\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoktug97%2Fpepg-es","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoktug97%2Fpepg-es","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoktug97%2Fpepg-es/lists"}