{"id":16928327,"url":"https://github.com/maartengr/reinlife","last_synced_at":"2025-10-14T17:07:53.808Z","repository":{"id":40962543,"uuid":"239791231","full_name":"MaartenGr/ReinLife","owner":"MaartenGr","description":"Creating Artificial Life with Reinforcement Learning","archived":false,"fork":false,"pushed_at":"2024-05-03T19:46:50.000Z","size":65637,"stargazers_count":84,"open_issues_count":3,"forks_count":14,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-10-14T17:07:52.376Z","etag":null,"topics":["artificial-intelligence","artificial-life","deep-learning","python","reinforcement-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MaartenGr.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-11T15:08:00.000Z","updated_at":"2025-08-31T06:30:25.000Z","dependencies_parsed_at":"2024-10-26T21:18:43.641Z","dependency_job_id":"99fa5533-14f3-4bb1-8318-3576282a7a18","html_url":"https://github.com/MaartenGr/ReinLife","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/MaartenGr/ReinLife","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2FReinLife","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2FReinLife/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2FReinLife/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2FReinLife/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MaartenGr","download_url":"https://codeload.github.com/MaartenGr/ReinLife/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MaartenGr%2FReinLife/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279020085,"owners_count":26086805,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","artificial-life","deep-learning","python","reinforcement-learning"],"created_at":"2024-10-13T20:36:29.402Z","updated_at":"2025-10-14T17:07:53.777Z","avatar_url":"https://github.com/MaartenGr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![](images/logo.png)\n\n\u003e Code and instructions for creating Artificial Life in a non-traditional \nway, namely with Reinforcement Learning instead of Evolutionary Algorithms. \n\nAlthough Evolutionary Algorithms have shown to result in interesting behavior,\nthey focus on learning across generations whereas behavior could also be learned\nduring ones lifetime. This is where Reinforcement Learning comes in, which \nlearns through a reward/punishment system that allows it to learn new behavior\nduring its live time. Using Reinforcement Learning, entities learn to survive, \nreproduce, and make sure to maximize the fitness of their kin. \n\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"https://github.com/MaartenGr/ReinLife/blob/master/images/animation_medium.gif?raw=true\" height=\"366\"/\u003e\n  \u003cimg src=\"https://github.com/MaartenGr/ReinLife/blob/master/images/instruction.png?raw=true\" height=\"366\"/\u003e\n\u003c/p\u003e\n\n\n## Table of Contents  \n\u003ca name=\"toc\"/\u003e\u003c/a\u003e\n\u003c!--ts--\u003e\n   1. [About the Project](#about)\n   2. [Getting Started](#gettingstarted)  \n        2.1. [Prerequisites](#prerequisites)  \n        2.2. [Usage](#usage)   \n        2.3. [Google Colaboratory](#colab)  \n   3. [Environment](#env)  \n        3.1. [Agents](#env-agents)  \n        3.2. [Observation](#env-observation)  \n        3.3. [Reward](#env-reward)  \n        3.4. [Algorithms](#env-algorithms)  \n   4. [Results](#results)  \n   5. [Documentation](#documentation)  \n        5.1. [Training](#doc-training)  \n        5.2. [Testing](#doc-testing)  \n\u003c!--te--\u003e\n\n---\n\u003ca name=\"about\"/\u003e\u003c/a\u003e\n## 1. About the Project\n[Back to ToC](#toc)  \nThe simulation above is a good summary of what this project is about. \nEntities move and learn independently, eat, attack other entities, and reproduce. \nThis is all possible by applying Reinforcement Learning algorithms to each\nentity, such as \n\u003ca href=\"https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf\"\u003eDQN\u003c/a\u003e and \n\u003ca href=\"https://arxiv.org/pdf/1707.06347.pdf\"\u003ePPO\u003c/a\u003e. \n\nThe general principle is simple, each entity starts by randomly executing\nsome actions and will slowly learn, based on specific rewards, whether \nthose actions helped or not. The entity is punished if the action is poor and\nrewarded if it was helpful. \n\nIt is then up to the entities to find a way to survive as long as possible while \nalso making sure their kin is in good shape as possible. \n\n---\n\u003ca name=\"gettingstarted\"/\u003e\u003c/a\u003e\n##  2. Getting Started\n[Back to ToC](#toc)  \n\nTo get started, you will only need to install the requirements and \nfork/download the ReinLife package, together with the train.py and\ntest.py files. \n\n\u003ca name=\"prerequisites\"/\u003e\u003c/a\u003e\n###  2.1. Prerequisites\nTo install the requirements, simply run the following:  \n```pip install -r requirements.txt```\n\n\u003ca name=\"usage\"/\u003e\u003c/a\u003e\n###  2.2. Usage\n\nDue to the many parameters within each model and the environment itself, \nit is advised to start with train.py and test.py. These files have been \nprepared such that you can run them as is.  \n\n#### Training\nTo train one or models, simply run:\n```python\nfrom ReinLife.Models import PERD3QN\nfrom ReinLife.Helpers import trainer\n\nbrains = [PERD3QN(), \n          PERD3QN()]\n\ntrainer(brains, n_episodes=15_000, update_interval=300, width=30, height=30, max_agents=100,\n        visualize_results=True, print_results=False, static_families=False, training=True, save=True)\n``` \nThis will start training the models for 15_000 episodes. The most important variable\nhere is `static_families`. If this is set to True, then there will be at most as\nmany genes as the number of brains chosen. Thus, you will only see two colors.\nIf you set this to False, then any number of genes will be created each with \ntheir own brain. \n\n#### Testing\nTo test one or models, simply run:\n```python\nfrom ReinLife import tester\nfrom ReinLife.Models import DQN, D3QN, PERD3QN, PPO, PERDQN\n\nmain_brains = [PPO(load_model=\"pretrained/PPO/PPO/brain_gene_0.pt\"),\n               DQN(load_model=\"pretrained/DQN/DQN/brain_gene_0.pt\", training=False),\n               D3QN(load_model=\"pretrained/D3QN/D3QN/brain_gene_0.pt\", training=False),\n               PERD3QN(load_model=\"pretrained/PERD3QN/Static Families/PERD3QN/brain_gene_1.pt\", training=False),\n               PERDQN(load_model=\"pretrained/PERDQN/PERDQN/brain_gene_1.pt\", training=False)]\ntester(main_brains, width=30, height=30, max_agents=100, static_families=True, fps=10)\n``` \nThe models above are pre-trained (see results below).  \nYou can choose any number of brains that you have trained previously. Note,\nmake sure to set all models (except PPO) to training=False, otherwise it will\ndemonstrate more random behavior.\n  \n\u003ca name=\"colab\"/\u003e\u003c/a\u003e\n### 2.3. Google Colaboratory\nIt is possible to run the training code in google colaboratory if you need\nmore computing power. You start by installing pygame and cloning the repo:\n\n```jupyter\n!pip install pygame\n!git clone https://github.com/MaartenGr/ReinLife.git\n%cd ReinLife\n```\n\nAfter that, you are ready to run the training code: \n```python\nfrom ReinLife.Models import PERD3QN\nfrom ReinLife.Helpers import trainer\n\nn_episodes = 15_000\n\nbrains = [PERD3QN(train_freq=10), PERD3QN(train_freq=10)]\n\nenv = trainer(brains, n_episodes=n_episodes, update_interval=300, width=30, height=30, max_agents=100,\n        visualize_results=True, print_results=False, google_colab=True, render=False, static_families=True,\n        training=True, save=True)\n```\n\nThen, simply look at the files on the left in ReinLife/experiments/... to \nfind the experiment that was run. \n\n---\n\u003ca name=\"env\"/\u003e\u003c/a\u003e\n##  3. Environment\n[Back to TOC](#toc)\n\nThe environment is build upon a numpy matrix of size `n` * `m` where\neach grid has a pixel size of 24 by 24. Each location within the matrix \nrepresents a location which can be occupied by only a single entity.\n\n\u003ca name=\"env-agents\"/\u003e\u003c/a\u003e\n### 3.1. Agents ![](images/agents_transparent.png)\n\nAgents are entities or organisms in the simulation that can move, attack, \nreproduce, and act independently. \n\nEach agent has the following characteristics:\n* `Health`\n    * Starts at 200 and decreases with 10 each step\n    * Their health cannot exceed 200 \n* `Age`\n    * Starts at 0 and increases 1 with each step  \n    * Their maximum age is 50, after which they die\n* `Gene`\n    * Each agents is given a gene, which simply represents an integer\n    * All their offspring have the same gene value\n    * Any new agent that is created not through reproduction gets a new value\n    * This `gene` is represented by the color of the body    \n    \nAn agent can perform one of the following **eight actions**:\n* Move one space left, right, up, or down\n* Attack in the left, right, up, or down direction \n\nThe **order** of action execution is as follows:\n* Attack -\u003e Move -\u003e Eat -\u003e Reproduce\n\n#### Movement\nAn agent can occupy any un-occupied space and, from that position, can move up, \ndown, left or right. Entities cannot move diagonally. The environment has no walls, which means that if an entity moves left from the \nmost left position in the numpy matrix, then it will move to the most right \nposition. In other words, the environment is a fully-connected world.  \n\nAlthough the movement in itself is not complex, it becomes more difficult as \nmultiple entities want to move into the same spot. For that reason, each entity \nchecks whether the target coordinate is unoccupied and if no other entity wants \nto move in that space. It does this iteratively as the target coordinate changes \nif an entity cannot move.    \n\n#### Attacking ![](images/agent_attacking.png)\nAn agent can attack in one of four directions:\n* Up, Down, Left, or Right\n\nThey stand still if they attack. However, since it is the first thing they\ndo, the other agent cannot move away. When the agent successfully attacks \nanother agent, the other agent dies and the attacker increases its health. \nMoreover, if the agent successfully attacks another agent, \nits **border** becomes **red**.\n\n#### (Re)production\nEach agent learns continuously during its lifetime. The end of an episode\nis marked by the end of an agents life. \n\nWhen a new entity is **reproduced**, it inherits its brain (RL-algorithm)\nfrom its parents. \n\nWhen a new entity is **produced**, it inherits its brain (RL-algorithm) from \none of the best agents we have seen so far. A list of 10 of the best agents\nis tracked during the simulation. \n\n\u003ca name=\"env-observation\"/\u003e\u003c/a\u003e\n### 3.2. Observation\nThe field of view of each agent is a square surrounding the agent. \nSince the world is fully-connected, the agent can see \"through\" walls.\n\nThe input for the neural network can be see in the image below:   \n\n![test](images/observation.png) \n\nThere are three grids of 7x7 (example shows 5x5) that each show a specific\nobservation of the environment: \n* `Health`\n    * Shows the health of all agents within the agent's fov\n* `Kinship`\n    * Shows whether agents within the agent's fov are related to the agent\n* `Nutrition`\n    * Shows the nutritrional value of food items within the agent's fov\n\nThus, there are 3 * (7 * 7) + 6 = **153 input values**. \n\n\n\u003ca name=\"env-reward\"/\u003e\u003c/a\u003e\n###  3.3. Reward\nThe reward structure is tricky as you want to minimize the amount you steer\nthe entity towards certain behavior. For that reason, I've adopted a simple and \nstraightforward fitness measure, namely: \n\n![test](images/reward.png) \n\nWhere `r` is the reward given to agent `i` at time `t`. The `δ`\nis the Kronecker delta which is one if the the gene of agent `i`, `gi`, equals\nthe gene of agent `j`, `gj`, and zero otherwise. `n` is the total number of\nagents that are alive at time `t`. Thus, the reward essentially\nchecks how many agents are alive that share a gene with agent `i` at time\n`t` and divides by the total number of agents alive. \n\nThe result is that an agent's behavior is only steered towards making sure \nits gene lives on for as long as possible. \n\n\u003ca name=\"env-algorithms\"/\u003e\u003c/a\u003e\n### 3.4. Algorithms\nCurrently, the following algorithms are implemented that can be used as brains:\n* Deep Q Network (DQN)\n* Prioritized Experience Replay Deep Q Network (PER-DQN)\n* Double Dueling Deep Q Network (D3QN)\n* Prioritized Experience Replay Double Dueling Deep Q Network (PER-D3QN)\n* Proximal Policy Optimization (PPO)\n\n---\n\u003ca name=\"results\"/\u003e\u003c/a\u003e\n## 4. Results\n[Back to TOC](#toc)\n\nIn order to test the quality of the trained algorithms, I ran each\nalgorithm independently against a copy of itself to test the speed at which\nthey converge to a high fitness. Below, you can see all algorithms\nbattling it out with PER-D3QN coming out on top. Note, this does not mean\nit is necessarily the best algorithm. It might have converged faster than \nothers which limits their learning ability. \n\n![test](pretrained/All/results.png) \n\n\nMoreover, for each algorithm, I ran simulations with and without static families.   \n### DQN\n\u003cdetails\u003e\n\u003csummary\u003eWith static families\u003c/summary\u003e\n\u003cimg src=\"https://github.com/MaartenGr/ReinLife/blob/master/pretrained/DQN/results.png?raw=true\"/\u003e\n\u003c/details\u003e\n\n### PER-DQN\n\u003cdetails\u003e\n\u003csummary\u003eWith static families\u003c/summary\u003e\n\u003cimg src=\"https://github.com/MaartenGr/ReinLife/blob/master/pretrained/PERDQN/results.png?raw=true\"/\u003e\n\u003c/details\u003e\n\n### D3QN\n\u003cdetails\u003e\n\u003csummary\u003eWith static families\u003c/summary\u003e\n\u003cimg src=\"https://github.com/MaartenGr/ReinLife/blob/master/pretrained/D3QN/results.png?raw=true\"/\u003e\n\u003c/details\u003e\n\n### PER-D3QN\n\u003cdetails\u003e\n\u003csummary\u003eWith static families\u003c/summary\u003e\n\u003cimg src=\"https://github.com/MaartenGr/ReinLife/blob/master/pretrained/PERD3QN/Static%20Families/results.png?raw=true\"/\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eWith static families\u003c/summary\u003e\n\u003cimg src=\"https://github.com/MaartenGr/ReinLife/blob/master/pretrained/PERD3QN/No%20Static%20Families/results.png?raw=true\"/\u003e\n\u003c/details\u003e\n\n### PPO\n\u003cdetails\u003e\n\u003csummary\u003eWith static families\u003c/summary\u003e\n\u003cimg src=\"https://github.com/MaartenGr/ReinLife/blob/master/pretrained/PPO/results.png?raw=true\"/\u003e\n\u003c/details\u003e\n\n---\n\u003ca name=\"documentation\"/\u003e\u003c/a\u003e\n## 5. Documentation\n[Back to TOC](#toc)\n\n\u003ca name=\"doc-training\"/\u003e\u003c/a\u003e\n### 5.1. Training\n\nThe parameters for **train.py**:\n\n| Parameter                        | Description                              | Default value                  |\n| :------------------------------- | :--------------------------------------- | :----------------------------- |\n| brains | Contains a list of brains defined as Agents by the ReinLife.Models folder. ||\n| n_episodes | The number of epsiodes to run the training sequence. | 10_000 |\n| width, height | The width and height of the environment. | 30, 30 |\n| visualize_results | Whether to visualize the results interactively in matplotlib.  | False |\n| google_colab | If you want to visualize your results interactively in google_colab, also set this parameter to True as well as the one above.| False |\n| update_interval | The interval at which average the results | 500 |\n| print_results | Whether to print the results to the console | True |\n| max_agents | The maximum number of agents can occupy the environment. | 100 |\n| render | Whether to render the environment in pygame whilst training. | False |\n| static_families | Whether you want a set number of families to be used. Each family has its own brain defined by the models in the variable brains. | False|\n| training | Whether you want to train using the settings above or simply show the result. | True |\n| limit_reproduction | If False, agents can reproduce indefinitely. If True, all agents can only reproduce once. | False |\n| incentivize_killing | Whether to incentivize killing by adding 0.2 everytime an agent kills another | True |\n\n\u003ca name=\"doc-testing\"/\u003e\u003c/a\u003e\n### 5.2. Testing\n\nThe parameters for **test.py**:\n\n| Parameter                        | Description                              | Default value                  |\n| :------------------------------- | :--------------------------------------- | :----------------------------- |\n| brains | Contains a list of brains defined as Agents by the ReinLife.Models folder. ||\n| width, height | The width and height of the environment. | 30, 30 |\n| pastel_colors | Whether to automatically generate random pastel colors  | False |\n| max_agents | The maximum number of agents can occupy the environment. | 100 |\n| static_families | Whether you want a set number of families to be used. Each family has its own brain defined by the models in the variable brains. | False|\n| limit_reproduction | If False, agents can reproduce indefinitely. If True, all agents can only reproduce once. | False |\n| fps | Frames per second | 10 |\n\n\n### Other work\nReinLife was based on:\n* Abrantes, J. P., Abrantes, A. J., \u0026 Oliehoek, F. A. (2020). Mimicking Evolution with Reinforcement Learning. arXiv preprint arXiv:2004.00048.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaartengr%2Freinlife","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaartengr%2Freinlife","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaartengr%2Freinlife/lists"}