{"id":21465806,"url":"https://github.com/jeffasante/rl-ppo-sharpshooter","last_synced_at":"2026-05-17T21:02:49.290Z","repository":{"id":254350342,"uuid":"846274850","full_name":"jeffasante/RL-PPO-SharpShooter","owner":"jeffasante","description":"Custom-built Proximal Policy Optimization (PPO) agent learns to master a 2D shooter game. Features from-scratch PPO implementation, Pygame-based environment, and OpenAI Gym integration. Showcases reinforcement learning in game AI, combining advanced algorithm development with practical game design.","archived":false,"fork":false,"pushed_at":"2024-08-29T13:04:28.000Z","size":35648,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-23T15:19:41.213Z","etag":null,"topics":["2d-shooter","ai","artificial-intelligence","custom-environment","deep-learning","game-development","machine-learning","ppo","python","reinforcement-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jeffasante.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-22T21:47:23.000Z","updated_at":"2024-09-05T22:00:37.000Z","dependencies_parsed_at":"2024-11-23T08:12:16.978Z","dependency_job_id":"8645a9a3-e4e5-4aaf-a5d1-18b3e6dbe81d","html_url":"https://github.com/jeffasante/RL-PPO-SharpShooter","commit_stats":null,"previous_names":["jeffasante/rl-ppo-shooter"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffasante%2FRL-PPO-SharpShooter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffasante%2FRL-PPO-SharpShooter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffasante%2FRL-PPO-SharpShooter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffasante%2FRL-PPO-SharpShooter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jeffasante","download_url":"https://codeload.github.com/jeffasante/RL-PPO-SharpShooter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243982138,"owners_count":20378606,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["2d-shooter","ai","artificial-intelligence","custom-environment","deep-learning","game-development","machine-learning","ppo","python","reinforcement-learning"],"created_at":"2024-11-23T08:12:09.224Z","updated_at":"2026-05-17T21:02:44.242Z","avatar_url":"https://github.com/jeffasante.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RL SharpShooter: AI-Powered 2D Space Defender\n\n## Custom PPO Implementation for 2D Shooter Game\n\nPPO-SharpShooter is a reinforcement learning project that implements a Proximal Policy Optimization (PPO) agent from scratch to master a custom-built 2D shooter game. This project showcases the power of reinforcement learning in game environments and demonstrates a deep understanding of both PPO algorithm implementation and game development.\n\n## Project Overview\n\nIn PPO-SharpShooter, a custom-built PPO agent learns to play a 2D shooter game, competing against a CPU-controlled opponent. The game environment (`ShooterEnv`) is developed using Pygame and integrated with OpenAI's Gym framework, while the PPO algorithm is implemented from the ground up using PyTorch. This project serves as a comprehensive example of applying advanced reinforcement learning techniques to game AI.\n\n## Table of Contents\n\n1. [Key Features](#key-features)\n2. [Technologies Used](#technologies-used)\n3. [Project Structure](#project-structure)\n4. [Installation](#installation)\n5. [Environment: ShooterEnv](#environment-shooterenv)\n6. [Custom PPO Implementation](#custom-ppo-implementation)\n7. [Training the Agent](#training-the-agent)\n8. [Results and Visualization](#results-and-visualization)\n9. [Usage](#usage)\n10. [Customization](#customization)\n11. [Future Improvements](#future-improvements)\n12. [Contributing](#contributing)\n13. [License](#license)\n14. [References](#references)\n\n## Key Features\n\n- **Custom PPO Implementation:** Designed from scratch, showcasing a deep understanding of the algorithm.\n- **Actor-Critic Network Architecture:** Built using PyTorch to predict actions and value functions.\n- **2D Shooter Game Environment:** Developed with Pygame and integrated with OpenAI Gym for easy interaction.\n- **Adjustable Game Speed and Frame Skipping:** Facilitates efficient training by controlling the pace of the game.\n- **Visualization of Training Progress:** Includes loss curves, episode rewards, and game play performance.\n- **Detailed Documentation:** Provides both practical implementation guides and theoretical explanations.\n\n## Technologies Used\n\n- **Python 3.x**\n- **PyTorch**\n- **Pygame**\n- **OpenAI Gym**\n- **Matplotlib**\n- **NumPy**\n\n## Project Structure\n\n```plaintext\n├── shooter_env.py         # ShooterEnv game environment\n├── ppo_agent.py           # Custom PPO implementation and Actor-Critic network\n├── train.py               # Script for training the PPO agent\n├── evaluate.py            # Script for evaluating a trained agent\n├── models/                # Directory for saved models\n├── results/               # Directory for training visualizations\n├── requirements.txt       # List of required Python packages\n└── README.md              # Project documentation\n├── shooter_game.py        # Script to run a sample game session\n\n```\n\n## Installation\n\n1. **Clone this repository:**\n   ```bash\n   git clone https://github.com/jeffasante/RL-PPO-Shooter.git\n   cd RL-PPO-Shooter\n   ```\n\n2. **Install the required dependencies:**\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n## Environment: ShooterEnv\n\n### Overview\n\n`ShooterEnv` is a custom game environment that simulates a 2D shooter game where an agent competes against a CPU-controlled opponent. It provides a standard Gym interface for reinforcement learning, making it easy to integrate with various RL algorithms.\n\n### Key Features\n\n- **State Space:** An 84x84 grayscale image of the game screen.\n- **Action Space:** `Discrete(3)` — [Move Left, Move Right, Shoot].\n- **Reward System:**\n  - +100 for defeating the CPU.\n  - -100 for losing to the CPU.\n  - Ongoing rewards based on the health difference between the player and CPU.\n- **Customizable Parameters:**\n  - `speed_multiplier`: Adjusts game speed.\n  - `skip_frames`: Number of frames to skip between actions.\n\n## Custom PPO Implementation\n\n### Overview\n\nThe PPO algorithm is implemented from scratch, showcasing a deep understanding of policy gradient methods and the specific improvements introduced by PPO.\n\n### Key Components\n\n```python\nclass ActorCritic(nn.Module):\n    def __init__(self, state_dim, action_dim):\n        # Shared convolutional layers\n        # Separate actor and critic heads\n\nclass PPO:\n    def __init__(self, state_dim, action_dim, lr, gamma, epsilon, value_coef, entropy_coef):\n        # Initialize PPO parameters\n\n    def get_action(self, state):\n        # Action selection logic\n\n    def update(self, states, actions, old_log_probs, rewards, dones):\n        # PPO update algorithm implementation\n        # Includes clipped objective and value function loss\n```\n\n### Important Aspects\n\n- **Advantage Estimation and Normalization:** Improves training stability.\n- **PPO Clipping:** Prevents large updates, ensuring stable policy updates.\n- **Separate Value Function and Entropy Loss Components:** For more efficient learning.\n- **Customizable Hyperparameters:** Fine-tune the learning process to achieve better results.\n\n## Training the Agent\n\nThe training process involves:\n\n1. **Interacting with the `ShooterEnv`:** The agent observes the environment and takes actions.\n2. **Collecting Experiences:** States, actions, rewards, and other data are collected for training.\n3. **Updating the PPO Model:** The model is updated periodically using the collected experiences.\n4. **Visualizing the Training Progress:** Losses and rewards are plotted to monitor the training.\n\n```python\ndef train(env, ppo_agent, num_episodes, save_freq=100):\n    for episode in range(num_episodes):\n        # Collect episode data\n        # Update PPO agent\n        # Save model and plot losses periodically\n```\n\n## Results and Visualization\n\nThe training process generates two main visualizations:\n\n1. `ppo_losses.png`: Actor loss, Critic loss, and Entropy loss over time.\n2. `episode_rewards.png`: Rewards obtained in each episode during training.\n\nThese visualizations provide insights into the agent's learning progress and performance.\n\n## Usage\n\n### Training the Agent\n\n```python\nfrom shooter_env import ShooterEnv\nfrom ppo_agent import PPO\n\nenv = ShooterEnv(speed_multiplier=2, skip_frames=2)\nstate_dim = (1, 84, 84)\naction_dim = env.action_space.n\n\nppo_agent = PPO(state_dim, action_dim, lr=3e-4, gamma=0.99, epsilon=0.2, value_coef=0.5, entropy_coef=0.01)\ntrain(env, ppo_agent, num_episodes=1000, save_freq=100)\n```\n\n### Evaluating a Trained Agent\n\n```python\nenv = ShooterEnv(speed_multiplier=1, skip_frames=1)\nppo_agent = PPO(state_dim, action_dim, lr=3e-4, gamma=0.99, epsilon=0.2, value_coef=0.5, entropy_coef=0.01)\nppo_agent.load_model('models/ppo_model_final.pth')\n\nstate = env.reset()\ndone = False\nwhile not done:\n    action, _ = ppo_agent.get_action(state)\n    state, reward, done, _ = env.step(action)\n    env.render()\n```\n\n **Run inference:**\n   ```bash\n   python evaluate.py \n   ```\n\n## Customization\n\n- **Modify `ShooterEnv` Parameters:** Customize the game difficulty by adjusting player and CPU health, bullet speed, etc.\n- **Adjust the Actor-Critic Network Architecture:** Experiment with different network structures in the `ActorCritic` class.\n- **Experiment with PPO Hyperparameters:** Fine-tune learning rates, clipping values, and other parameters for optimized training.\n\n## Future Improvements\n\n- **Implement Parallel Environments:** Speed up training by using multiple environments simultaneously.\n- **Explore Different Network Architectures:** Test LSTMs or other architectures to capture temporal dependencies.\n- **Support for Continuous Action Spaces:** Expand the environment to handle continuous action spaces.\n- **Compare with Other RL Algorithms:** Implement A2C, SAC, or other algorithms to benchmark performance.\n- **Enhance the CPU Opponent's AI:** Make the CPU a more formidable opponent by improving its decision-making capabilities.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request or open an issue if you encounter any problems.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## References\n\n- Schulman, J., Wolski, F., Dhariwal, P., Radford, A., \u0026 Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347.\n- Pygame Documentation: [Pygame Docs](https://www.pygame.org/docs/)\n- PyTorch Documentation: [PyTorch Docs](https://pytorch.org/docs/)\n- OpenAI Gym: [OpenAI Gym](https://gym.openai.com/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeffasante%2Frl-ppo-sharpshooter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjeffasante%2Frl-ppo-sharpshooter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeffasante%2Frl-ppo-sharpshooter/lists"}