{"id":21629166,"url":"https://github.com/bparanj/hivegrid.dev","last_synced_at":"2026-04-10T13:31:52.192Z","repository":{"id":232085913,"uuid":"783435301","full_name":"bparanj/hivegrid.dev","owner":"bparanj","description":"HiveGrid is a infrastructure deployment tool. Install, configure and provision a EC2 instance to deploy Rails 7 apps.","archived":false,"fork":false,"pushed_at":"2024-05-20T21:08:00.000Z","size":132,"stargazers_count":25,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-21T22:39:07.970Z","etag":null,"topics":["ansible","aws-ec2","aws-s3","aws-secrets-manager","packer","terraform"],"latest_commit_sha":null,"homepage":"https://www.hivegrid.dev/","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bparanj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-04-07T21:39:53.000Z","updated_at":"2025-01-16T16:55:27.000Z","dependencies_parsed_at":"2024-04-13T20:48:58.620Z","dependency_job_id":null,"html_url":"https://github.com/bparanj/hivegrid.dev","commit_stats":null,"previous_names":["bparanj/hivegrid.dev"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bparanj/hivegrid.dev","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bparanj%2Fhivegrid.dev","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bparanj%2Fhivegrid.dev/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bparanj%2Fhivegrid.dev/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bparanj%2Fhivegrid.dev/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bparanj","download_url":"https://codeload.github.com/bparanj/hivegrid.dev/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bparanj%2Fhivegrid.dev/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31645274,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-10T07:40:12.752Z","status":"ssl_error","status_checked_at":"2026-04-10T07:40:11.664Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","aws-ec2","aws-s3","aws-secrets-manager","packer","terraform"],"created_at":"2024-11-25T02:06:53.791Z","updated_at":"2026-04-10T13:31:52.155Z","avatar_url":"https://github.com/bparanj.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HiveGrid\n\nHiveGrid is a toolchain that simplifies the deployment of Rails 7 applications on AWS by automating the provisioning and configuration of EC2 instances.\n\nRead about the [background](https://www.hivegrid.dev/about/) to learn why this project exists or the entertaining [background story](./sweat.md).\n\n## Purpose\n\nAutomate the software delivery process by using a toolchain that leverages Packer, Terraform, Ansible and cloud provider SDKs. The toolchain simplifies infrastructure management tasks by using a declarative language, pre-built templates for common use cases and glue code to integrate with cloud providers.\n\nIt targets developers at companies with limited or no DevOps resources. It reduces the time spent on manual infrastructure tasks from days to minutes. Developers can focus more on application development rather than infrastructure complexities. It is designed for those comfortable with the command-line interface, offering customizable templates for each phase of software delivery to streamline processes and enhance efficiency in the software delivery workflow.\n\n## For Whom\n\nHiveGrid is tailored for early-stage startups that are still searching for product/market fit. Scalability is not the immediate concern and modular solution allows startups to iterate quickly based on market feedback.\n\nWith this lightweight toochain, you can rapidly build, deploy and experiment with features without the burden of complex architectures. You can validate ideas, make data-driven decisions and adapt your product as you learn more about your target market.\n\nBy leveraging the modular approach, you can start small, gradually evolve and prioritize features that resonate with users. Whether you're a small team or a solo founder, HiveGrid is designed to keep your development process lean, agile and focused on rapid experimentation and learning about your target market.\n\n## Modular Toolchain\n\nHiveGrid differs from the competition by offering a modular design that prioritizes simplicity and flexibility. A set of reusable building blocks can be choosen and you can use only what you need for your specific use case. This avoids bloat and keeps your system lean.\n\nIt does not impose opinionated choices or predefined architectures. You have the freedom to select the components that fits your requirements. You can avoid over-engineering and unnecessary complexity, especially in the early stages of your startup.\n\nYou don't need to start with a complex architecture. Complex systems have steep learning curves and can be overwhelming. With the modular design, you can start with a simple architecture that meets your immediate needs and gradually evolve your architecture organically as your business grows. Beginning with a simpler architecture makes it easier to learn and adapt over time.\n\nThe building blocks are designed to be flexible and adaptable, allowing you to build a new system from scratch. The modular components can be combined and customized to create a solution tailored to your needs.\n\n```mermaid\ngraph TD\n\nsubgraph StackA[Ruby Stack A]\n    A[deploy_user] --\u003e B[install_ruby]\n    B --\u003e C[postgreSQL]\n    C --\u003e D[install_caddy]\n    \n    style StackA fill:#1f3864,stroke:#333,stroke-width:2px,color:#fff\n    style C fill:#00758F,stroke:#333,stroke-width:2px,color:#fff\nend\n```\n\n```mermaid\ngraph TD\n\nsubgraph StackB[Ruby Stack B]\n    A[deploy_user] --\u003e B[install_ruby]\n    B --\u003e C[MySQL]\n    C --\u003e D[install_caddy]\n    \n    style StackB fill:#1f3864,stroke:#333,stroke-width:2px,color:#fff\n    style C fill:#00758F,stroke:#333,stroke-width:2px,color:#fff\nend\n```\n\nChoose simplicity, flexibility and modularity and stay focused on what matters most — building a successful product.\n\n## Internal Applications\n\nMany companies build web applications for internal use, where scalability is not a concern. In such cases, an EC2 t2.medium instance is sufficient. With HiveGrid, everything can be contained within a single server instance, making it an ideal solution for this scenario.\n\nWhen building internal web applications, there is typically no need for advanced features like load balancers, zero-downtime deployments, staging environments or other complex setups. HiveGrid provides a simpler approach, allowing companies to focus on developing their application without the overhead of unnecessary infrastructure complexities.\n\nBy leveraging HiveGrid, companies can quickly deploy and manage their internal web applications on a single EC2 instance, simplifying the development and deployment process while keeping costs down.\n\n## Project Scope\n\n| Task              | Description                                                                                                 | Tools            |\n| ----------------- | ----------------------------------------------------------------------------------------------------------- | ---------------- |\n| Install           | Install the software binaries and all dependencies                                                          | Ansible, Packer  |\n| Configure         | Configure the software at runtime. TLS certs, firewall settings etc                                         | Ansible          |\n| Provision         | Provision the infrastructure. Includes servers, network configuration, IAM permissions                      | Terraform        |\n\n\nThese tools have clear separation of responsibilities, making it easier to extend and customize the functionality according to our needs.\n\n## Tooling\n\nWe've knitted together Ansible, Packer, and Terraform to whip up servers on AWS like it's nobody's business. Ansible cuts through the configuration chaos, letting us cherry-pick cloud services, slap on software, and select our web framework with a declarative flair. Packer is our Swiss Army knife, slicing across cloud platforms to provision servers without breaking a sweat. And Terraform? It’s the smart cookie that keeps our code lean and mean, thanks to its idempotent magic. Meshing these tools together, we've streamlined our AWS server setup, keeping our workflow slick, adaptable, and blazing fast.\n\n```mermaid\ngraph LR\n    A[Base Image] --\u003e B[Packer]\n    B --\u003e C[Custom Image]\n    C --\u003e D[Terraform]\n    D --\u003e E[EC2 Instance]\n    F[Code Repository] --\u003e G[Capistrano]\n    G --\u003e E\n    \n    B -- Ansible Provisioner --\u003e B\n\n    style B fill:#1f77b4,stroke:#333,stroke-width:2px,color:#fff\n    style D fill:#2ca02c,stroke:#333,stroke-width:2px,color:#fff\n    style G fill:#ff7f0e,stroke:#333,stroke-width:2px,color:#fff\n```\n\nTo learn more, read [Toolchain](https://hivegrid.dev/articles/toolchain)\n\n### Advantages of the Toolchain\n\n- Ensures consistent, reliable and reproducible environments\n- Automates processes, saving time and reducing human errors\n- Enables easy scalability, configuration, and cross-environment deployment\n- Facilitates version control, collaboration and rollbacks\n\n## Anti-Features\n\nDoes not require:\n\n- Docker\n- Load balancer\n- Cloudflare\n- Traefik\n\n## Requirements\n\n### Development - Control Node\n\n| Name      | Version     |\n| --------- | ----------- |\n| Python    | 3.12.1      |\n| Ansible   | core 2.16.1 |\n| Packer    | 1.10.1      |\n| Terraform | 1.6.6       |\n| Node      | v21.6.2     |\n| npm       | 10.5.0      |\n\n## Packer Image\n\n```mermaid\ngraph LR\n    A[Base Image] --\u003e B(Packer)\n    B --\u003e C(Ansible)\n    C --\u003e D[Custom Image]\n    \n    style B fill:#1F77B4,stroke:#333,stroke-width:2px,color:#fff\n    style C fill:#EE0000,stroke:#333,stroke-width:2px,color:#fff\n```\n\n* **Base Image:**  This is your starting point. It is a generic operating system image (e.g., Ubuntu, CentOS).\n* **Packer:** This is the tool that automates the process of image creation. You define the desired configuration of your custom image in a Packer template.\n* **Ansible:** Packer uses Ansible as a provisioner to configure and customize the base image.\n* **Custom Image:** This is the final product produced by Packer. It includes all the modifications you specified, such as:\n    * Installed software packages\n    * System configurations\n\nLarge EC2 instance is used to reduce the time taken to create the image. \n\n## Rails Stack\n\nThe custom AMI created by Packer provides:\n\n| Name         | Version                          | Release Date   |\n| ------------ | -------------------------------- | -------------- |\n| Ruby         | 3.3.1                            | April 23, 2024 |\n| RubyGem      | 3.5.10                           | May 3, 2024    |\n| Caddy        | 2.7.6                            | Dec 7, 2023    |\n| PostgreSQL   | 16.2 (Ubuntu 16.2-1.pgdg22.04+1) | Feb 8, 2024    |\n| Redis        | 7.2.4                            | Jan 9, 2024    |\n| Git          | 2.45.0                           | April 29, 2024 |\n| Goss         | 0.4.6                            | March 24, 2024 |\n\nSee [versions](./VERSIONS.md) for more details.\n\n### Customized Image\n\nIf the image created by Packer does not meet your requirements, you can:\n\n1. Customize the Packer template:\n   - Open the file `packer/aws-ubuntu.pkr.hcl`\n   - Modify the base image and other configurations as needed\n\n2. Modify the master playbook:\n   - Open the file `ansible/playbooks/master_playbook.yml`\n   - Change the included playbooks to suit your needs\n     - For example, you can create a new playbook for MySQL and replace the Postgres playbook\n\n3. Customize versions in existing playbooks:\n   - Go to the `ansible/playbooks` folder\n   - Modify the versions for Ruby, Postgresql, Redis, etc. in the corresponding playbooks\n\nBy following these steps, you can tailor the Packer-generated image to your specific requirements.\n\n## Terraform Provisioning\n\nThe custom image created in the previous step is the input to Terraform and the output is a running EC2 instance:\n\n```mermaid\ngraph LR\n    A[Custom Image] --\u003e B(Terraform)\n    B --\u003e C[EC2 Instance]\n    \n    style B fill:#1F77B4,stroke:#333,stroke-width:2px,color:#fff\n```\n\n1. The custom image, which was created using Packer and Ansible, serves as an input to Terraform.\n2. Terraform, as an infrastructure as code (IaC) tool, uses the custom image to define and provision the desired infrastructure.\n3. Terraform creates an EC2 instance based on the specifications defined in the Terraform configuration files.\n4. The EC2 instance is launched using the custom image, ensuring that it includes all the necessary software, configurations, and customizations.\n\nThis diagram illustrates the workflow where the custom image, created through the Packer and Ansible, is consumed by Terraform to provision an EC2 instance. Terraform allows you to define the desired state of your infrastructure using declarative configuration files, and it automatically provisions and manages the EC2 instance based on that configuration.\n\nBy using a custom image for the EC2 instance, the instance is pre-configured with the required software and settings, minimizing manual setup and configuration after the instance is launched.\n\nThe Terraform template defines the following configuration:\n\n- Instance Type: t2.medium\n- Region: us-west-2\n\nSee [main.tf](./terraform/main.tf) for more details. You can change it in [variables.tf](.terraform/variables.tf).\n\n## Ansible Playbooks\n\nPacker uses Ansible as the provisioner. The Ansible playbooks are included in the master playbook. Packer runs the master playbook to create a custom AMI from a base Ubuntu 22.04 image. The playbooks:\n\n- Install required packages on Ubuntu 22.04\n- Implement Intrusion Detection\n- Setup deploy user\n- Harden SSH Configuration\n- Install and Configure Caddy Server\n- Install Ruby 3.3.1\n- Install PostgreSQL 16\n- Install and Setup Redis\n- Set server timezone to UTC\n\nSee [playbooks](./PLAYBOOKS.md) for more details.\n\nYou can:\n\n- Create a new playbook and include it in the master playbook.\n- Modify existing playbook included in the master playbook.\n- Remove existing playbook from the master playbook.\n\nAnsible Galaxy is only used for database backups. This means we can install the latest version sooner than it takes for the external dependency to update to newer versions.\n\n### Intrusion Detection\n\nFrom the ansible folder, run the playbook to update the notification email for fail2ban:\n\n```bash\nansible-playbook playbooks/deploy/fail2ban-notification.yml -e \"destemail=your-email@your-domain.com\"\n```\n\nReplace the place holder `your-email@your-domain.com` with your email address. This playbook can be found here: ansible/playbooks/deploy/fail2ban-notification.yml\n\n### Parallel Processing\n\nParallel processing is enabled in Ansible for speedup. See packer/ansible.cfg for the configuration of Ansible.\n\n### Directory Structure\n\nThe ansible folder has directories for deploy and maintain. The playbooks at the ansible folder level are used for creating a custom image. Playbooks that are run for deploying Rails app are inside the deploy folder. The maintain folder is for database backup, CRON job setup or anything that is done as part of maintaining the production server.\n\nYou need to provide your IP and PEM file location in ansible/inventory.ini file to use the playbooks inside deploy or maintain folders.\n\n## Testing\n\nThe image is tested using Goss. The tests folder contains the tests. Test results are exposed as a JSON endpoint. It can be accessed only within the EC2 instance. SSH into your EC2 instance and run:\n\n```bash\ncurl http://localhost:8080/healthz | jq .\n```\n\n### Goss Test Setup\n\nFor adding tests:\n\n- Review the Packer and Terraform template\n- Manually `run goss autoadd` on the server\n- Copy the generated file on the server to tests/goss.yaml file in the project\n\n## Image and Provisioning Template Catalog\n\nEach stack has its own image and provisioning template, illustrating the separation between the different stacks.\n\n```mermaid\ngraph LR\n  %% Define style for Rails Stack\n  style RailsStack fill:#cc0000,stroke:#333,stroke-width:2px\n  subgraph RailsStack\n    RailsImage[Rails Stack Image] --\u003e RailsTemplate[Rails Provisioning Template]\n  end\n\n  %% Define style for Django Stack\n  style DjangoStack fill:#092E20,stroke:#333,stroke-width:2px\n  subgraph DjangoStack\n    DjangoImage[Django Stack Image] --\u003e DjangoTemplate[Django Provisioning Template]\n  end\n```\n\n## Benefits\n\n1. **Ease of Use**: Simplifies server setup through a few terminal commands.\n2. **Automation**: Prevents errors by automating manual tasks and making the process reproducible.\n3. **Cost-Effective**: It's free, helping early-stage companies save money.\n4. **Basic Security**: Provides basic security features for applications.\n5. **Customization**: Allows users to customize their stack by installing preferred software.\n6. **Speed**: Offers prebuilt images and tested playbooks to accelerate the process to go live.\n7. **Zero Dependency**: Does not require modifications to the application codebase or dependencies.\n8. **Technical Support**: Provides quick, personalized support within 24 hours on weekdays.\n\n## Getting Started\n\n### Prerequisites\n\n- An AWS account\n- An IAM user with appropriate policies for EC2, S3, and AWS Secrets Manager\n- Packer and Terraform installed and configured\n\nSSH Access:\n\nThe PEM file used for SSH access to your EC2 instance, is stored in AWS Secrets Manager. This means you can retrieve the PEM file even if you lose the original downloaded copy. You will be charged for storing the PEM file in Secrets Manager. To avoid the cost, you can delete the PEM file from the AWS console after downloading it.\n\nDatabase Backups:\n\nAWS S3 is used for storing database backups.\n\n### Clone the Project\n\n```bash\ngit clone git@github.com:bparanj/hivegrid.dev.git\n```\n\n### Provision the Server \n\nFrom the root of the project, run the Terraform template:\n\n```bash\nterraform apply terraform/main.tf\n```\n\n### Download PEM File\n\nYou must have nodejs installed on your laptop. To install dependencies, go to javascript folder directory and run:\n\n```\nnpm install\n```\n\nSet the following environment variables to proper values:\n\n```bash\nAWS_ACCESS_KEY_ID\nAWS_SECRET_ACCESS_KEY\nROR_SECRET_KEY\n```\n\nYou will see the value for `ROR_SECRET_KEY` from the output of the `terraform apply` command.\n\nTo download the PEM file, run:\n\n```bash\nnode keyDownload.js\n```\n\n## Lifecyle Hooks\n\n```mermaid\ntimeline\n    title Software Delivery Lifecycle\n    Base Image : Ubuntu\n    Custom Hook : Your Code\n         : Boto3\n         : Playbook\n    Custom Image : AMI\n    Custom Hook : Your Code\n         : Boto3\n         : Playbook    \n    Provision Instance : EC2 Instance\n    Custom Hook : Your Code\n         : Boto3\n         : Playbook\n    Capistrano Deploy : Your App\n```\n\nYou can customize the process in any of the following phases:\n\n1. Base Image\n2. Your Custom Hook\n3. Custom Image - Packer Build\n4. Your Custom Hook\n5. Provision Instance - Terraform Provision\n6. Your Custom Hook\n7. Capistrano Deploy\n\nYour custom hooks can be cloud-init script, ansible playbook, code written using Ruby, Python, Java or any other SDK for AWS.\n\n## Deploying Rails App\n\nYou can use Capistrano to deploy your Rails 7.1 app to the provisioned server. We will be using `dotenv` gem to manage environment variables on the production server.\n\n```mermaid\ngraph LR\n    A[Code] --\u003e B(Capistrano)\n    B -- Deploy --\u003e C[EC2 Instance]\n    \n    style B fill:#1C1B39,stroke:#333,stroke-width:2px,color:#fff\n    style C stroke:#333,stroke-width:2px\n    \n    class C server\n```\n\nThe EC2 instance, provisioned using Terraform with a custom image, serves as the target environment for the deployed application. Capistrano is used minimally,  mainly because the DSL has a learning curve. If a task can be done in Ansible, it is preferred over Capistrano.\n\nFor more details, refer the [sample Rails app](https://github.com/bparanj/capt) with working Capistrano setup.\n\n## Process at a High Level\n\n```mermaid\ngraph TD\n    A[Create IAM dev user] --\u003e B{Default Image Fits Needs?}\n    B --\u003e|Yes| C[Use Default Image]\n    B --\u003e|No| D[Create Image]\n    C --\u003e E[Provision a Server]\n    D --\u003e E\n    E --\u003e F[Setup DNS Records]\n    F --\u003e G[Run SSL playbook]\n    G --\u003e H[Run Puma playbook]\n    H --\u003e I[Change DB Password]\n    I --\u003e J[Change Fail2Ban Email]\n    J --\u003e K[Capistrano Deploy]\n```\n\nCreate Image step is optional. It is only required if the default image does not fit your needs.\n\n## Request Flow\n\n```mermaid\ngraph LR\n    A[Browser] --\u003e B[Caddy]\n    B --\u003e C[Puma]\n    C --\u003e D[PostgreSQL]\n```\n\nCaddy is configured as the reverse proxy to Puma process.\n\n## Getting Started Guide\n\n### Create a AWS Account\n\n[![AWS Account Setup](https://img.youtube.com/vi/qSq6f6fZ_bs/0.jpg)](https://www.youtube.com/watch?v=qSq6f6fZ_bs)\n\n- [Create AWS admin IAM user](https://youtu.be/rLuj3EWRV_4)\n- [Create AWS dev IAM user](https://youtu.be/HaIVVql4e8Q)\n- [Packer Build Image](https://youtu.be/oIalLv92sqg)\n- [Packer Build Image - Part 2](https://youtu.be/me4fX_IhAvw)\n- [Packer Public Image](https://youtu.be/uhQse0jWIP8)\n- [Terraform Provision](https://youtu.be/lEcJD2DWyMk)\n- [Terraform Provision Instance](https://youtu.be/F3R9e4R3I8s)\n- [Using Capistrano to Deploy Rails App](https://youtu.be/2bZhikHKTbw)\n\n## License\n\nHiveGrid is released under the [MIT License](https://opensource.org/licenses/MIT).\n\n## Alternatives\n\n- [Zero](https://github.com/commitdev/zero)\n\n## Where to Get Help\n\nJoin the [discussions](https://github.com/bparanj/hivegrid.dev/discussions) to get help.\n\n## Contributing\n\nJoin the [discussions](https://github.com/bparanj/hivegrid.dev/discussions) to start contributing to this project. Ways to contribute:\n\n- Ansible playbooks for other SQL and NoSQL database servers\n- Support other Linux flavors like RHEL, Debian etc.\n- Packer and Terraform template for other cloud providers like:\n\n| #    | Provider        | Packer Builder    |\n|------|-----------------|-------------------|\n| 1    | Microsoft Azure | Azure ARM         |\n| 2    | Google Cloud    | Google Compute    |\n| 3    | Digital Ocean   | DigitalOcean      |\n| 4    | Linode          | Linode            |\n| 5    | Rackspace       | OpenStack         |\n\n\n- Add support for other full stack MVC frameworks like: \n\n| #    | Framework        | Language     | \n|------|------------------|--------------|\n| 1    | Django           | Python       |\n| 2    | Laravel          | PHP          |\n| 3    | ASP.NET Core MVC | C#           |\n| 4    | Spring Boot      | Java         |\n| 5    | Phoenix          | Elixir       |\n| 6    | Play Framework   | Java/Scala   |\n\n\n## Endorsements\n\n\u003e Your endorsement with name can go here.\n\nTake it for a spin and if you like it, send me your endorsement.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbparanj%2Fhivegrid.dev","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbparanj%2Fhivegrid.dev","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbparanj%2Fhivegrid.dev/lists"}