{"id":16143186,"url":"https://github.com/johnbedeir/end-to-end-devops-aws-python-mysql","last_synced_at":"2025-04-06T19:25:09.828Z","repository":{"id":211558225,"uuid":"728476199","full_name":"johnbedeir/End-to-End-DevOps-AWS-Python-MySQL","owner":"johnbedeir","description":"This is End to end devops project using Python and MySQL database","archived":false,"fork":false,"pushed_at":"2024-05-26T17:04:47.000Z","size":1310,"stargazers_count":7,"open_issues_count":1,"forks_count":7,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-13T01:29:37.388Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/johnbedeir.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-07T02:44:34.000Z","updated_at":"2024-12-25T00:05:52.000Z","dependencies_parsed_at":"2023-12-09T11:20:27.649Z","dependency_job_id":"53d361df-1448-4320-8255-caa6c4e84383","html_url":"https://github.com/johnbedeir/End-to-End-DevOps-AWS-Python-MySQL","commit_stats":null,"previous_names":["johnbedeir/end-to-end-devops-python-mysql","johnbedeir/end-to-end-devops-aws-python-mysql"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbedeir%2FEnd-to-End-DevOps-AWS-Python-MySQL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbedeir%2FEnd-to-End-DevOps-AWS-Python-MySQL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbedeir%2FEnd-to-End-DevOps-AWS-Python-MySQL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnbedeir%2FEnd-to-End-DevOps-AWS-Python-MySQL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/johnbedeir","download_url":"https://codeload.github.com/johnbedeir/End-to-End-DevOps-AWS-Python-MySQL/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247537114,"owners_count":20954887,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-10T00:08:49.648Z","updated_at":"2025-04-06T19:25:09.803Z","avatar_url":"https://github.com/johnbedeir.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# End-to-end DevOps | Python - MySQL App\n\n\u003cimg src=imgs/cover.png\u003e\n\nThis is a simple Python Flask application that performs CRUD operations on a MySQL database, the project contains scripts and Kubernetes manifests for deploying the Python application with Mysql Database on an AWS EKS cluster and RDS with an accompanying ECR repositories. The deployment includes setting up monitoring with Prometheus and Grafana, and a CI/CD pipeline.\n\n## Prerequisites\n\nBefore you begin, ensure you have met the following requirements:\n\n- [Python](https://www.python.org/downloads/)\n- [MySQL For Ubuntu](https://dev.mysql.com/downloads/repo/apt/) | [MySQL For Windows](https://dev.mysql.com/downloads/installer/)\n- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) configured with appropriate permissions\n- [Docker](https://docs.docker.com/engine/install/) installed and configured\n- [kubectl](https://kubernetes.io/docs/tasks/tools/) installed and configured to interact with your Kubernetes cluster\n- [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)\n- [Helm](https://helm.sh/docs/intro/install/)\n- [GitHub_CLI](https://github.com/cli/cli)\n- [K9s](https://k9scli.io/topics/install/)\n- [Beekeeper-Studio](https://www.beekeeperstudio.io/) `For Database Access`\n\n## 1. Running the Application Locally\n\n### Setting Up the Database\n\n1. Start your MySQL server.\n2. Create a new MySQL database for the application.\n3. Update the database configuration in `app.py` to match your local MySQL settings:\n\n   - DB_HOST: localhost\n   - DB_USER: your MySQL username\n   - DB_PASSWORD: your MySQL password\n   - DB_DATABASE: your database name\n\n4. Create a table in the database that will be used by your application\n   ```\n   CREATE TABLE tasks (\n   id SERIAL PRIMARY KEY,\n   title VARCHAR(255) NOT NULL,\n   description TEXT,\n   is_complete BOOLEAN DEFAULT false\n   );\n   ```\n\n### Running the App\n\n1. Clone the repository:\n\n   ```\n   git clone https://github.com/johnbedeir/End-to-end-DevOps-Python-MySQL\n   cd End-to-end-DevOps-Python-MySQL/todo-app\n   ```\n\n2. Create a virtual environment and activate it:\n\n   ```\n   python3 -m venv venv\n   source venv/bin/activate  # On Windows use `venv\\Scripts\\activate`\n   ```\n\n3. Install the required dependencies:\n\n   ```\n   pip3 install -r requirements.txt\n   ```\n\n4. Start the Flask application:\n\n   ```\n   python3 app.py\n   ```\n\n5. Access the application at `http://localhost:5000`.\n\n## 2. Running with Docker (Optional)\n\n1. Build the Docker image:\n\n   ```\n   docker build -t my-flask-app .\n   ```\n\n2. Run the Docker container with host network (to access the local MySQL server):\n\n   ```\n   docker run --network=host my-flask-app\n   ```\n\n3. Access the application at `http://localhost:5000`.\n\n## 3. Running App and Database with Docker compose (Optional)\n\nTo run the application using docker compose:\n\n```\ndocker-compose up\n```\n\nThis will Run both the application and the database containers and will also create a table in the database using the sql script `init-db.sql`\n\nTo take it down run the following command:\n\n```\ndocker-compose down\n```\n\n## 4. Build, Deploy and Run the application on AWS EKS and RDS\n\nTo build and deploy the application on AWS EKS and RDS execute the following script:\n\n```\n./build.sh\n```\n\nThis will build the infrastructure, Deploy Monitoring Tools, and run some commands:\n\n1.  EKS (Kubernetes Cluster)\n2.  2x ECR (Elastic Container Registry) `One for the App Image and one for the DB K8s Job`\n3.  RDS (Relational Database Service) `RDS Cluster with One Instance`\n4.  Generate and store RDS credentials into AWS Secret Manager\n5.  VPC, Subnets and Network Configuration\n6.  Monitoring Tools Deployment (Alert Manager, Prometheus, Grafana)\n7.  Build and Push the Dockerfile for the Application and MySQL Kubernetes Job to the ECR\n8.  Create Kubernetes Secrets with the RDS Credentials\n9.  Create Namespace and Deploy the application and the Job\n10. Reveal the LoadBalancer URL for the application, alertmanager, prometheus and grafana\n\n`IMPORTANT: Make sure to update the Variables in the script`\n\n## 5. Access the application\n\nOnce the `build.sh` script is executed, you should see the URLs for all the deployed applications:\n\n**NOTE: For the `alert-manager` use the port `9093` after the URL, and for `prometheus` use port `9090`.**\n\n\u003cimg src=imgs/loadbalancer.png\u003e\n\nThe application should look like this:\n\n\u003cimg src=imgs/app-1.png\u003e\n\nAnd once you add data should look like this:\n\n\u003cimg src=imgs/app-2.png\u003e\n\nYou can `Complete` or `Delete` the task and this will take effect automatically in the database.\n\n## 6. Access the database\n\n### Option 1: Terminal\n\nYou will also get the database endpoint URL:\n\n\u003cimg src=imgs/rds.png\u003e\n\nUse that URL and access the database using the following comand:\n\n```\nmysql -h RDS_ENDPOINT_URL -P 3306 -u root -p\n```\n\n**NOTE: Once you run the command you will be asked for the database password, you have 2 methods to get the database password:**\n\n#### Method 1:\n\nOpen `K9s` through the terminal, `Ctrl+A` to navicate to the main screen `/secrets` to search for the k8s secrets and inside the secrets `/rds` to search for names that include `rds`, you should get the following:\n\n\u003cimg src=imgs/secrets-1.png\u003e\n\nOver `rds-password` tab `x` so you can decode the encrypted password\n\n\u003cimg src=imgs/secrets-2.png\u003e\n\n#### Method 2:\n\nThrough AWS, navigate to `Secret Manager` on `AWS`\n\n\u003cimg src=imgs/aws-1.png\u003e\n\nClick on the created secret `rds-cluster-secret` and from the Overview tab click on `Retrieve secret value`\n\n\u003cimg src=imgs/aws-2.png\u003e\n\nThis will show you the username and the generated password for the database.\n\n\u003cimg src=imgs/aws-3.png\u003e\n\nOnce you connected to the database throug the terminal you can run the following commands to check the data into the database:\n\n```\nshow databases;\nuse DATABASE_NAME;\nshow tables;\nselect * from TABLE_NAME;\n```\n\n\u003cimg src=imgs/db-terminal.png width=500\u003e\n\n### Option 2: Beekeeper Studio\n\nChoose the database `MYSQL`, fill in the `Host`, `User`, `Password`, make sue the port is `3306` then connect.\n\n\u003cimg src=imgs/db-connect.png\u003e\n\n\u003cimg src=imgs/db-connect-2.png\u003e\n\n\u003cimg src=imgs/db-connect-3.png\u003e\n\n## 7. CI/CD Workflows\n\nThis project is equipped with GitHub Actions workflows to automate the Continuous Integration (CI) and Continuous Deployment (CD) processes.\n\n### Continuous Integration Workflow\n\nThe CI workflow is triggered on pushes to the `main` branch. It performs the following tasks:\n\n- Checks out the code from the repository.\n- Configures AWS credentials using secrets stored in the GitHub repository.\n- Logs in to Amazon ECR.\n- Builds the Docker image for the Python app.\n- Builds the Docker image for MySQL Kubernetes job.\n- Tags the images and pushes each one to the it's Amazon ECR repository.\n\n### Continuous Deployment Workflow\n\nThe CD workflow is triggered upon the successful completion of the CI workflow. It performs the following tasks:\n\n- Checks out the code from the repository.\n- Configures AWS credentials using secrets stored in the GitHub repository.\n- Sets up `kubectl` with the required Kubernetes version.\n- Deploys the Kubernetes manifests found in the `k8s` directory to the EKS cluster.\n\n### GitHub Actions Secrets:\n\nThe following secrets need to be set in your GitHub repository for the workflows to function correctly:\n\n- `AWS_ACCESS_KEY_ID`: Your AWS Access Key ID.\n- `AWS_SECRET_ACCESS_KEY`: Your AWS Secret Access Key.\n- `KUBECONFIG_SECRET`: Your Kubernetes config file encoded in base64.\n\n#### 1. Setting Up GitHub Secrets for AWS\n\nBefore using the GitHub Actions workflows, you need to set up the AWS credentials as secrets in your GitHub repository. The included `github_secrets.sh` script automates the process of adding your AWS credentials to GitHub Secrets, which are then used by the workflows. To use this script:\n\n1. Ensure you have the GitHub CLI (`gh`) installed and authenticated.\n2. Run the script with the following command:\n\n   ```bash\n   ./github_secrets.sh\n   ```\n\nThis script will:\n\n- Extract your AWS Access Key ID and Secret Access Key from your local AWS configuration.\n- Use the GitHub CLI to set these as secrets in your GitHub repository.\n\n**Note**: It's crucial to handle AWS credentials securely. The provided script is for demonstration purposes, and in a production environment, you should use a secure method to inject these credentials into your CI/CD pipeline.\n\nThese secrets are consumed by the GitHub Actions workflows to access your AWS resources and manage your Kubernetes cluster.\n\n#### 2. Adding KUBECONFIG to GitHub Secrets\n\nFor the Continuous Deployment workflow to function properly, it requires access to your Kubernetes cluster. This access is granted through the `KUBECONFIG` file. You need to add this file manually to your GitHub repository's secrets to ensure secure and proper deployment.\n\nTo add your `KUBECONFIG` to GitHub Secrets, follow these steps:\n\n1. Encode your `KUBECONFIG` file to a base64 string:\n\n   ```bash\n   cat ~/.kube/config | base64\n   ```\n\n2. Copy the encoded output to your clipboard.\n\n3. Navigate to your GitHub repository on the web.\n\n4. Go to `Settings` \u003e `Secrets` \u003e `New repository secret`.\n\n5. Name the secret `KUBECONFIG_SECRET`.\n\n6. Paste the base64-encoded `KUBECONFIG` data into the secret's value field.\n\n7. Click `Add secret` to save the new secret.\n\nThis `KUBECONFIG_SECRET` is then used by the CD workflow to authenticate with your Kubernetes cluster and apply the required configurations.\n\n**Important**: Be cautious with your `KUBECONFIG` data as it provides administrative access to your Kubernetes cluster. Only store it in secure locations, and never expose it in logs or to unauthorized users.\n\n## 8. Destroying the Infrastructure\n\nIn case you need to tear down the infrastructure and services that you have deployed, a script named `destroy.sh` is provided in the repository. This script will:\n\n- Log in to Amazon ECR.\n- Delete the specified Docker image from the ECR repository.\n- Delete the Kubernetes deployment and associated resources.\n- Delete the Kubernetes namespace.\n- Destroy the AWS resources created by Terraform.\n\n### Before you run\n\n1. Open the `destroy.sh` script.\n2. Ensure that the variables at the top of the script match your AWS and Kubernetes settings:\n\n   ```bash\n   $1=\"ECR_REPOSITORY_NAME\"\n   $2=\"REGION\"\n   ```\n\n### How to Run the Destroy Script\n\n1. Save the script and make it executable:\n\n   ```bash\n   chmod +x destroy.sh\n   ```\n\n2. Run the script:\n\n   ```bash\n   ./destroy.sh\n   ```\n\nThis script will execute another script `ecr-img-delete.sh` which will delete all the images on the two `ECR` to make sure the both `ECR` are empty then `terraform` commands to destroy all resources related to your deployment.\n\nOnce the `terraform destroy` starts the `RDS` will start creating a `Snapshot` as a backup for the database, in that case the process of destroying will fail at some point.\n\nThe script will delete the created `Snapshot` then run `terraform destroy` again to make sure all resources are deleted.\n\n```\naws rds delete-db-cluster-snapshot --db-cluster-snapshot-identifier $rds_snapshot_name --region $region\n```\n\nIt is essential to verify that the script has completed successfully to ensure that all resources have been cleaned up and no unexpected costs are incurred.\n\n```\nMake sure to replace URLs, database configuration details, and any other specific instructions to fit your project. This README provides a basic guideline for users to set up and run your application both locally, with Docker, with Docker Compose, minikube and also over EKS and RDS.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnbedeir%2Fend-to-end-devops-aws-python-mysql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjohnbedeir%2Fend-to-end-devops-aws-python-mysql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnbedeir%2Fend-to-end-devops-aws-python-mysql/lists"}