{"id":18359728,"url":"https://github.com/miladhzzzz/prod-automation","last_synced_at":"2026-05-17T01:50:17.803Z","repository":{"id":225353545,"uuid":"765749626","full_name":"miladhzzzz/prod-automation","owner":"miladhzzzz","description":"automation for your local development or small production !","archived":false,"fork":false,"pushed_at":"2024-04-19T16:08:21.000Z","size":388,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-15T18:51:35.407Z","etag":null,"topics":["automation","container-as-a-service","devops-tools","docker","python","rest-api","serverless-framework"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/miladhzzzz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-01T14:43:31.000Z","updated_at":"2024-03-26T17:05:26.000Z","dependencies_parsed_at":"2024-03-13T11:30:07.304Z","dependency_job_id":"c7b416c6-7040-4768-b446-a254b305ec8a","html_url":"https://github.com/miladhzzzz/prod-automation","commit_stats":null,"previous_names":["miladhzzzz/prod-automation"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miladhzzzz%2Fprod-automation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miladhzzzz%2Fprod-automation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miladhzzzz%2Fprod-automation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/miladhzzzz%2Fprod-automation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/miladhzzzz","download_url":"https://codeload.github.com/miladhzzzz/prod-automation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248150027,"owners_count":21055848,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","container-as-a-service","devops-tools","docker","python","rest-api","serverless-framework"],"created_at":"2024-11-05T22:24:16.731Z","updated_at":"2026-05-17T01:50:12.779Z","avatar_url":"https://github.com/miladhzzzz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Automagic DevOps Pipeline / Container as a Service\n\n## Introduction\n\nWelcome to the Automagic DevOps Pipeline -- your all-in-one solution for effortlessly automating the software development lifecycle! Say goodbye to manual build and deployment hassles, and hello to seamless CI/CD integration, Docker-powered containerization, and comprehensive logging and monitoring.\n\n## Description\n\nThe Automagic DevOps Pipeline is your ultimate DevOps companion, designed to simplify and streamline your development process from start to finish. Whether you're a solo developer or part of a team, this platform empowers you to automate tedious tasks, accelerate delivery, and ensure consistent, reliable deployments every time.\n\n## Architecture\n\n* **Highlevel Design of the System.**\n* feel free to open an ISSUE mentioning any problems or ideas you might have!\n\n  ![](arch/prod-auto.png)\n\n## Key Features\n\n* **GitHub Webhook Integration:** Seamlessly trigger pipeline workflows with GitHub webhook events, ensuring your CI/CD processes kick off automatically with every code push.\n  \n* **Continuous Integration (CI):** Effortlessly build and test your applications on every commit, ensuring code quality and reliability before deployment.\n  \n* **Continuous Deployment (CD):** Automatically deploy your applications to production or staging environments after successful CI runs, with minimal manual intervention.\n  \n* **Logging and Monitoring:** Gain insights into your pipeline executions and application performance with robust logging and monitoring solutions, enabling proactive issue detection and resolution.\n  \n## Get Started\n\n1. **Clone the Repository:** Get started by cloning the Automagic DevOps Pipeline repository to your local machine.\n\n    ```shell\n    git clone https://github.com/miladhzzzz/prod-automation\n    ```\n\n2. **Run Setup Script:** Execute the Makefile or init.sh script provided in the repository to automate the setup and initialization process.\n\n    ```shell\n    cd prod-automation \u0026\u0026 chmod +x init.sh\n    # This will effectively do ALL THE BELOW STEPS! plus setting up the registy configuration and cronjobs! \n    ./init.sh \u003cYOUR_GITHUB_SECRET_HERE\u003e\n    ```\n\n    **!!SKIP to step 6 if you use init.sh!!**\n\n    Or just follow along this guide to set everything up manually!(also have a better picture of what needs to be done incase anything goes wrong!)\n\n    ```shell\n    # Setting up host if its freshly installed skip this if you have docker / docker-compose / git installed!!\n    make setup\n    ```\n\n    Alternatively, manually execute the setup script:\n\n    ```shell\n    cd scripts\n    chmod +x setup-host.sh\n    ./setup-host.sh\n    ```\n\n3. **Setup Host Docker Registry:** Setup the host docker engine and /etc/hosts file to point the registry service.\n\n    ```shell\n    cd scripts \u0026\u0026 chmod +x hosts-registry.sh\n    ./hosts-registry.sh insecure_registry # this will make the /etc/docker/daemon.json file and restart docker.\n    ````\n\n4. **Deploy System Components:** Deploy the entire system, including Traefik reverse proxy, Grafana, Prometheus for observability, and the production automation container.\n\n    ```shell\n    make keygen # make keygen will generate a AES 256 CBC key for keeping your secrects safe in vault!\n    make up # make up will bring the project up using docker-compose\n    ./scripts/hosts-registry.sh update_hosts # this will update /etc/hosts file with the registry container ip for DNS resolution.\n    ```\n\n5. **Set Github Webhook Secret:** set the github secret inside the prod-auto container with the script just like below:\n\n    ```shell\n    cd scripts \u0026\u0026 chmod +x set-env.sh\n    ./set-env.sh GITHUB_WEBHOOK_SECRET \u003cYOUR_SECRET_HERE\u003e\n    ```\n\n6. **Kubernetes Integration:** Integrate [kube-o-matic](https://github.com/miladhzzzz/kube-o-matic) in your deployment environment to automate CD for kubernetes!\n\n    ```shell\n    make cd \n    ```\n\n    Extract and inject your kubeconfig file to kube-o-matic:\n\n    ```shell\n    cd scripts \u0026\u0026 chmod +x extract-kubeconfig.sh\n    ./extract-kubeconfig.sh -c \u003cCONTEXT_YOU_WANT_EXTRACTED\u003e -a \u003chttps://'prod-auto-public-ip/dns'\u003e\n    ```\n\n7. **Utilize Docker Compose:** Use a `docker-compose.yml` in the root of your repository or a Dockerfile to build your project and define services.\n\n    * Make sure you have either a docker-compose.yml as shown below or a Dockerfile present in the ROOT_DIR of your project!\n\n    ```yaml\n    # EXAMPLE YAML FILE FOR PROJECT AUTOMATION PIPELINE.\n    version: '3.9'\n\n    services:\n        example-service:\n            build:\n              context: .\n              dockerfile: Dockerfile\n\n            ports: []\n            # - \"1111:1111\"  # Adjust port if necessary\n            labels: # Provide nescesary routing for traefik to pick up and automatically route to your service!\n              - \"traefik.enable=true\"\n              - \"traefik.http.routers.example-service.rule=PathPrefix(`/api`)\"\n              - \"traefik.http.middlewares.example-service.stripprefix.prefixes=/api\"\n              - \"traefik.http.routers.example-service.middlewares=example-service@docker\"\n            volumes:\n              - \"/var/run/docker.sock:/var/run/docker.sock\"\n            # use environment to read secrets from your vault \n            environment:\n            - DATABASE_URL: $DARABASE_URL\n            networks:\n            - prod-automation_prod-auto-inet\n\n    networks:\n    prod-automation_prod-auto-inet:\n        external: true\n\n    ```\n\n8. **Set Environment Variables:** Use the API endpoint **'/vault/{project_name}'** to set environment variables for projects, ensuring smooth application execution without manual intervention.\n\n    * Send this as a json payload to the endpoint above to set your vault secrets.\n\n    ```json\n    {\n        \"DATABASE_URL\": \"postgresql://user:password@localhost/dbname\",\n        \"API_KEY\": \"abc123\", \n    }\n    ```\n\n9. **Automate Build Triggers:** Trigger builds automatically when a webhook event is received, with the necessary environment variables pre-configured for seamless deployment.\n\n   * Go to your github repository \u003e settings \u003e webhook and point the webhook to **'http(s)://\u003cYOUR-DNS/PUBLIC-IP\u003e/webhook'**.\n   * Set your webhook secret.\n   * Use application/json as the content type.\n\n10. **Track Pipeline Status:** Keep track of the pipeline status in **'/status/{project_name}'** and **'/jobs'** for monitoring and reporting purposes.\n  \n## Experience the Magic\n\n* **Push Code Changes:** Simply push your code changes to your GitHub repository.\n  \n* **Automated Workflows:** Watch as the Automagic DevOps Pipeline automates build, test, and deployment tasks seamlessly.\n  \n* **Monitor and Manage:** Keep tabs on your pipeline status and logs using the provided endpoints, ensuring smooth sailing throughout the development journey.\n\n* **Version Control:** we Keep your projects updated using Git version control so you can easily revert breaking changes with a single GET request.\n\n* **Reverse Proxy:** we also use Traefik reverse proxy you can set labels in your docker-compose to expose the services dynamically!.\n\n* **Observability** we use Prometheus / Grafana to monitor the traefik entry point we also have a /prometheus config dir you can change it to observe all your containers or other components easily!\n\n* ## TODO\n\n  * [ ] Tenanat Certificate manager(Using BYOS, LetsEncrypt)\n  * [ ] Client\n  * [ ] Test and custom pipeline implementation\n  * [ ] System state awareness\n  * [x] Consitent Data management and restructuring(for Db, projects, logs directory)\n  * [ ] Re-implementing the pipeline controller for scalablity(Celery)\n  * [ ] Notification system\n  \n## Join the Journey\n\nReady to embark on a journey of automation and efficiency? Dive into the Automagic DevOps Pipeline, contribute your ideas, and let's make DevOps a breeze together!\n\n## License\n\nThis project is licensed under the MIT License.\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiladhzzzz%2Fprod-automation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmiladhzzzz%2Fprod-automation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmiladhzzzz%2Fprod-automation/lists"}