{"id":38654732,"url":"https://github.com/ibrahimroshdy/refresher","last_synced_at":"2026-01-17T09:25:48.033Z","repository":{"id":189472416,"uuid":"496631610","full_name":"ibrahimroshdy/refresher","owner":"ibrahimroshdy","description":"Refresher project is a complete infrastructure that embodies different technologies to serve a series of tasks. The refresher project aims to create a complete guide in every aspect of those technologies to serve as a guide or a template for jumpstarting similar projects.","archived":false,"fork":false,"pushed_at":"2025-03-22T21:23:02.000Z","size":10415,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-22T22:27:18.671Z","etag":null,"topics":["celery","django","docker","github-actions","grafana","nginx","prometheus","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ibrahimroshdy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-26T13:29:23.000Z","updated_at":"2025-03-22T21:23:06.000Z","dependencies_parsed_at":"2023-08-20T09:43:50.755Z","dependency_job_id":"f785184f-d0ff-4b87-ade7-fdd38d379c57","html_url":"https://github.com/ibrahimroshdy/refresher","commit_stats":null,"previous_names":["ibrahimroshdy/refresher"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ibrahimroshdy/refresher","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibrahimroshdy%2Frefresher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibrahimroshdy%2Frefresher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibrahimroshdy%2Frefresher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibrahimroshdy%2Frefresher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ibrahimroshdy","download_url":"https://codeload.github.com/ibrahimroshdy/refresher/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibrahimroshdy%2Frefresher/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28505550,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T06:57:29.758Z","status":"ssl_error","status_checked_at":"2026-01-17T06:56:03.931Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","django","docker","github-actions","grafana","nginx","prometheus","python"],"created_at":"2026-01-17T09:25:47.878Z","updated_at":"2026-01-17T09:25:47.992Z","avatar_url":"https://github.com/ibrahimroshdy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Refresher\n[![Python](https://upload.wikimedia.org/wikipedia/commons/f/fc/Blue_Python_3.7_Shield_Badge.svg)]()\n[![Docker](https://img.shields.io/badge/docker-v20.10.12-blue)]()\n[![Docker Compose](https://img.shields.io/badge/docker--compose-v1.25.0-blue)]()\n[![Refresher CI](https://github.com/ibrahimroshdy/refresher/actions/workflows/docker-image.yml/badge.svg?branch=main)](https://github.com/ibrahimroshdy/refresher/actions/workflows/docker-image.yml)\n[![CircleCI](https://dl.circleci.com/status-badge/img/gh/ibrahimroshdy/refresher/tree/main.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/ibrahimroshdy/refresher/tree/main)\n[![codecov](https://codecov.io/gh/ibrahimroshdy/refresher/branch/main/graph/badge.svg?token=G77HXOM8GX)](https://codecov.io/gh/ibrahimroshdy/refresher)\n\n\n## Introduction\nYou probably have tons of ideas laying around in some notebook about a specific project you want to start implementing. \nYou probably also google the same set of instructions over and over. You always forget some of the syntax in a `Dockerfile`.\n\nInstead, why don't you have a complete project that grows as your experience grows and works as your own technical refresher. \nThe Refresher project aims to refresh technical skill and create an environment that is ready for experimenting ideas and expansion. \nEach version of this project should project a design scheme. \nThis is refresher v1.\n\n\u003csummary\u003e\u003ch4 style=\"display: inline-block\"\u003eTable of Contents\u003c/h4\u003e\u003c/summary\u003e\n    \u003col\u003e\n        \u003cli\u003e\n            Refresher Project\n              \u003cul\u003e\n                \u003cli\u003e\u003ca href=\"#overview\"\u003eOverview\u003c/a\u003e\u003c/li\u003e\n                \u003cli\u003e\u003ca href=\"#refresher-use-case\"\u003eRefresher Use case\u003c/a\u003e\u003c/li\u003e\n              \u003c/ul\u003e\n        \u003c/li\u003e\n        \u003cli\u003e\n            Technology\n              \u003cul\u003e\n                \u003cli\u003e\u003ca href=\"#tech-stack\"\u003eTech Stack\u003c/a\u003e\u003c/li\u003e\n                \u003cli\u003e\u003ca href=\"#architecture\"\u003eArchitecture\u003c/a\u003e\u003c/li\u003e\n                \u003cli\u003e\u003ca href=\"#application-level-road-trip\"\u003eApplication Level Road trip\u003c/a\u003e\u003c/li\u003e\n              \u003c/ul\u003e\n        \u003c/li\u003e\n        \u003cli\u003e\n            Code\n              \u003cul\u003e\n                \u003cli\u003e\u003ca href=\"#folder-structure\"\u003eFolder Structure\u003c/a\u003e\u003c/li\u003e\n                \u003cli\u003e\u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e\u003c/li\u003e\n              \u003c/ul\u003e\n        \u003c/li\u003e\n\u003c/ol\u003e\n\n\n\n---\n\n\u003cdetails open=\"open\"\u003e\n  \u003csummary\u003e\u003ch2 style=\"display: inline-block\"\u003eRefresher Project\u003c/h2\u003e\u003c/summary\u003e\n\n### Overview\nRefresher project is a complete infrastructure that embodies different technologies to serve a series of tasks. \nThe refresher project aims to create a complete guide in every aspect of those technologies\nto serve as a guide or a template for jumpstarting similar projects. \nThere is basically no definitive aim for this project, rather than having an educational pipeline that is so complex yet so simple to \nunderstand in introduction how to develop, stage, release and maintain projects.\n\n#### Refresher Use Case\nImagine you need a cronjob (scheduled) based application that executes specific tasks at \nspecific intervals of your choice **and** stores information regarding these tasks (output, logs…etc.). Moreover, you need to monitor these \ntasks in terms of successes and failures. You also need to monitor your system usage, database usage and your docker services (containers) health. \nFinally, you need to visualize all of these, rather than looking at tens of scrambled data in tables, queries on your console. \nAfter all that, you need to serve your application(s) publicly and securely.\n\nThe refresher project is a Django backend using PostgresSQL database with asynchronous task scheduler - celery -  using a Redis broker. \nThere is one Django app in the refresher project called **speedtester** app. \nThe speedtester app mainly runs internet speedtests every specific interval **using** celery workers \nand stores the `download`, `upload`  speeds and `url` of each test to the PostgresSQL using Django's ORM.\nUsing celery beat database scheduler, the database now holds information about celery task and defined interval. \nThis means you can instruct a specific task written in your backend to run on an interval simply through your admin portal. \n\nA user adds a periodic task using the **Django** admin portal, this automatically adds the periodic task \nand its defined interval in the designated data tables in the database. \nCelery — Asynchronous Task Scheduler — has two major processes running (celery **worker** and celery **beat**).\nUsing `django-celery-beat` application, there is an established connection between **Django** and **Celery**, where the \n**Celery Beat** scheduler process is now looking for tasks that are stored in the database. In other words, Django sets \ncelery beat to look for scheduled tasks stored in the designated data tables mentioned above. Celery beat then collects those tasks \nand adds them to a broker (**Redis**) — fancy word for an inmemory db. Those tasks in Redis are waiting to be picked up \nby a **Celery Worker** — the responsible process for executing task units. Finally, the task itself — the speedtester unit in this case — \nis executed where it uses `RefresherSpeedtest()` class to perform an internet speed test and \nstores data into a **PostgresSQL** database via **Django's** ORM as a db entry. \nNow the worker marks the task as completed and picks up the next task in line. \n\n![](images/architecture/refresher-speedtester.png)\n\nThe refresher project now has some basic ideas to build an infrastructure around it.\n1. A fully functioning **Django** backend\n2. **Asynchronous Task Scheduler** (basically well managed cronjobs)\n3. A **PostgresSQL** database\n\nThe speedtester collects `download`, `upload`  speeds and `url` of each conducted speedtest\n`apps.speedtester.tasks.process_speedtest`. In doing so it also retains a list of the pinged servers' details per \nspeedtest for example (country, country code, coordinates).\nWith all of that saved in a database, it now **needs** — can —  to visualize our data. \n\n![](images/screenshots/refresher-grafana-speedtest-dashboard.png)\n\nAbove, is a [Grafana](https://grafana.com/) dashboard that visuals the data tables of the `speedtester` application.\nThis can easily be done by connecting Grafana to the database.\n\n![](images/architecture/refresher-speedtester-grafana-db.png)\n\n\u003c/details\u003e\n\n---\n\n\u003cdetails open=\"open\"\u003e\n  \u003csummary\u003e\u003ch2 style=\"display: inline-block\"\u003eTechnology \u003c/h2\u003e\u003c/summary\u003e\n\n### Tech Stack \nIn this section, the refresher project's tech stack is visualized and explained. \nThe tech stack is a chart with all the components and tools used to build this project and\nnot just technical tools but also project management tools, Continuous Integration (CI), code coverage as well. \n\n![](images/architecture/refresher-techstack.png)\n\n### Architecture\nThe refresher project V1 runs on a single host machine (virtual machine) to simplify the architecture visualization. \nRunning on Azure Compute Virtual Machine and using Docker to run containers. \nIn V2 and onwards, refresher project will change its architecture as the project grows.\n`Refresher V1` is a monolithical single host project using dockerized containers sharing the same network to serve different purposes. \n\n\u003eThere are so many wrongs here but for a jumpstart option this **can** be considered.\n\n![](images/architecture/refresher-architecture.png)\n\n\n\u003e Well, let's break this down. The refresher project has single subdomain attached to it which is `speedtester.withnoedge.tech`\n\nThe [SWAG](https://www.linuxserver.io/blog/2020-08-21-introducing-swag) - Secure Web Application Gateway - \nis a tool that encapsulates tools that are essential for running a secure web application, as it says \n'sets up an Nginx web server and reverse proxy with PHP support and a \nbuilt-in certbot client that automates free SSL server certificate generation and renewal processes'\nSWAG has:\n1. [Certbot](https://certbot.eff.org/)\n2. [NGINX](https://www.nginx.com/)\n\n![](https://www.linuxserver.io/user/pages/content/images/2019/04/reverseproxy.png)\n\nBasically, there is the main domain name `withnoedge.tech` with a subdomain `speedtester` attached to it, \nso it becomes `speedtester.withnoedge.tech` that *subdomain* now \nlinks a [*A DNS record*](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) to an [Azure Virtual Machine](https://azure.microsoft.com/en-us/services/virtual-machines/#overview)'s *Public IPV4* address. After that, every time you *visit* `speedtester.withnoedge.tech` you will be routed to the Virtual Machine.\n\u003e Did you notice the word visit was in italics?\n\nWell, yes. Visiting a domain name (a virtual machine) here needs to identified – or to make this technically correct it needs to be **identified** then **routed**.\n\n### Application Level Road trip\nFirst, you need to identify what are the applications you will need to access from your server\n(your virtual machine is a server now!), and [applications](https://www.imperva.com/learn/application-security/osi-model/) \nhere are meant to describe the generic usage of tools (HTTP, ssh, telnet…etc.) – *layer 7 in OSI* – for example most of\nusage in this project will HTTP(s) requests via a Web Browser. We will also need SSH to access the machine's console securely.\n\nSecondly, you need to route your applications to the ports you \nwill be mainly using, for example, by default `HTTPS runs on port 443` and `ssh run on port 22`. \nSo simply speaking, having a machine in a local network with `port 22` exposed, you can now use your terminal to\nconnect to that machine's console and if you have a web server running on `port 9000`, \nyou can simply write the local machine's IP in your browser with `port 9000` and **it works**!\n\nComplexly speaking, we are doing the same thing but on a larger scale.\n\nSo meanwhile, *SWAG* takes care of distributing, securing and routing of our three main applications. \n\n1. [Django](https://www.djangoproject.com/) \n2. [Grafana](https://grafana.com/)\n3. [Flower](https://flower.readthedocs.io/en/latest/)\n\n\u003c/details\u003e\n\n---\n\n\u003cdetails open=\"open\"\u003e\n  \u003csummary\u003e\u003ch2 style=\"display: inline-block\"\u003eCode\u003c/h2\u003e\u003c/summary\u003e\n\n\n\n### Folder Structure\n\n```\n           __               _               \n _ __ ___ / _|_ __ ___  ___| |__   ___ _ __ \n| '__/ _ \\ |_| '__/ _ \\/ __| '_ \\ / _ \\ '__|\n| | |  __/  _| | |  __/\\__ \\ | | |  __/ |   \n|_|  \\___|_| |_|  \\___||___/_| |_|\\___|_|   \n\n         ├── README.md # Readme Markdown \n         ├── apps # Django apps\n         ├── docker # docker folder (Dockerfile, docker-compose, grafana and prometheus setup)\n         ├── images # screenshots\n         ├── manage.py \n         ├── nginx # nginx conf files\n         ├── pyproject.toml # python packages list using poetry \n         ├── refresher_config # django main configuration app\n         ├── requirements.txt # python packages requirements list \n         ├── scripts # shell scripts \n         └── static # static files for django project\n\n```\n\n## Getting Started\n\n### 1. CI/CD Pipeline\nThe project uses two CI/CD platforms:\n\n#### 1.1 CircleCI\n- Runs tests in a native Python environment\n- Uses Poetry for dependency management\n- Executes Django migrations and tests\n- Configured in `.circleci/config.yml`\n\n#### 1.2 Github Actions\nThe project uses two main workflows:\n\n##### 1.2.1 Docker Image Workflow (`docker-image.yml`)\n- Builds and pushes Docker images to GitHub Container Registry (ghcr.io)\n- Uses custom actions for build/push operations\n- Runs tests inside Docker container\n- Uploads test coverage to Codecov\n\n##### 1.2.2 Native Runner Workflow (`native-runner.yml`) \n- Runs tests in native Python environment\n- Uses Poetry for dependency management\n- Uploads coverage reports to Codecov\n\n### 2. Local Development Setup\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/ibrahimroshdy/refresher.git\ncd refresher\n```\n\n2. Install Poetry (dependency management):\n```bash\ncurl -sSL https://install.python-poetry.org | python3 -\n```\n\n3. Install dependencies:\n```bash\npoetry install\n```\n\n4. Run migrations:\n```bash\n./scripts/local_migrate.sh\n```\n\n5. Start development server:\n```bash\npoetry run python manage.py runserver\n```\n\n### 3. Docker Deployment\n\n1. Development Environment:\n```bash\ndocker-compose -f docker/docker-compose.dev.yml up -d\n```\n\n2. Staging Environment:\n```bash\ndocker-compose -f docker/docker-compose.staging.yml up -d\n```\n\n3. Production Environment:\n```bash\ndocker-compose -f docker/docker-compose.yml up -d\n```\n\n### 4. Monitoring Stack\n\nThe project includes a comprehensive monitoring stack:\n\n1. **Grafana** (Data Visualization)\n   - Access at: `http://localhost:3000`\n   - Pre-configured dashboards for:\n     - System metrics\n     - Docker container stats\n     - PostgreSQL metrics\n     - Speed test results\n\n2. **Prometheus** (Metrics Collection)\n   - Collects metrics from:\n     - Node Exporter (system metrics)\n     - cAdvisor (container metrics)\n     - PostgreSQL Exporter\n\n3. **Flower** (Celery Monitoring)\n   - Access at: `http://localhost:8888`\n   - Monitor Celery tasks and workers\n   - View task history and statistics\n\n### 5. Application Components\n\n1. **Django Backend**\n   - Admin interface at: `http://localhost:8000/admin`\n   - Manages speed test data and scheduling\n   - Handles database operations\n\n2. **Celery Workers**\n   - Executes scheduled speed tests\n   - Processes results asynchronously\n   - Stores data in PostgreSQL\n\n3. **PostgreSQL Database**\n   - Stores application data\n   - Maintains test history\n   - Handles Celery beat scheduling\n\n4. **Redis**\n   - Message broker for Celery\n   - Caches temporary data\n\n### 6. Security\n\nThe project uses SWAG (Secure Web Application Gateway) which provides:\n- Automatic SSL certificate management via Let's Encrypt\n- Reverse proxy configuration with Nginx\n- Security headers and best practices\n\n### 7. Deployment Pipeline\n\n1. Code changes are pushed to GitHub\n2. CI/CD pipeline runs tests and builds Docker image\n3. Image is pushed to GitHub Container Registry\n4. Deployment script pulls new image and updates services\n5. SWAG handles SSL termination and routing\n\nFor more detailed information about each component, check the respective configuration files in the repository.\n\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibrahimroshdy%2Frefresher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fibrahimroshdy%2Frefresher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibrahimroshdy%2Frefresher/lists"}