{"id":22389033,"url":"https://github.com/sporestudio/web-server","last_synced_at":"2026-04-15T05:32:09.875Z","repository":{"id":261513147,"uuid":"884464249","full_name":"sporestudio/web-server","owner":"sporestudio","description":"Automated deployment of an apache web server using Docker containers, as well as monitoring tools for the server.","archived":false,"fork":false,"pushed_at":"2025-01-06T17:29:27.000Z","size":6013,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-26T21:30:20.072Z","etag":null,"topics":["apache","bash","docker","docker-compose","jenkins","python"],"latest_commit_sha":null,"homepage":"","language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sporestudio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-06T19:52:39.000Z","updated_at":"2025-01-06T12:04:38.000Z","dependencies_parsed_at":"2024-11-14T19:41:34.502Z","dependency_job_id":"f1687b61-9bb2-4c7e-b315-a58946745639","html_url":"https://github.com/sporestudio/web-server","commit_stats":null,"previous_names":["sporestudio/url_shortener","sporestudio/web-server"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sporestudio/web-server","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sporestudio%2Fweb-server","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sporestudio%2Fweb-server/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sporestudio%2Fweb-server/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sporestudio%2Fweb-server/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sporestudio","download_url":"https://codeload.github.com/sporestudio/web-server/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sporestudio%2Fweb-server/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31828531,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T18:05:02.291Z","status":"online","status_checked_at":"2026-04-15T02:00:06.175Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache","bash","docker","docker-compose","jenkins","python"],"created_at":"2024-12-05T03:08:43.725Z","updated_at":"2026-04-15T05:32:09.853Z","avatar_url":"https://github.com/sporestudio.png","language":"CSS","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web Server Deployment\n\n## Project Overview\n\nAutomated deployment of a web server made with Docker containers. The server is composed by a main domain with the main page of the web, and serveral subdomains used for two aplications: **an url shortener and a youtube video and audio downloader**. \nWe have another two subdomains, one of these for the **grafana monitoring interface** and another one for the **jenkins administration panel**.\n\n## Index\n\n- **Web Server Deployment**\n    - [Structure](#structure)\n    - [Key Features](#key-features)\n    - [Dependencies](#dependencies)\n    - [Previous Configurations](#previous-configurations)\n    - [Deployment](#deployment-of-the-project)\n    - [Web Server Configuration](#web-server-configuration)\n    - [Url Shortener](#url-shortener-app)\n    - [YouTube Downloader](#youtube-downloader-app)\n    - [Grafana Monitoring](#grafana-monitoring-tool)\n    - [CI/CD Pipeline](#cicd-pipeline-with-jenkins)\n    - [Benchmarks and Tests](#benchmarks-and-tests)\n    - [License](#license)\n    - [Contribute](#contribute)\n    - [Author](#author)\n\n## Structure\n\n```\n├── web-server\n│   ├── certbot\n│   │   ├── htdocs\n|   |   |   └── index.html\n|   |   └── httpd.conf\n|   ├── dyndns\n|   |   ├── Dockerfile\n|   |   ├── cronjob\n|   |   ├── geturl.sh\n|   |   └── update.sh\n│   ├── url-shortner\n|   |   ├── static\n|   |   |   └── style.css\n|   |   ├── templates\n|   |   |   └── index.html\n|   |   ├── tests\n|   |   |   ├── test_path.py\n|   |   |   └── test_unitary.py\n|   |   ├── app.py\n|   |   ├── dns_manager.py\n|   |   ├── Dockerfile\n|   |   ├── requirements.txt\n|   |   └── widgets.py\n|   ├── web\n|   |   ├── htdocs\n|   |   |   ├── admin\n|   |   |   |   └── index.html\n|   |   |   ├── assets\n|   |   |   ├── styles\n|   |   |   |   └── style.css\n|   |   |   ├── .htpasswd\n|   |   |   ├── contact.html\n|   |   |   ├── error404.html\n|   |   |   ├── forbidden403.html\n|   |   |   ├── index.html\n|   |   |   └── logo.png\n|   |   ├── httpd-vhosts.conf\n|   |   └── httpd.conf\n|   ├── web-downloader\n|   |   ├── static\n|   |   |   └── style.css\n|   |   ├── templates\n|   |   |   └── index.html\n|   |   ├── app.py\n|   |   ├── Dockerfile\n|   |   └── requirements.txt\n|   ├── .gitignore\n|   ├── Jenkinsfile\n|   ├── LICENSE\n|   ├── Makefile\n|   ├── README.md\n|   ├── compose.yml\n|   ├── prometheus.yml\n|   └── test.hurl\n\n```\n\n## Key Features\n\n- Self-hosted deployment leveraging **Docker Compose** for container orchestation.\n- Secure access through **HTTPS** via Let's Encrypt certificates and reverse proxy configured with Apache\n- System Monitoring with **Grafana**.\n- **CI/CD Pipeline** using Jenkins.\n\n\u003e It's important to note that the **CI/CD pipeline** is still under development.\n\n## Dependencies\n\nThe required dependecies for deploy the project are:\n\n- **Docker**\n- **Python**\n- **Docker-compose**\n- **Make**\n\n## Previous configurations\n\n### Router configurations\n\nIn this project the server will be my personal computer, which does not have a public IP, so we must map port 80 of our router with port 8080 of our machine, as well as port 443 with port 4433 of localhos to allow HTTPs traffic.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./.assets/img/router2.png\"\u003e\n\u003c/div\u003e\n\nWe have to login in the router and go to the *NAT/PAD* that is usually where we can open ports (at least in Orange routers)\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./.assets/img/router1.png\"\u003e\n\u003c/div\u003e\n\n#### Dynamic DNS \n\nAs we do not have a static IP configured in our router, but we have a dynamic IP that changes from time to time, we will need a dynamic DNS service for our domain to point to our IP even though it may change.\nIn this case, I have purchased the domain from IONOS, so **this documentation is based on the steps to follow to configure the dynamic DNS service with IONOS as the provider**.\n\nThe first thing we have to do is to generate an API KEY to be able to interact with the IONOS service. To do this we have to visit the web: https://developer.hosting.ionos.es/?source=IonosControlPanel, go to the IONOS developer section and here we will find an option called manage keys.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./.assets/img/ionos-developer.png\"\u003e\n\u003c/div\u003e\n\nOnce we have our API Key we have to go the [DNS documentation section](https://developer.hosting.ionos.es/docs/dns) and authorize the service with our API Key.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./.assets/img/authorize1.png\"\u003e\n\u003c/div\u003e\n\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./.assets/img/authorize2.png\"\u003e\n\u003c/div\u003e\n\nWe must make a POST request with our API Key and our domains and we will receive the url in JSON format.\n\n```json\n{\n  \"bulkId\": \"e0d3db69-19cc-449a-aca0-7fb4b69ba774\",\n  \"updateUrl\": \"https://ipv4.api.hosting.ionos.com/dns/v1/dyndns?q=YTFmN2Q5Y2VkYmQ0NDE2OWJlZDEwNDBiZDRlNTFlNTkuUTNPaWJTaG1rMnBpUVVoMUhlcDdlVWpyZ2Mxb0J0MEdsbHZrSWF6dzVlTURZMjZON2VtUlFKS1k2SFhfeEVMaEs1Y1cyRjJvV1NhcUhTajVUNVlSYlE\",\n  \"domains\": [\n    \"sporestudio.me\",\n    \"www.sporestudio.me\"\n  ],\n  \"description\": \"My DynamicDns\"\n}\n```\n\n\u003e[!NOTE]\n\u003e We can obtain the update url automatically executing the geturl.sh script, you just change the fields of the API Key with your code and change my domains to yours.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./.assets/img/geturl.jpeg\"\u003e\n\u003c/div\u003e\n\nTo make sure that the IP is always update we encapsulated this service in a docker container that is running cron updating the IP address every minute. We've created an image using the official Docker's Debian image where we copy the script `update.sh` to the container and a crontab will be running the script every minute.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/update.sh\"\u003e\n\u003c/div\u003e\n\nThis script will be running in our Docker container as a service, so we've created a docker image for it.\n\n```Dockerfile\nFROM debian:12.8\nRUN apt-get update \u0026\u0026 apt-get -y install cron curl\n\nWORKDIR /app\n\nCOPY update.sh update.sh\nCOPY update_url update_url\nRUN chmod +x update.sh\n\nCOPY cronjob /etc/cron.d/cronjob\nRUN chmod 0644 /etc/cron.d/cronjob\nRUN crontab /etc/cron.d/cronjob\n\nCMD [\"cron\", \"-f\", \"/etc/cron.d/cronjob\"]\n```\n\n### Install dependencies for the project\n\nBefore deploy the project we have to install the necessary dependencies for our apps.\n\n#### Install url-shortener dependencies\n\nWe have to navigate to the app directory and create a virtual enviroment (optional but recommended).\n\n```bash\n$ cd url-shortener/\n$ python -m venv venv\n$ source venv/bin/activate\n```\n\nNow install dependencies with `pip install`.\n\n```bash\n$ pip install -r requirements.txt\n```\n\n\n#### Install web-downloader dependencies\n\nThe steps to install the dependencies of web-downloader will be the same, we have to navigate to web-downloader directory and install the dependencies with `pip install`.\n\n```bash\n$ cd web-downloader/\n$ python -m venv venv\n$ source venv/bin/activate\n$ pip install -r requirements.txt\n```\n\n### Create .env file\n\nYou will need to create a `.env` file with the following fields for the project to work.\n\n```bash\n# FLASK Configuration\nFLASK_APP=\n\n# IONOS API Configuration\nIONOS_BASE_URL=\nIONOS_API_KEY=\n\n# Apache global vars\nDOMAIN_NAME=\nGRAFANA_DOMAIN=\nDOMAIN_ID=\nSERVER_ADMIN=\n\nSTATUS_PASSWD=\n\n# Variables for Jenkins configs\nJENKINS_URL=\nJENKINS_AGENT= \nJENKINS_SECRET=\n```\n\n\u003e[!CAUTION]\n\u003e Never commit sensitive values to control version.\n\n\n## Deployment of the project\n\nWe have several ways to deploy this project, one of them using the `Makefile` or using `docker-compose`.\n\n### Using Makefile\n\n- Deploy the server and generate/renew SSL certificates:\n\n```bash\n$ make all\n```\n\n- Deploy without generating new certificates:\n\n```bash\n$ make deploy\n```\n\n### Using docker-compose\n\n- Deploy the server building the docker images.\n\n```bash\n$ docker-compose up --build -d\n```\n\n## Web server configuration\n\n### Obtain the SSL certificates via Cerbot\n\nIn order to provide a safe and encrypted connection to our server, we will need a valid certificate. We will do using [certbot](https://hub.docker.com/r/certbot/certbot) which is a docker image provide by [**Let's Encrypt**](https://letsencrypt.org/es/), an open and free certificate authority.\n\nCetbot uses the ACME (Automatic Certificate Management Enviroment)  protocol, provided by Let's Encrypt to obtain and renewing the SSL certificates.\n\n#### How Certbot and ACME work together\n\n1. **Domain Ownership Validation**:\n    - Certbot proves to the Let's Encrypt Certificate Authority (CA) that you control the domain for which you are requesting an SSL certificate.\n\n    - The ACME protocol facilitates this validation process.\n\n2. **Validation method**: Certbot uses severals methods to complete domain validation, in our case, we're going to explain the HTTP-01 Challenge that is what we used in this project, to more info about the other methods you can check the [certbot official documentation](https://eff-certbot.readthedocs.io/en/stable/).\n\n    - **HTTP-01 Challenge**: Is a method used by Certbot and other ACME clients to validate domain ownership for *SSL/TLS* certificate issuance. It works by creating a unique token and placing it in a specific file (`/.well-known/acme-challenge/*TOKEN*`) on your server. \n    \n        The Certificate Authority (CA), such as Let's Encrypt, then makes an HTTP request to retrieve this file and confirm its contents. If the file is correctly served, the CA verifies that you control the domain and issues the certificate. This method requires the domain to be publicly accessible via HTTP (port 80) but is simple to automate, making it ideal for many web servers.\n\n3. **Certificate Issuance**:\n    - Once ownership is confirmed, Let's Encrypt issues an *SSL/TLS* certificate for your domain.\n\n    - Certbot downloads and configures this certificate, usually placing the files in `/etc/letsencrypt/live/\u003cyourdomain\u003e/`.\n\n4. **Automatic Renewal**:\n    - Certificates from Let's Encrypt are valid for 90 days.\n\n    - Certbot includes a built-in renewal mechanism *(certbot renew)*, which ensures your certificates remain valid without manual intervention.\n\nSo in this project to obtain the SSL certificates I've created a temporary web server listening on port 80 with the domain we want to certificate, and once the server is listening on port 80, we will run the certbot container to obtain the certificates:\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/makefile.jpeg\"\u003e\n\u003c/div\u003e\n\n\u003e [!NOTE]\n\u003e The certificates will be created on the container's directory `/etc/letsencrypt`, so we created a permanent docker volume called **certs** to preserve it.\n\n### Apache configuration\n\nThe main web server will be running in a Docker container with the official image of Apache: [**httpd**](https://hub.docker.com/_/httpd)\n\n```yaml\nweb:\n    image: httpd:latest\n    container_name: web\n    env_file:\n        - .env\n    volumes:\n        - ./web/htdocs:/usr/local/apache2/htdocs\n        - ./web/httpd.conf:/usr/local/apache2/conf/httpd.conf\n        - ./web/httpd-vhosts.conf:/usr/local/apache2/conf/extra/httpd-vhosts.conf\n        - certs:/etc/letsencrypt\n    ports:\n        - 8080:80\n        - 4433:443\n```\n\nAs we can see we map the certs docker volume to the container directory **/etc/letsencrypt**. The certificates will be located at `/etc/letsencrypt/live/[your-domain]/fullchain.pem` and `/etc/letsencrypt/live/[your-domain]/privkey.pem`\n\n#### Virtual hosts configuration\n\n**The server have several virtual hosts configured**, one is for the main server, and we have another four virtual hosts, two of them for the applications **url shortener and youtube downloader**, other for the **Grafana monitoring system** and another one for the **Jenkins administration panel**.\n\n##### Reverse proxy configuration\n\n**Apache acts as an intermediary**, forwarding external client requests to the internal services, allowing this services to remain isolated from direct public access.\n\nHow can see for example in the virtual host for the url shortener application, we forward HTTP requests to the internal service at `http://url-shortener:5000/`.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/shortener-vhost.jpeg\"\u003e\n\u003c/div\u003e\n\n## Url shortener app\n\nThe url shortener app is a web service deployed as part of the self-hosted infrastructure at `https://url.sporestudio.me`. Its primary purpose is to simplify lengthy URLs into compact links, making them easier to share or manage.\n\n### Key features\n-  **Custom Shortened URLs**: Converts longs URLs into concise, shareable links.\n\n- **Hash-Based Identification**: Uses a hashing algorithm (SHA-256) to generate unique codes for each URL.\n\n- **Persistent Storage**:Ensures that shortened links are stored in a database for retrieval and analytics.\n\n- **Redirect Service**: Automatically redirects users from the short link to the original, full URL.\n\n### Technology Stack\n- **Backend**:\n    - Built using **Python** with the [**Flask**](https://flask.palletsprojects.com/en/stable/) web framework.\n\n    - Implements cryptographic functions via the [hashlib](https://docs.python.org/3/library/hashlib.html) library for generating unique hash codes for URLs.\n\n- **Frontend**: A minimal interface for submitting URLs, generating shortened links, and monitoring usage.\n\n- **Database**: In this case we've used the IONOS **DNS server** as database, storing the links as **TXT type records** within the server.\n\n- **Reverse Proxy**: Managed through Apache, which handles HTTP(S) requests to the app.\n\n\n### Workflow\n\nFirst the users **submit a long URL** through the app interface.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/urlshort-interface.png\"\u003e\n\u003c/div\u003e\n\nThen we catch this long url input by the users via form through POST method. To this long url we apply a function that generates a **hash of 6 alphanumeric characters**, which will be our short url, and using the function of our library [dns_manager.py](https://github.com/sporestudio/web-server/blob/main/url-shortener/dns_manager.py), created by us, we create a **TXT record in the DNS server with the shortened url** and then print it in the `index.html` using the **render_template** function of Flask.\n\n```python\ndef shortener_url(original_url):\n    return hashlib.sha256(original_url.encode()).hexdigest()[:6]\n\n\n@app.route('/', methods=['GET', 'POST'])\ndef index():\n    if request.method == 'POST':\n        original_url = request.form['url']\n        short_url = shortener_url(original_url) \n        \n        if create_txt_record(short_url, original_url):\n            url_shortener = f\"https://url.{DOMAIN_NAME}/{short_url}\"\n            return render_template('index.html', short_url=url_shortener, original_url=original_url)\n        else:\n            return \"Error creating DNS TXT record\", 500\n\n    return render_template('index.html')\n```\n\nFinally **we have to handle the redirection** from the short url to the long url making use again of the [dns_manager.py](https://github.com/sporestudio/web-server/blob/main/url-shortener/dns_manager.py) library, where we make a DNS query so that if there is a TXT record for that short url we redirect it to the value of that record (which is the original url).\n\n```python\n@app.route('/\u003cshort_url\u003e')\ndef redirect_url(short_url):\n    original_url = get_original_url(short_url)\n    if original_url:\n        return redirect(original_url)\n    else:\n        return \"Short url not found\", 404\n```\n\n### Tests\n\nWe have created some tests that guarantee the performance and uniformity of the application code. For this I have used the Pytest library that allows you to create tests for your application easily in a few lines of code.\n\n#### Path test\n\n- **Purpose**: Verifies that the application's main page is accessible and displays the correct content.\n\n- **Details**: \n\n    - **Tested Funcionality**: Sends a GET request to the home (/) route.\n\n    - **Assertions**: \n\n        - The HTTP status code of the response is 200, indicating the page loaded successfully.\n\n        - The response contains the text URLs shortener, confirming the main page displays the intended content.\n\n```python\nimport pytest\nfrom app import app\n\n@pytest.fixture\ndef client():\n    with app.test_client() as client:\n        yield client\n\ndef test_home_path(client):\n    response = client.get(\"/\")\n    assert response.status_code == 200\n    assert b\"URLs shortener\" in response.data\n```\n\n#### Short URL Genereation Test\n\n- **Purpose**: Ensures the `shortener_url` function generates valid, unique short URLs.\n\n- **Details**: \n\n    - **Tested Funcionality**: Calls the `shortener_url` function with a sample URL as input.\n\n    - **Assertions**: \n\n        - The generated short URL is exactly 6 characters long, as per the application's design.\n\n        - The short URL is alphanumeric, ensuring it is user-friendly and URL-compatible.\n\n```python\nimport pytest\nfrom app import shortener_url\n\ndef test_generate_short_url():\n    original_url = \"https://example.com\"\n    short_url = shortener_url(original_url)\n    assert len(short_url) == 6\n    assert short_url.isalnum()\n```\n\n#### How to run the tests\n\n1. Ensure that all dependencies are installed. Navigate to url-shortener directory, create a virtual enviroment and activate it (optional but recommended).\n\n```bash\n$ cd url-shortener/\n$ pyhton3 -m venv venv\n$ source venv/bin/activate\n```\n\n2. Then install the dependencies from the `requirements.txt` with pip.\n\n```bash\n$ pip install -r requirements.txt\n```\n\n3. Run the tests.\n\n```bash\n$ pytest\n```\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/pytest.png\"\u003e\n\u003c/div\u003e\n\n## YouTube downloader app\n\n**YouTube downloader** application is a Python-based tool using [Flask](https://flask.palletsprojects.com/en/stable/) for the web server and [pytubefix](https://github.com/JuanBindez/pytubefix) (I had some issues with pytube lib) for handling YouTube video downloads.\n\n### Key features\n\n- **Web Interface**: A simple page where users can paste the YouTube url and select the quality of the video or even just download the audio.\n\n- **Video Quality Options**:\n    \n    - **Highest**: Downloads the video in the highest available resolution.\n\n    - **Lowest**: Downloads the video in the lowest available resolution.\n\n    - **Audio**: Downloads only the audio stream (useful for music or podcasts).\n\n- **Temporary File Management**: Downloads the file to a temporary location and deletes it after serving it to the user `(os.remove(output_path))`.\n\n### Workflow\n\nThe main function of the app uses route `/download` that defines the URL endpoint where the video download process is handled. When the user interacts with the webpage, such as by submitting a form to download a video, the browser sends a POST request to the `/download` URL.\n\n```python\n@app.route('/download', methods=['POST'])\ndef down_video():\n    url = request.form['url']\n    quality = request.form['quality']\n    try:\n        yt = YouTube(url)\n\n        if quality == 'highest':\n            stream = yt.streams.get_highest_resolution()\n        elif quality == 'lowest':\n            stream = yt.streams.get_lowest_resolution()\n        elif quality == 'audio':\n            stream = yt.streams.filter(only_audio=True).first()\n        else:\n            return \"Invalid quality selected\"\n    \n        output_path = stream.download()\n        response = send_file(output_path, as_attachment=True, download_name=f\"{yt.title}.{stream.subtype}\")\n        os.remove(output_path)\n        return response\n    except Exception as e:\n        return f\"An error occurred: {str(e)}\"\n```\n\nThen we extracting the user input from the form and the quality specified. As we can see we have to create a **YouTube object** for the provided URL using `pytubefix` library.\n\nWe select the appropriate stream for downloading and the path of the downloaded file is stored in **output_path** varibale. To finish we send the file to the user.\n\n## Grafana monitoring tool\n\nTo provide our server with monitor functions, we will use [Grafana](https://grafana.com/docs/grafana/latest/) in combination with [Prometheus](https://prometheus.io/docs/visualization/grafana/) and [apache_exporter](https://github.com/Lusitaniae/apache_exporter).\n\nWe will use apache_exporter to scrape data from **/status** (`mod_status`) and transform this data into a format that Prometheus can understand. \n\nSo we will use the official apache exporter image from **Docker Hub** to create the apache exporter container.\n\n\u003e We will pass the URL where we allocated the `mod_status` and the authentication.\n\n```yaml\napache-exporter: \n    image: lusotycoon/apache-exporter\n    container_name: apache-exporter\n    depends_on:\n      - web\n    privileged: true\n    expose:\n      - 9117\n    restart: unless-stopped\n    extra_hosts:\n    - \"localhost:127.17.0.1\"\n    env_file:\n      - .env\n    entrypoint: /bin/apache_exporter --scrape_uri=\"https://sysadmin:${STATUS_PASSWD}@sporestudio.me/status?auto/\"\n```\n\nNow we have to configure the `prometheus.yml` to indicate the socket of the host that have the data we want to display. In our case, *apache_exporter:9117*.\n\n\u003e [!NOTE]\n\u003e Since we have all the containers in the same compose file, docker-compose will resolve the addresses for us.\n\n```yaml\n# my global config\nglobal:\n  scrape_interval: 1m \n\n# A scrape configuration containing exactly one endpoint to scrape:\nscrape_configs:\n  - job_name: \"apache_exporter\"\n    static_configs:\n      - targets: [\"apache-exporter:9117\"]\n```\n* *prometheus.yml*\n\nOnce we have this configured, we have to create the container for Prometheus service, we are using the official Prometheus image from **Docker hub**.\n\n```yaml\nprometheus:\n    image: prom/prometheus\n    container_name: prometheus\n    depends_on:\n      - apache-exporter\n    restart: unless-stopped\n    volumes:\n      - ./prometheus.yml:/etc/prometheus/prometheus.yml\n      - prometheus_data:/prometheus\n    command:\n      - '--config.file=/etc/prometheus/prometheus.yml'\n      - '--storage.tsdb.path=/prometheus'\n      - '--web.console.libraries=/etc/prometheus/console_libraries'\n      - '--web.console.templates=/etc/prometheus/consoles'\n      - '--web.enable-lifecycle'\n    expose:\n      - 9090\n```\n\n\u003e *Prometheus will be listen in port 9090*.\n\n### Grafana configuration\n\nLast but not least, we have to configure the grafana container using the official image from, again, Docker Hub.\n\n```yaml\ngrafana:\n    image: grafana/grafana\n    container_name: grafana\n    expose:\n      - 3000\n    volumes:\n      - grafana_data:/var/lib/grafana\n```\n\nI've created a virtual host for garafana to can configure it from a subdomain within my page. Here is the configuration for this subdomain:\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/grafana-domain.jpeg\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n\nSo now when we visit https://grafana.sporestudio.me we access the grafana login panel.\n\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/grafana-login.png\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n\nOnce inside Grafana we have to navigate to **Menu \u003e Connections \u003e Add new connection**, and here we have to select *Prometheus* as data source.\n\n\nIt is important to note that the **URL in our case is the name of the Docker container** with the prometheus service, since docker compose will resolve the IPs for us.\n\n\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/prometheus-connect.png\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n\nOnce we save and test the data source connection, we have to see a message like this:\n\n\n\u003cimg src=\".assets/img/successful.png\"\u003e\n\n\u003e *Successful connection*.\n\nAfter complete the connection, we can create our dashboard with the apache exporter instances that we choose.\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/grafana-data.png\"\u003e\n\u003c/div\u003e\n\n## CI/CD Pipeline with Jenkins\n\n\u003e [!WARNING]\n\u003e This service is still under developoment.\n\n## Benchmarks and Tests\n\n### Apache Benchmark\n\n[Apache Benchmark](https://httpd.apache.org/docs/2.4/programs/ab.html) was used to evaluate server's performance. Several tests have been applied to the main page and its paths.\n\n#### 100 clients and 1000 requets\n\nWith the *-k* flag:\n\n- **sporestudio.me/**\n\n```bash\n$ ab -f SSL3 -k -c 100 -n 1000 -H \"Accept-Encoding: gzip, deflate\" https://sporestudio.me\n```\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/test1-ab.png\"\u003e\n\u003c/div\u003e\n\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/test-ab22.png\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n\n- **sporestudio/admin**\n\n```bash\n$ ab -f SSL3 -k -c 100 -n 1000 -A admin:{passwd} -H \"Accept-Encoding: gzip, deflate\" https://sporestudio.me/admin\n```\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/admin-ab.png\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n\nWithout *-k* flag:\n\n- **sporestudio.me/**\n\n```bash\n$ ab -f SSL3 -c 100 -n 1000 -H \"Accept-Encoding: gzip, deflate\" https://sporestudio.me\n```\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/nok-ab.png\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n- **sporestudio/admin**\n\n```bash\n$ ab -f SSL3 -c 100 -n 1000 -A admin:{passwd} -H \"Accept-Encoding: gzip, deflate\" https://sporestudio.me/admin\n```\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/nok-admin.png\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n\n#### 1000 clients and 10000 requests\n\nWith *-k* flag:\n\n- **sporestudio.me/**\n\n```bash\n$ ab -f SSL3 -k -c 1000 -n 10000 -H \"Accept-Encoding: gzip, deflate\" https://sporestudio.me\n```\n\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/ab-test5.png\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n\n- **sporestudio/admin**\n\n```bash\n$ ab -f SSL3 -k -c 1000 -n 10000 -A admin:{passwd} -H \"Accept-Encoding: gzip, deflate\" https://sporestudio.me/admin\n```\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/admin-abtest5.png\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n\nWithout *-k* flag:\n\n- **sporestudio.me/**\n\n```bash\n$ ab -f SSL3 -c 1000 -n 10000 -H \"Accept-Encoding: gzip, deflate\" https://sporestudio.me\n```\n\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/nok-1000.png\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n\n- **sporestudio/admin**\n\n```bash\n$ ab -f SSL3 -c 1000 -n 10000 -A admin:{passwd} -H \"Accept-Encoding: gzip, deflate\" https://sporestudio.me/admin\n```\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/admin-nok1000.png\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n\n#### Conclusions\n\n- We can see that if we enable the Keep-Alive connections (*-k*), tests significantly improves performance.\n\n- Adding headers helped reduce overall bandwidth usage but didn’t entirely mitigate server overload.\n\n- The test with 100 clients concurrency shows the highest Requests per Second and the lowest Time per Request, indicating improved efficiency at lower concurrency levels. \n\n### Hurl\n\nWe can run a test with [hurl](https://hurl.dev/) that performs GET requests to our website and different subdomains, to check that they exist.\n\n```bash\n# Index page exists\nGET https://{{site}}/\nHTTP 200\n\n# Logo exists\nGET https://{{site}}/logo.png\nHTTP 200\n\n# Administration\nGET https://{{site}}/admin\n[BasicAuth]\nadmin: asir\n\n# Status\nGET https://{{site}}/status\n[BasicAuth]\nsysadmin: risa\n\n# Url shortener subdomain\nGET https://url.{{site}}\nHTTP 200\n\n# YouTube downloader subdomain\nGET https://downs.{{site}}\nHTTP 200\n```\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\".assets/img/hurl-test.png\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\n## License\n\nThis project is under \u003ca href=\"https://github.com/sporestudio/web-server/blob/main/LICENSE\"\u003eGNU General Public License v3.0\u003c/a\u003e.\n\n## Contribute\n\nWant to contribute? There are multiple ways you can contribute to this project. Here are some ideas:\n\n* [Translate the web into multiple languages!](./CONTRIBUTING.md#translations)\n* [Fix some easy issues](CONTRIBUTING.md#Reporting-Issues)\n* [Or check out some other issues](CONTRIBUTING.md#Reporting-Issues) (or translate them).\n\n## Author\n\nCreated by \u003ca href=\"https://github.com/sporestudio/\"\u003esporestudio\u003c/a\u003e.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsporestudio%2Fweb-server","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsporestudio%2Fweb-server","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsporestudio%2Fweb-server/lists"}