{"id":20876881,"url":"https://github.com/dabico/nginx-access-log-digest","last_synced_at":"2026-04-17T15:33:33.746Z","repository":{"id":223202459,"uuid":"759578660","full_name":"dabico/nginx-access-log-digest","owner":"dabico","description":"An analysis of a platform's usage through data mining and visualisation","archived":false,"fork":false,"pushed_at":"2024-02-20T16:49:56.000Z","size":116,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-18T01:47:24.608Z","etag":null,"topics":["data-mining","data-science","data-visualisation","geopandas","information-retrieval","ip-geolocation","ipinfo","jupyter-notebook","matplotlib","nginx","pandas","python3","visual-analytics"],"latest_commit_sha":null,"homepage":"https://dabico.github.io/nginx-access-log-digest/notebook.html","language":"Jupyter Notebook","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dabico.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-18T23:04:55.000Z","updated_at":"2025-01-20T09:51:00.000Z","dependencies_parsed_at":null,"dependency_job_id":"b037fc13-0140-43f4-ae76-e97cf3aa87bf","html_url":"https://github.com/dabico/nginx-access-log-digest","commit_stats":null,"previous_names":["dabico/nginx-access-log-digest"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dabico/nginx-access-log-digest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dabico%2Fnginx-access-log-digest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dabico%2Fnginx-access-log-digest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dabico%2Fnginx-access-log-digest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dabico%2Fnginx-access-log-digest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dabico","download_url":"https://codeload.github.com/dabico/nginx-access-log-digest/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dabico%2Fnginx-access-log-digest/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31934349,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-17T12:37:54.787Z","status":"ssl_error","status_checked_at":"2026-04-17T12:37:25.095Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-mining","data-science","data-visualisation","geopandas","information-retrieval","ip-geolocation","ipinfo","jupyter-notebook","matplotlib","nginx","pandas","python3","visual-analytics"],"created_at":"2024-11-18T06:55:14.982Z","updated_at":"2026-04-17T15:33:33.730Z","avatar_url":"https://github.com/dabico.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NGINX Access Log Digest\n\nThis project is a two-part exercise in data analysis and visualization.\nThe first part is a Python script that reads an NGINX access log and\ntransforms it into a more structured format. The second part is a\nJupyter notebook that uses the produced data to analyze and visualize\nthe last two months of traffic on one of the platforms I work on:\n[GitHub Search](https://seart-ghs.si.usi.ch).\n\n## Mining Script\n\nThe mining script is a Python program that reads an NGINX access log\nline-by-line and extracts the following fields:\n\n- IP address\n- Timestamp\n- HTTP request\n  - Method\n  - Path\n- HTTP response\n  - Size in bytes\n  - Status code\n- User Agent\n  - Browser\n  - Device\n  - OS\n- Referrer\n\nApart from the fields above, the script also calculates the location of\neach IP address using the [ipinfo.io](https://ipinfo.io/) API. All this\ndata is loaded into dataclasses and then exported to a CSV file. The data\nmodel used can be represented with the following diagram:\n\n```mermaid\nerDiagram\n    Event ||--o{ Access : \"\"\n    Access {\n        string ip\n        timestamp timestamp\n        json query\n        int status\n        string size\n        string referer\n        json user_agent\n    }\n    Event ||--o{ City : \"\"\n    City {\n        string name\n        string region\n    }\n    City ||--|| Position : \"\"\n    Position {\n        float latitude\n        float longitude\n    }\n    City ||--|| Country : \"\"\n    Country {\n        string name\n        string code\n    }\n    Country ||--|| Continent : \"\"\n    Continent {\n        string name\n        string code\n    }\n```\n\n## Jupyter Notebook\n\nThe Jupyter Notebook uses the data produced by the mining script to\nperform the actual analysis and visualization. It's written in the\nstyle of both a report and a tutorial. For each section, it explains the\nindividual steps taken to produce the analysis and visualization, while\nalso providing commentary on the results. Although we provide the data\nto run the notebook yourself, we have also deployed it to a GitHub\nPages site, which you can access\n[here](https://dabico.github.io/nginx-access-log-digest/notebook.html).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdabico%2Fnginx-access-log-digest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdabico%2Fnginx-access-log-digest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdabico%2Fnginx-access-log-digest/lists"}