{"id":19531166,"url":"https://github.com/contino/web-usage-dashboard","last_synced_at":"2026-04-20T03:34:02.709Z","repository":{"id":77885112,"uuid":"477707571","full_name":"contino/web-usage-dashboard","owner":"contino","description":"AWS-native, event-driven web log analysis platform that scales","archived":false,"fork":false,"pushed_at":"2023-07-25T19:32:25.000Z","size":638,"stargazers_count":1,"open_issues_count":2,"forks_count":2,"subscribers_count":74,"default_branch":"main","last_synced_at":"2025-10-14T14:06:28.728Z","etag":null,"topics":["access-logs","apache-common-log","aws","aws-native","aws-opensearch","cloud-native","cloud-native-architectures","dashboard","elasticsearch","event-driven-architecture","kibana","lambda","makefile","opensearch","terraform","web-log-analysis"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/contino.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-04T13:13:14.000Z","updated_at":"2022-07-02T11:06:59.000Z","dependencies_parsed_at":"2024-11-11T01:41:28.162Z","dependency_job_id":null,"html_url":"https://github.com/contino/web-usage-dashboard","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/contino/web-usage-dashboard","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/contino%2Fweb-usage-dashboard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/contino%2Fweb-usage-dashboard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/contino%2Fweb-usage-dashboard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/contino%2Fweb-usage-dashboard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/contino","download_url":"https://codeload.github.com/contino/web-usage-dashboard/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/contino%2Fweb-usage-dashboard/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279019120,"owners_count":26086679,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["access-logs","apache-common-log","aws","aws-native","aws-opensearch","cloud-native","cloud-native-architectures","dashboard","elasticsearch","event-driven-architecture","kibana","lambda","makefile","opensearch","terraform","web-log-analysis"],"created_at":"2024-11-11T01:41:23.835Z","updated_at":"2025-10-14T14:07:29.242Z","avatar_url":"https://github.com/contino.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Objective\nThis is a demo project to build a dashboard for business to analyse usage of their website using an event driven architecture. Under the hood, it uses AWS OpenSearch service, S3 and Lambda. Project utilizes terraform and Makefile easy maintenance and deployment.\n\nCheck the [Contino blog](https://medium.com/contino-engineering/web-log-analysis-on-aws-up-and-running-in-minutes-59c4063f193c) for more details. \n\n### How it works\n\nWhenever a new file arrives at the s3 bucket, a lambda function is triggered which index the new logs into the target OpenSearch cluster. This way the dashboard is updated in near real time. \n\nFollowing is a sample screenshot of the dashboard.\n\u003cbr\u003e\n\n\u003ckbd\u003e![Dashboard](dashboard-screenshot.png)\u003c/kbd\u003e\n\n\u003cbr\u003e\n\n\u003eWe are assuming the web servers fleet has a mechanism to regularly push logs to an S3 log bucket. This article does not cover that part. However there are sample AWS CLI commands to mimic this behavior.\n\n\u003cbr\u003e\n\n### Design\n\n\n![Architecture](architecture.jpg)\n\n### Dataset\n\nFor testing, we are using a [public dataset](http://www.almhuette-raith.at/apache-log/access.log) which has about 3million Apache web access logs entries. Samples from this dataset are stored under ./sample_data directory.\n\n\n### Notes:\n\n\u003eThis is for demo only. For production use-cases, more strict security settings are required.\n\n\n* To keep it simple, no VPC is used.\n* Kibana Dashboard and ElasticSearch endpoint will be accessible from a single IP address only. This is the public IP from which the project is deployed.\n* At the time of this work, the latest Terraform AWS provider version was 4.8. The AWS change from ElasticSearch to OpenSearch is not updated on it. Hence the code is using ElasticSearch 7.10 which is the last version of ElasticSearch AWS supports. This change is in the pipeline for terraform. Ref: [Terraform ticket](https://github.com/hashicorp/terraform-provider-aws/issues/21787)\n* Project is relying on the UNIX make command to facilitate CI/CD. Run the \"make help\" command for more details.\n* Dashboard is generated from an existing file: kibana_dashboard_export.ndjson\n\u003cbr\u003e\n\n### Development Setup\n\nIf not already there, install Python3, Terraform, pipenv and AWS CLI. You would need to configure AWS CLI credentials prior to running the \"make deploy\" command.\n\n```\n# Install pipenv\npip install pipenv\n\n# Install \u0026 Configure AWS CLI\npip install awscli\naws configure\n```\n\nCheckout project\n```\ngit clone https://github.com/contino/web-usage-dashboard.git\ncd web-usage-dashboard\n```\n\nConfigure Python virtual environment and download dependencies:\n\n```\nmake init\n```\n\n### Test\nYou can run Python unit tests for Lambda code as follows.\n```\nmake test\n```\n\n### Build\nThis will package the lambda code along with its dependencies into a deployable zip file.\n```\nmake build\n```\n### Deploy \nThis will deploy the terraform code into AWS cloud.\n```\nmake deploy PROJECT_NAME=\u003cproject name\u003e\n```\nUnder the hood, it constructs and executes the \"terraform apply\" command. On successful completion, this will create the required cloud resources and generate an output as follows. \n```\nKibana Dashboard: \"https://\u003cdomain name\u003e.eu-west-2.es.amazonaws.com/_plugin/kibana/app/dashboards\"\n\nUpload a sample log file to test by using the following command:\naws s3 cp ./sample_data/2021-12.log  s3://manitest6-20220414145147066300000001/\n```\n\n### Demo\nThe application can be tested  by manually adding log files into S3. There is no data in the cluster immediately after it is created. To demo, follow the steps below.```\n```\n# Step-1: Deploy the infrastructure. This may take a few mins.\nmake deploy PROJECT_NAME=\u003cproject name\u003e\n\n# Step-2: Open Kibana URL in your preferred browser. \n#         Dashboard is empty at this stage as there is no data indexed yet.\n\n# Step-3: Add logs to bucket\naws s3 cp ./sample_data/2021-12.log  s3://\u003cbucket name\u003e/ \n\n# Step-4: Go back to the dashboard and refresh. It should have data for the period 2021-12 now.\n```\n## How to destroy \n```\nmake destroy PROJECT_NAME=\u003cproject name\u003e\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontino%2Fweb-usage-dashboard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcontino%2Fweb-usage-dashboard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontino%2Fweb-usage-dashboard/lists"}