{"id":26096167,"url":"https://github.com/molu8bits/squid-filebeat-kibana","last_synced_at":"2025-10-28T06:35:27.460Z","repository":{"id":177955945,"uuid":"159408363","full_name":"molu8bits/squid-filebeat-kibana","owner":"molu8bits","description":"Filebeat module for Squid access.log + Kibana dashboards. ELK 7.x","archived":false,"fork":false,"pushed_at":"2020-09-19T16:52:19.000Z","size":321,"stargazers_count":17,"open_issues_count":4,"forks_count":14,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-12T11:44:04.319Z","etag":null,"topics":["elasticsearch","elk","filebeat","kibana","kibana-dashboard","squid","squid-access","squid-filebeat-kibana","visualisation","visualisations"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/molu8bits.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-27T22:27:37.000Z","updated_at":"2024-06-18T20:44:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"116814aa-4ab8-4344-99ac-999abf23773b","html_url":"https://github.com/molu8bits/squid-filebeat-kibana","commit_stats":null,"previous_names":["molu8bits/squid-filebeat-kibana"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/molu8bits/squid-filebeat-kibana","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molu8bits%2Fsquid-filebeat-kibana","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molu8bits%2Fsquid-filebeat-kibana/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molu8bits%2Fsquid-filebeat-kibana/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molu8bits%2Fsquid-filebeat-kibana/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/molu8bits","download_url":"https://codeload.github.com/molu8bits/squid-filebeat-kibana/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/molu8bits%2Fsquid-filebeat-kibana/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281397340,"owners_count":26493908,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-28T02:00:06.022Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","elk","filebeat","kibana","kibana-dashboard","squid","squid-access","squid-filebeat-kibana","visualisation","visualisations"],"created_at":"2025-03-09T14:35:33.803Z","updated_at":"2025-10-28T06:35:27.442Z","avatar_url":"https://github.com/molu8bits.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# squid-filebeat-kibana\nFilebeat module for Squid access logs + Kibana dashboards. ELK 7.8\n\n\u003ch3\u003e TL;DR\u003cp\u003e \u003c/h3\u003e\nCollect your squid access.log with Filebeat, send directly to Elasticsearch.\nGet overview of Squid access log using Kibana dashboard.\n\n![](_images/kibana_dashboard_example.png)\n\n![](_images/squid_downloaded_per_host.png.png)\n\n\n\u003ch4\u003e Elastichsearch and Kibana \u003c/h4\u003e\n1. Elasticsearch and Kibana\n    \u003cp\u003ea.) Install Elasticsearch and Kibana.\n    \u003cp\u003eb.) Configure firewall to allow access from filebeat host to elasticsearch service.\n\n\n\u003ch4\u003e Filebeat + module squid installation \u003c/h4\u003e\n2. Configuration Filebeat (7.x recommended. Older versions may not work)\n   \u003cp\u003ea.) copy filebeat/module/squid into /usr/share/filebeat/module\n   \u003cp\u003eb.) copy filebeat/etc/filebeat/modules.d/squid.yml.disabled into /etc/filebeat/modules.d\n   \u003cp\u003ec.) configure /etc/filebeat/filebeat.yml - reference file placed in /etc/filebeat/filebeat.yml\n        (change  hosts [\"elasticsearch.local\"] in section output.elastichsearch to elastichsarch instance listening from filebeat host\n   \u003cp\u003ed.) enable Filebeat squid module by command \"filebeat modules enable squid\" (or just rename /etc/filebeat/modules.d/squid.yml.disabled to /etc/filename/modules.d/squid.yml\n   \u003cp\u003ee.) Replace /etc/filebeat/fields.yml with filebeat/etc/fields.yml from repo. (Important! - This must be done before run filebeat). This file is a compiled version from 7.3 so the rest of functionality should work. If filebeat with newer definitions is needed then just cut squid section and join into newer field.yml\"\n   \u003cp\u003ef.) restart Filebeat service - \"systemctl restart filebeat\"\n\n\u003ch4\u003eKibana configuration \u003c/h4\u003e\n3. Import Dashboard definition from kibana folder Kibana (*.ndjson file via GUI: Management -\u003e Stack Management -\u003e Kibana -\u003e Saved Objects -\u003e Import) \n\n\u003ch4\u003e Check Dashboard view on Kibana \u003c/h4\u003e\n4. Go to the Dashboard section and find \"[Filebeat Squid] Access log\". Set Time-Range according to expected log entries.\n\n\n\u003ch4\u003e Troubleshooting \u003c/h4\u003e\n\u003cp\u003eElasticsearch needs to know what types should be applied to particular fields during processing logs.\nFor all the modules they are available in /etc/filebeat/fields.yml and this repo contains of modified fields.yml filebeat 7.3 with compiled into mappings required for the module squid.\nIf they are not applied automatically on the Filebeat Index (e.g. index filebeat-* already exist and)\nthen try to remove index, filebeat ingest pipeline and filebeat mapping.\n\n\n```bash\ncat filebeat/etc/squid-fields.yml \u003e\u003e /etc/filebeat/fields.yml\ncurl -XDELETE elasticsearch.local:9200/filebeat-index-name\ncurl -XDELETE elasticsearch.local:9200/_ingest/pipeline/filebeat*squid*\nsystemctl restart filebeat\n```\n\n\u003cp\u003eModule has been tested with latest Elasticsearch/Kibana 7.9 based on docker-compose stack:\nhttps://www.elastic.co/guide/en/elasticsearch/reference/7.9/docker.html\nIn case of trouble such quick clean installation of ELK is recommended to test filebeat configuration.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmolu8bits%2Fsquid-filebeat-kibana","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmolu8bits%2Fsquid-filebeat-kibana","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmolu8bits%2Fsquid-filebeat-kibana/lists"}