{"id":20317917,"url":"https://github.com/crowdsecurity/blocklist-efficiency-tester","last_synced_at":"2026-05-11T17:32:48.806Z","repository":{"id":255926187,"uuid":"852796145","full_name":"crowdsecurity/blocklist-efficiency-tester","owner":"crowdsecurity","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-12T10:37:14.000Z","size":93,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-01T16:39:16.032Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/crowdsecurity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-05T12:48:26.000Z","updated_at":"2025-02-12T10:37:18.000Z","dependencies_parsed_at":"2024-11-14T18:51:24.313Z","dependency_job_id":null,"html_url":"https://github.com/crowdsecurity/blocklist-efficiency-tester","commit_stats":null,"previous_names":["crowdsecurity/blocklist-efficiency-tester"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crowdsecurity%2Fblocklist-efficiency-tester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crowdsecurity%2Fblocklist-efficiency-tester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crowdsecurity%2Fblocklist-efficiency-tester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crowdsecurity%2Fblocklist-efficiency-tester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/crowdsecurity","download_url":"https://codeload.github.com/crowdsecurity/blocklist-efficiency-tester/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241818899,"owners_count":20025212,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T18:37:17.633Z","updated_at":"2026-05-11T17:32:48.800Z","avatar_url":"https://github.com/crowdsecurity.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"![CrowdSec Logo](https://raw.githubusercontent.com/crowdsecurity/blocklist-efficiency-tester/main/crowdsec_logo.png)\n\n# CrowdSec Blocklist efficiency tester\n\u003e Fast evaluation of ingress traffic mass-attacks.\u003cbr\u003e\n\u003e Demonstrating the proactive value your [Blocklist as a Service integration](https://doc.crowdsec.net/u/integrations/intro)  \n\u003e We recommend a [Raw IP list integration](https://doc.crowdsec.net/u/integrations/rawiplist).\n\n## Requirements\n\nThe crowdsec-efficiency-tester.sh bash script requires:\n  - **CrowdSec Blocklist as a Service Endpoint** ([Raw IP list mode](https://doc.crowdsec.net/u/integrations/rawiplist))\n- A **log file from incoming traffic** or at least a file containing IPs that hit your server in the past 24-48hours\n- The curl command must be available on your system (to download the list)\n\n## Usage\n\n### ENV SETUP: Optionnal\n\nIf you plan to test multiple log files against the content of your Blocklist Integration content it's convenient to create a `.env` file.  \n\nCopy `.env.example`:\n```bash  \ncp .env.example .env\n```\n\nThen uncomment and replace the following values with your endpoint's credentials\n```bash\nBLOCKLIST_URL=\u003cintegration_url\u003e\nBLOCKLIST_USERNAME=your_username\nBLOCKLIST_PASSWORD=your_password\n```\n\n\u003e You can specify only BLOCKLIST_URL if using the `https://username:password@url` format  \n\u003e Hence you can pull from anywhere\n\n\n### Running the evaluation\n\n#### Getting your credentials:\n1. Go to the [CrowdSec Console](https://app.crowdsec.net/)\n2. Navigate to **Integrations** \u003e **Add Integration**\n3. Select **\"Raw IP list\"** integration\n4. Copy your **integration ID**, **username**, and **password**\n\n#### Running with .env file:\n\n```bash\n./crowdsec-efficiency-tester.sh /path/to/log/file.log\n```\n\n#### Overriding your .env you can run:\n\n```bash\nBLOCKLIST_URL=https://admin.api.crowdsec.net/v1/integrations/YOUR_INTEGRATION_ID/content \\\nBLOCKLIST_USERNAME=your_username \\\nBLOCKLIST_PASSWORD=your_password \\\n./crowdsec-efficiency-tester.sh /path/to/log/file.log\n```\n\n#### Cache and Performance:\n- Blocklist content is **cached for 10 minutes** to improve performance\n- Use the `-f` flag to force a fresh download: `./crowdsec-efficiency-tester.sh /path/to/log/file.log -f`\n- Cache files: `.cache` (metadata) and `latestBlocklistContent.ips` (IP list)\n\n### Log files that you can use\n\u003e ℹ️ Script currently supports logs formats where the **IP address** is the **first element** in the log line.\u003cbr\u003e\n\u003e Example of logs you might want to evaluate:\n- Auth logs\n- NGINX logs\n- HAProxy logs\n- AWS CloudFront access logs\n- Kubernetes ingress controller logs\n- FTP server logs\n- Mail server (Postfix, Exim) logs\n- ...\n\n### Alternate commands\n\u003e You can run the script directly from the repo\n\u003e You'll be prompted to enter the path to your file and API key\n``` \n/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/crowdsecurity/blocklist-efficiency-tester/main/crowdsec-efficiency-tester.sh)\"\n```\n\n### Example Output\n```\n✨✨✨ CrowdSec efficiency tester ✨✨✨\n\nExtracting and counting IP addresses from logs... ✅\nDownloading blocklist... ✅\nAnalyzing IPs against the blocklist... ✅\n\n\n=== Summary ===\nBlocklist hits (IPs)       : 1.85% (4/216)\nBlocklist hits (Requests)  : 73.78% (4890/6627)\n\nTOP 10 IPs in the blocklist:\n------------------------------------------------\nIP Address           | Count\n-------------------- | -----\n212.102.57.94        |  4853\n207.102.138.19       |  34  \n185.241.208.115      |  2   \n142.44.160.96        |  1   \n                     |   \n```\n\n## Troubleshooting\n- This script can take a few minutes. Average 1-2 minutes per 20k lines of log\n- If the log file is not found, the script will not work. Ensure you provide a valid path.\n- If the API key is incorrect the blocklist won't be downloaded. Note that API keys may expires depending on creation preferences\n- IPs in your log files must be ingress source IPs (be sure not to have CDN IPs)\n\n## More info about CrowdSec\n- [Blocklists](https://www.crowdsec.net/blocklists)\n- [Security Engine](https://www.crowdsec.net/security-engine)\n- [CTI](https://www.crowdsec.net/cyber-threat-intelligence)\n- [Integrations](https://www.crowdsec.net/integrations)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrowdsecurity%2Fblocklist-efficiency-tester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcrowdsecurity%2Fblocklist-efficiency-tester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrowdsecurity%2Fblocklist-efficiency-tester/lists"}