{"id":37086058,"url":"https://github.com/spacepatcher/firehol-ip-aggregator","last_synced_at":"2026-01-14T10:35:15.932Z","repository":{"id":57429230,"uuid":"105667433","full_name":"spacepatcher/firehol-ip-aggregator","owner":"spacepatcher","description":"App for storing FireHOL feeds (https://github.com/firehol/blocklist-ipsets) with a history of adding and removing each address","archived":false,"fork":false,"pushed_at":"2023-01-04T13:31:23.000Z","size":158,"stargazers_count":33,"open_issues_count":2,"forks_count":10,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-09-27T23:15:22.785Z","etag":null,"topics":["api","firehol","mongodb","python3","threat-intelligence"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spacepatcher.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-03T15:19:23.000Z","updated_at":"2024-12-15T11:56:35.000Z","dependencies_parsed_at":"2023-02-02T14:15:57.906Z","dependency_job_id":null,"html_url":"https://github.com/spacepatcher/firehol-ip-aggregator","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/spacepatcher/firehol-ip-aggregator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacepatcher%2Ffirehol-ip-aggregator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacepatcher%2Ffirehol-ip-aggregator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacepatcher%2Ffirehol-ip-aggregator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacepatcher%2Ffirehol-ip-aggregator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spacepatcher","download_url":"https://codeload.github.com/spacepatcher/firehol-ip-aggregator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacepatcher%2Ffirehol-ip-aggregator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28417662,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T10:25:19.714Z","status":"ssl_error","status_checked_at":"2026-01-14T10:22:49.371Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","firehol","mongodb","python3","threat-intelligence"],"created_at":"2026-01-14T10:35:15.223Z","updated_at":"2026-01-14T10:35:15.926Z","avatar_url":"https://github.com/spacepatcher.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Aggregator of FireHOL IP lists\r\n\r\nApplication for keeping feeds from \u003ca href=\"https://github.com/firehol/blocklist-ipsets\" target=\"_blank\"\u003eblocklist-ipsets\u003c/a\u003e (only *.netset and *.ipset files are aggregated) with including historical changes. HTTP-based API service for search requests developed.\r\n\r\nSome features of keeping and processing data:\r\n* New data is written to existing data with `last_added` field update\r\n* Data deleted from reputation feed is not deleted from the application database\r\n* Field `timeline` is based on events of adding and removing item from reputation list\r\n\r\n### Start application\r\n\r\nTo start the collection module and the HTTP-based API service, just type:\r\n```\r\ndocker-compose up\r\n```\r\nThe collection module will start in container `sync`.\r\n\r\nBy default, the HTTP-based API service running in container `api` and is available on port 8000.\r\n\r\n### API\r\n\r\nThere are several API-functions for search requests:\r\n\r\n* POST `/search` - retrieve all information about requested IP or CIDR format objects\r\n* GET `/search/ip` - retrieve all information about single requested IP or CIDR format object\r\n\r\nAccess to the API documentation can be obtained by requesting any unspecified URL.\r\n\r\n### Example usage\r\n\r\nA simple python3 package `fiaclient` is designed as a client for FireHOL-IP-Aggregator API.\r\n\r\nInstall the package with pip:\r\n```\r\npip install fiaclient\r\n```\r\n\r\nGet a client object in python3 console:\r\n```\r\nfrom fiaclient import fiaclient\r\nclient = fiaclient.FIAClient(fia_url=\"http://127.0.0.1:8000/\")\r\n```\r\n\r\nTo get information about `fiaclient` package visit https://github.com/spacepatcher/FireHOL-IP-Aggregator/blob/develop/fiaclient/README.md.\r\n\r\nThe application is able to get search requests in IP or CIDR format, also in mixed list of both data types. To search, run the command in python3 console:\r\n```\r\nresult = client.search(payload=[\"149.255.60.136\"])\r\n```\r\n\r\nAlso you can generate search requests using cURL:\r\n* For HTTP POST search API function:\r\n```\r\ncurl -X POST --data '8.8.8.8,1.1.1.1' -H 'Content-Type: text/html' localhost:8000/search\r\n```\r\n* For HTTP GET search API function:\r\n```\r\ncurl -X GET localhost:8000/search/ip?v=8.8.8.8\r\n```\r\n\r\nHere is an example of the result of the requested payload:\r\n```\r\n{\r\n  \"request_time\": \"2019-01-10T15:39:03.927874+03:00\",\r\n  \"records_count\": 1601585,\r\n  \"requested_count\": 1,\r\n  \"blacklisted_count\": 1,\r\n  \"currently_blacklisted_count\": 1,\r\n  \"results\": [\r\n    {\r\n      \"ip\": \"5.153.47.228\",\r\n      \"categories\": [\r\n        \"malware\"\r\n      ],\r\n      \"first_seen\": \"2019-01-10T12:37:09.164000\",\r\n      \"last_added\": \"2019-01-10T12:37:09.164000\",\r\n      \"hits_count\": 1,\r\n      \"currently_blacklisted\": true,\r\n      \"hits\": [\r\n        {\r\n          \"feed_name\": \"vxvault\",\r\n          \"category\": \"malware\",\r\n          \"maintainer\": \"VxVault\",\r\n          \"maintainer_url\": \"http://vxvault.net\",\r\n          \"list_source_url\": \"http://vxvault.net/ViriList.php?s=0\u0026m=100\",\r\n          \"source_file_date\": \"Thu Jan 10 03:24:39 UTC 2019\",\r\n          \"entries\": \"76 unique IPs\",\r\n          \"first_seen\": \"2019-01-10T12:37:09.164000\",\r\n          \"last_added\": \"2019-01-10T12:37:09.164000\",\r\n          \"last_removed\": null,\r\n          \"current_status\": \"present\",\r\n          \"timeline\": [\r\n            {\r\n              \"added\": \"2019-01-10T12:37:09.164000\",\r\n              \"removed\": null\r\n            }\r\n          ]\r\n        }\r\n      ]\r\n    }\r\n  ]\r\n}\r\n```\r\n\r\nIf the observable is not found in the application database, the response will looks like this:\r\n```\r\n{\r\n  \"request_time\": \"2019-01-10T15:30:03.813983+03:00\",\r\n  \"records_count\": 1601585,\r\n  \"requested_count\": 1,\r\n  \"blacklisted_count\": 0,\r\n  \"currently_blacklisted_count\": 0,\r\n  \"results\": []\r\n}\r\n```\r\n\r\n### Important files\r\n\r\n* `docker-persistent/conf/app.conf` - the main application configuration file\r\n* `docker-persistent/app/log/run.log` - the main log file\r\n\r\n### Application configuration\r\n\r\nThe most important configuration parameters from `docker-persistent/conf/app.conf` are listed in the table below.\r\n\r\n| Parameter | Description |\r\n| ------ | ------ |\r\n| `\"unique_ips_limit\"` | Defines the limit of the number of unique IP addresses in FireHOL feeds that will be aggregated (the goal is to filter out huge feeds) |\r\n| `\"sync_period_h\"` | Defines time period for syncing with FireHOL IP list repository |\r\n| `\"firehol_ipsets_git\"` | Defines FireHOL IP lists repository url |\r\n\r\nAlso it's possible to change count of workers that process queries to API in `docker/Dockerfile.api` by changing `--workers` argument value in `ENTRYPOINT`.\r\n```\r\nENTRYPOINT [\"gunicorn\", \"--bind=0.0.0.0:8000\", \"--workers=4\", \"--timeout\", \"3600\", \"api:__hug_wsgi__\"]\r\n```\r\n\r\nTo apply configuration changes you should rebuild containers:\r\n```\r\ndocker-compose down\r\ndocker-compose up --build\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspacepatcher%2Ffirehol-ip-aggregator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspacepatcher%2Ffirehol-ip-aggregator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspacepatcher%2Ffirehol-ip-aggregator/lists"}