{"id":13581929,"url":"https://github.com/crowdsecurity/hub","last_synced_at":"2025-05-15T23:03:23.427Z","repository":{"id":37857019,"uuid":"261387435","full_name":"crowdsecurity/hub","owner":"crowdsecurity","description":"Main repository for crowdsec scenarios/parsers","archived":false,"fork":false,"pushed_at":"2025-05-15T18:03:59.000Z","size":200749,"stargazers_count":192,"open_issues_count":127,"forks_count":170,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-05-15T23:02:31.484Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://hub.crowdsec.net","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/crowdsecurity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-05-05T07:19:59.000Z","updated_at":"2025-05-15T18:04:03.000Z","dependencies_parsed_at":"2023-10-16T09:28:23.904Z","dependency_job_id":"176e6343-9ff1-46c1-b620-de4893be73e9","html_url":"https://github.com/crowdsecurity/hub","commit_stats":{"total_commits":3848,"total_committers":113,"mean_commits":34.05309734513274,"dds":0.1990644490644491,"last_synced_commit":"3ed05eafa2c616a86ffa2457243ddb477144b3ff"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crowdsecurity%2Fhub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crowdsecurity%2Fhub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crowdsecurity%2Fhub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crowdsecurity%2Fhub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/crowdsecurity","download_url":"https://codeload.github.com/crowdsecurity/hub/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254436944,"owners_count":22070946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:02:19.808Z","updated_at":"2025-05-15T23:03:23.347Z","avatar_url":"https://github.com/crowdsecurity.png","language":"Python","funding_links":[],"categories":["Python","Official Resources"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/crowdsecurity/hub/master/assets/crowdsec_hub.svg\" alt=\"CrowdSec\" title=\"CrowdSec\" width=\"400\" height=\"260\"/\u003e\n\u003c/p\u003e\n\u003c/br\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/AlteredCoder/ed74e50c43e3b17bdfc4d93149f23d37/raw/hub_parsers_badge.json\"\u003e\n\u003cimg src=\"https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/AlteredCoder/ed74e50c43e3b17bdfc4d93149f23d37/raw/hub_scenarios_badge.json\"\u003e\n\u003cimg src=\"https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/AlteredCoder/ed74e50c43e3b17bdfc4d93149f23d37/raw/hub_appsec_badge.json\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n:books: \u003ca href=\"https://doc.crowdsec.net\"\u003eDocumentation\u003c/a\u003e\n:diamond_shape_with_a_dot_inside: \u003ca href=\"https://hub.crowdsec.net\"\u003eConfiguration Hub\u003c/a\u003e\n:speech_balloon: \u003ca href=\"https://discourse.crowdsec.net\"\u003eDiscourse (Forum)\u003c/a\u003e\n:speech_balloon: \u003ca href=\"https://discord.gg/crowdsec\"\u003eDiscord (Live Chat)\u003c/a\u003e\n\u003c/p\u003e\n\n\u003e CrowdSec Hub for parsers, enrichers and scenarios.\n\n# Foreword\n\nThis repository stores most of the official parsers/scenarios/collections for crowdsec.\n\nThe repository is not intended for use as-is, but rather as source of truth for the [CrowdSec Hub](https://hub.crowdsec.net/) and `cscli`.\n\nFeel free to use the parsers/scenarios here as a source of inspiration.\n\n# Testing \u0026 Continuous integration\n\n`cscli` provides a `hubtest` sub-command to help contributors to create tests for parsers and scenarios.\n\n## View \u0026 use existing tests\n\n:warning: most of `cscli hubtest` commands are expected to be run from the root directory of the hub. A git clone of this repository is the easier way to work :warning:\n\n\u003e list existing tests\n\n`cscli hubtest list`\n\n\u003e run a specific test\n\n`cscli hubtest run [test-name]`\n\n\u003e show current tests coverage\n\n`cscli hubtest coverage`\n\n## Create your own (parser) test\n\nWe're going to create the CI tests for the dovecot-parser. Before you start :\n\n- you will need some _actual_ logs\n- you'd better know if the service logs on its own or via syslog (we're in the later case here)\n\n1. Create a new test\n\n```bash\n▶ cscli hubtest create dovecot-logs --type syslog\n\n  Test name                   :  dovecot-logs\n  Test path                   :  .../github/hub/.tests/dovecot-logs\n  Log file                    :  .../github/hub/.tests/dovecot-logs/dovecot-logs.log (please fill it with logs)\n  Parser assertion file       :  .../github/hub/.tests/dovecot-logs/parser.assert (please fill it with assertion)\n  Scenario assertion file     :  .../github/hub/.tests/dovecot-logs/parser.assert (please fill it with assertion)\n  Configuration File          :  .../github/hub/.tests/dovecot-logs/config.yaml (please fill it with parsers, scenarios...)\n\n```\n\nWhat is relevant here is that every test is composed of :\n\n- A log file and it's associated type (same `type` as seen in acquis `labels:type`)\n- A configuration specifying which parsers and/or scenarios must be enabled for the test\n- A _ultimately_ list of assertions that must be run against the parsers and/or scenarios output\n\nNote: You can provide the parsers and scenarios you want in your test with `--parsers` and `--scenarios` (you can provide multiple parsers and scenarios)\n\nIf you want to test only a scenario, you can specify (`--ignore-parsers`) or set the `ignore_parsers` to `true` in the config.yaml\n\n2. Configure your test\n\nWe need to edit the test configuration to use the relevant parsers :\n\n```bash\n▶ cat .../github/hub/.tests/dovecot-logs/config.yaml\nparsers:\n- crowdsecurity/syslog-logs\n- crowdsecurity/dovecot-logs\nscenarios:\npostoverflows:\n- \"\"\nlog_file: dovecot-logs.log\nlog_type: syslog\n\n```\n\n_note: the order doesn't matter. If the parser name is in the form `author/parser` it's from the hub, but relative paths are allowed for non-versioned parsers_\n\nNow we need to dump some actual logs into the test's log file :\n\n```bash\n▶ cat \u003e .tests/dovecot-logs/dovecot-logs.log\nJan 28 10:16:13 dovecot-box dovecot[7508]: imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=\u003ctoto@toto.com\u003e, method=PLAIN, rip=4.4.4.4, lip=7.7.7.7, TLS, session=\u003c3650VvK5bdIaW-iK\u003e\nSep 8 07:16:29 canyon dovecot: auth-worker(24058): pam(toto,1.1.1.1,\u003cyoupi\u003e): pam_authenticate() failed: Authentication failure (password mismatch?)\nSep 8 07:46:51 canyon dovecot: auth-worker(24544): pam(toto,1.1.1.1): unknown user\n\n```\n\n3. Run the test for the first time\n\nNow that we have config \u0026 logs, let's run it for the first time :\n\n```bash\n▶ cscli hubtest run dovecot-logs\n\nINFO[27-09-2021 06:13:59 PM] Running test 'dovecot-logs'\nINFO[27-09-2021 06:13:59 PM] parser 'crowdsecurity/dovecot-logs' installed successfully in runtime environment\nINFO[27-09-2021 06:13:59 PM] parser 'crowdsecurity/syslog-logs' installed successfully in runtime environment\nWARN[27-09-2021 06:14:02 PM] Assert file '/home/bui/github/hub/.tests/dovecot-logs/parser.assert' is empty, generating assertion:\n\nresults[\"s00-raw\"][\"crowdsecurity/syslog-logs\"][0].Success == true\n...\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Success == true\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"pid\"] == \"7508\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"timestamp\"] == \"Jan 28 10:16:13\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"dovecot_login_result\"] == \"Disconnected (auth failed, 1 attempts in 6 secs)\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"message\"] == \"imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=\u003ctoto@toto.com\u003e, method=PLAIN, rip=4.4.4.4, lip=7.7.7.7, TLS, session=\u003c3650VvK5bdIaW-iK\u003e\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"logsource\"] == \"syslog\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"program\"] == \"dovecot\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"protocol\"] == \"imap\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"dovecot_local_ip\"] == \"7.7.7.7\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"dovecot_remote_ip\"] == \"4.4.4.4\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"dovecot_user\"] == \"toto@toto.com\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Meta[\"datasource_path\"] == \"dovecot-logs.log\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Meta[\"datasource_type\"] == \"file\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Meta[\"log_type\"] == \"dovecot_logs\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Meta[\"source_ip\"] == \"4.4.4.4\"\n...\n```\n\nAs our `parser.assert` is empty, the tool is generating some \"suggested\" asserts for us.\nYour careful eye will keep only the ones relevant to the parser you're testing :\n\n```bash\n▶ cat \u003e .tests/dovecot-logs/parser.assert\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Success == true\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"pid\"] == \"7508\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"timestamp\"] == \"Jan 28 10:16:13\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"dovecot_login_result\"] == \"Disconnected (auth failed, 1 attempts in 6 secs)\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"message\"] == \"imap-login: Disconnected (auth failed, 1 attempts in 6 secs): user=\u003ctoto@toto.com\u003e, method=PLAIN, rip=4.4.4.4, lip=7.7.7.7, TLS, session=\u003c3650VvK5bdIaW-iK\u003e\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"logsource\"] == \"syslog\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"program\"] == \"dovecot\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"protocol\"] == \"imap\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"dovecot_local_ip\"] == \"7.7.7.7\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"dovecot_remote_ip\"] == \"4.4.4.4\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Parsed[\"dovecot_user\"] == \"toto@toto.com\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Meta[\"datasource_path\"] == \"dovecot-logs.log\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Meta[\"datasource_type\"] == \"file\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Meta[\"log_type\"] == \"dovecot_logs\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][0].Evt.Meta[\"source_ip\"] == \"4.4.4.4\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][1].Success == true\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][1].Evt.Parsed[\"dovecot_login_result\"] == \"Authentication failure (password mismatch?)\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][1].Evt.Parsed[\"logsource\"] == \"syslog\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][1].Evt.Parsed[\"message\"] == \"auth-worker(24058): pam(toto,1.1.1.1,\u003cyoupi\u003e): pam_authenticate() failed: Authentication failure (password mismatch?)\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][1].Evt.Parsed[\"program\"] == \"dovecot\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][1].Evt.Parsed[\"timestamp\"] == \"Sep 8 07:16:29\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][1].Evt.Parsed[\"dovecot_remote_ip\"] == \"1.1.1.1\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][1].Evt.Parsed[\"dovecot_user\"] == \"toto\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][1].Evt.Meta[\"datasource_path\"] == \"dovecot-logs.log\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][1].Evt.Meta[\"datasource_type\"] == \"file\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][1].Evt.Meta[\"log_type\"] == \"dovecot_logs\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][1].Evt.Meta[\"source_ip\"] == \"1.1.1.1\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Success == true\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Evt.Parsed[\"dovecot_login_result\"] == \"unknown user\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Evt.Parsed[\"program\"] == \"dovecot\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Evt.Parsed[\"dovecot_remote_ip\"] == \"1.1.1.1\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Evt.Parsed[\"dovecot_user\"] == \"toto\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Evt.Parsed[\"logsource\"] == \"syslog\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Evt.Parsed[\"message\"] == \"auth-worker(24544): pam(toto,1.1.1.1): unknown user\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Evt.Parsed[\"timestamp\"] == \"Sep 8 07:46:51\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Evt.Meta[\"log_type\"] == \"dovecot_logs\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Evt.Meta[\"source_ip\"] == \"1.1.1.1\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Evt.Meta[\"datasource_path\"] == \"dovecot-logs.log\"\nresults[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Evt.Meta[\"datasource_type\"] == \"file\"\n```\n\n4. Test your newly crafted test\n\n```bash\n▶ cscli hubtest run dovecot-logs\nINFO[27-09-2021 06:19:33 PM] Running test 'dovecot-logs'\nINFO[27-09-2021 06:19:33 PM] parser 'crowdsecurity/syslog-logs' installed successfully in runtime environment\nINFO[27-09-2021 06:19:33 PM] parser 'crowdsecurity/dovecot-logs' installed successfully in runtime environment\nTest 'dovecot-logs' passed successfully (39 assertions) 🟩\n```\n\nAnd be amazed.\n\n## Debug your own (parser) test\n\nThings went wrong ? Don't panic\n\nWhen working on a test, you can as well pass expressions directly to `hubtest` command and see the results :\n\n```bash\n▶ cscli hubtest  eval dovecot-logs -e 'results[\"s01-parse\"][\"crowdsecurity/dovecot-logs\"][2].Evt.Parsed'\ndovecot_login_result: unknown user\ndovecot_remote_ip: 1.1.1.1\ndovecot_user: toto\nfacility: \"\"\nlogsource: syslog\nmessage: 'auth-worker(24544): pam(toto,1.1.1.1): unknown user'\npid: \"\"\npriority: \"\"\nprogram: dovecot\ntimestamp: Sep 8 07:46:51\ntimestamp8601: \"\"\n```\n\n## Contributing\n\nWe gratefully accept PR to enrich the content of Hub. Follow the steps on our documentation website: [https://docs.crowdsec.net/docs/next/contributing/contributing_hub/](https://docs.crowdsec.net/docs/next/contributing/contributing_hub/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrowdsecurity%2Fhub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcrowdsecurity%2Fhub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrowdsecurity%2Fhub/lists"}