{"id":13454001,"url":"https://github.com/rapid7/recog","last_synced_at":"2025-05-13T21:12:09.052Z","repository":{"id":12785969,"uuid":"15459617","full_name":"rapid7/recog","owner":"rapid7","description":"Pattern recognition for hosts, services, and content","archived":false,"fork":false,"pushed_at":"2025-04-07T22:11:40.000Z","size":7282,"stargazers_count":718,"open_issues_count":17,"forks_count":205,"subscribers_count":198,"default_branch":"main","last_synced_at":"2025-04-28T17:21:24.623Z","etag":null,"topics":["fingerprinting","hacktoberfest","regex","ruby","xml"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rapid7.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2013-12-26T20:16:05.000Z","updated_at":"2025-04-17T05:53:39.000Z","dependencies_parsed_at":"2023-10-02T16:04:45.648Z","dependency_job_id":"bbe158a6-5cf5-4c48-bcc2-20c82ba21330","html_url":"https://github.com/rapid7/recog","commit_stats":{"total_commits":1710,"total_committers":74,"mean_commits":23.10810810810811,"dds":0.6128654970760234,"last_synced_commit":"0ffcd3e0aa9041bc9d06db73e957d427d372d1d3"},"previous_names":[],"tags_count":150,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapid7%2Frecog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapid7%2Frecog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapid7%2Frecog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapid7%2Frecog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rapid7","download_url":"https://codeload.github.com/rapid7/recog/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254029008,"owners_count":22002284,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fingerprinting","hacktoberfest","regex","ruby","xml"],"created_at":"2024-07-31T08:00:50.216Z","updated_at":"2025-05-13T21:12:04.039Z","avatar_url":"https://github.com/rapid7.png","language":"Ruby","readme":"# Recog: A Recognition Framework\n\n[![CI Workflow](https://github.com/rapid7/recog/actions/workflows/ci.yml/badge.svg)](https://github.com/rapid7/recog/actions/workflows/ci.yml)\n[![Verify Workflow](https://github.com/rapid7/recog/actions/workflows/verify.yml/badge.svg)](https://github.com/rapid7/recog/actions/workflows/verify.yml)\n\nRecog is a framework for identifying products, services, operating systems, and hardware by matching fingerprints against data returned from various network probes. Recog makes it simple to extract useful information from web server banners, snmp system description fields, and a whole lot more.\n\nRecog is open source, please see the [LICENSE](LICENSE) file for more information.\n\n## Table of Contents\n\n1. [Repository split](#repository-split)\n    1. [Default branch rename](#default-branch-rename)\n1. [Recog library language implementations](#recog-library-language-implementations)\n    1. [Feature parity](#feature-parity)\n1. [Installation](#installation)\n1. [Maturity](#maturity)\n1. [Fingerprints](#fingerprints)\n1. [Testing matches](#testing-matches)\n    1. [ftp_banners](#ftp_banners)\n    1. [http_cookies](#http_cookies)\n    1. [http_servers](#http_servers)\n    1. [favicons](#favicons)\n    1. [http_wwwauth](#http_wwwauth)\n    1. [tls_jarm](#tls_jarm)\n1. [Contributing](#contributing)\n\n## Repository split\n\nOn March 31, 2022, the Recog content - XML fingerprint files and utilities - were split from the Recog framework library implementation. The original [Recog](https://github.com/rapid7/recog) repository now contains the Recog content and the [Recog-Ruby](https://github.com/rapid7/recog-ruby) repository contains the Ruby language implementation. The Recog content is included in Recog-Ruby as a git submodule and is nested under the `recog` directory. All post-split Recog gem versions equal or greater than 3.0.0 will: 1. contain the XML fingerprint directory under the `recog` directory, and 2. only include the `recog_match` tool since the other tools are focused on fingerprint management.\n\n[^back to top](#recog-a-recognition-framework)\n\n### Default branch rename\n\nAlong with the repository split the default branch was renamed from `master` to `main`. Any clones created prior to these changes will have to be manually updated in your local environment. If you have a fork, navigate to your fork's settings and follow the instructions on [renaming a branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/renaming-a-branch) to change the default branch to `main`.\n\n```\ngit branch -m master main\ngit fetch origin\ngit branch -u origin/main main\ngit remote set-head origin -a\n```\n\nOptionally, run the following command to remove tracking references to the old branch name.\n```\n# dry-run to confirm stale references that will be deleted before proceeding\ngit remote prune origin --dry-run\ngit remote prune origin\n```\n\nIf you previously used the upstream tracking branch `upstream-master` run the following commands to remove the old branch and create a new upstream tracking branch.\n```\ngit branch -d upstream-master\ngit checkout -b upstream-main --track upstream/main\n```\n\nOptionally, run the following command to remove the tracking references to the old upstream branch name.\n```\n# dry-run to confirm stale references that will be deleted before proceeding\ngit remote prune upstream --dry-run\ngit remote prune upstream\n```\n\n[^back to top](#recog-a-recognition-framework)\n\n## Recog library language implementations\n\n* Ruby: [rapid7/recog-ruby](https://github.com/rapid7/recog-ruby)\n* Java: [rapid7/recog-java](https://github.com/rapid7/recog-java)\n* Go: [runZeroInc/recog-go](https://github.com/runZeroInc/recog-go)\n\n[^back to top](#recog-a-recognition-framework)\n\n### Feature parity\n\n| Feature :sparkles:                          | rapid7/recog-ruby  | rapid7/recog-java  | runZeroInc/recog-go |\n|---------------------------------------------|:------------------:|:------------------:|:-------------------:|\n| Fingerprint verification CLI tool           | :white_check_mark: | :white_check_mark: | :white_check_mark:  |\n| Fingerprint match CLI tool                  | :white_check_mark: |                    | :white_check_mark:  |\n| Supports base64 encoded examples            | :white_check_mark: | :white_check_mark: | :white_check_mark:  |\n| Supports filesystem-based external examples | :white_check_mark: | :white_check_mark: | :white_check_mark:  |\n| Fingerprint match CPE param interpolation   | :white_check_mark: | :white_check_mark: | :white_check_mark:  |\n\n[^back to top](#recog-a-recognition-framework)\n\n## Installation\n\nRecog consists of both XML fingerprint files and an assortment of code, mostly in Ruby, that makes it easy to develop, test, and use the contained fingerprints. In order to use the included ruby code, a recent version of Ruby (2.31+) is required, along with Rubygems and the `bundler` gem. Once these dependencies are in place, use the following commands to grab the latest source code and install any additional dependencies.\n\n```shell\n$ git clone git@github.com:rapid7/recog.git\n$ cd recog\n$ bundle install\n```\n\n[^back to top](#recog-a-recognition-framework)\n\n## Maturity\n\nPlease note that while the XML fingerprints themselves are quite stable and well-tested, the Ruby codebase is still fairly new and subject to change quickly. Please contact us (research[at]rapid7.com) before leveraging the Recog code within any production projects.\n\n[^back to top](#recog-a-recognition-framework)\n\n## Fingerprints\n\nThe fingerprints within Recog are stored in XML files, each of which is designed to match a specific protocol response string or field. For example, the file [ssh_banners.xml](xml/ssh_banners.xml) can determine the os, vendor, and sometimes hardware product by matching the initial SSH daemon banner string.\n\nA fingerprint file consists of an XML document like the following:\n\n```xml\n\u003cfingerprints matches=\"ssh.banner\"\u003e\n  \u003cfingerprint pattern=\"^RomSShell_([\\d\\.]+)$\"\u003e\n    \u003cdescription\u003eAllegro RomSShell SSH\u003c/description\u003e\n    \u003cexample service.version=\"4.62\"\u003eRomSShell_4.62\u003c/example\u003e\n    \u003cparam pos=\"0\" name=\"service.vendor\" value=\"Allegro\"/\u003e\n    \u003cparam pos=\"0\" name=\"service.product\" value=\"RomSShell\"/\u003e\n    \u003cparam pos=\"1\" name=\"service.version\"/\u003e\n  \u003c/fingerprint\u003e\n\u003c/fingerprints\u003e\n```\n\nThe first line should always consist of the XML version declaration. The first element should always be a `fingerprints` block with a `matches` attribute indicating what data this fingerprint file is supposed to match. The `matches` attribute is normally in the form of `protocol.field`.\n\nInside of the `fingerprints` element there should be one or more `fingerprint` elements. Every `fingerprint` must contain a `pattern` attribute, which contains the regular expression to be used to match against the data.  An optional `flags` attribute can be specified to control how the regular expression is to be interpreted.  See [the Recog documentation for `FLAG_MAP`](http://www.rubydoc.info/gems/recog/Recog/Fingerprint/RegexpFactory#FLAG_MAP-constant) for more information.\n\nInside of the fingerprint, a `description` element should contain a human-readable string describing this fingerprint.\n\nAt least one `example` element should be present, however multiple `example` elements are preferred.  These elements are used as part of the test coverage present in `rspec` which validates that the provided data matches the specified regular expression.  Additionally, if the fingerprint is using the `param` elements to extract field values from the data (described next), you can add these expected extractions as attributes for the `example` elements.  In the example above, this:\n\n```xml\n\u003cexample service.version=\"4.62\"\u003eRomSShell_4.62\u003c/example\u003e\n```\n\ntests that `RomSShell_4.62` matches the provided regular expression and that the value of `service.version` is 4.62.\n\nThe `example` string can be base64 encoded to permit the use of unprintable characters.  To signal this to Recog an `_encoding` attribute with the value of `base64` is added to the `example` element.  Based64 encoded text that is longer than 80 characters may be wrapped with newlines as shown below to aid in readability.\n\n```xml\n\u003cexample _encoding=\"base64\"\u003e\n  dGllczGEAAAAlQQWMS4yLjg0MC4xMTM1NTYuMS40LjgwMAQuZGF0YS5yZW1vdmVkLjCEAAAAK\n  AQdZG9tYWluQ29udHJvbGxlckZ1bmN0aW9uYWxpdHkxhAAAAAMEATc=\n\u003c/example\u003e\n```\n\nAdditionally, examples can be placed in a directory with the same base name as the XML file, in the same directory as the XML file:\n\n```\nxml/services.xml\nxml/services/file1\nxml/services/file2\n...\n```\n\nThey can then be loaded using the `_filename` attribute:\n\n```xml\n\u003cexample _filename=\"file1\"/\u003e\n```\n\nThis is useful for long examples.\n\nThe `param` elements contain a `pos` attribute, which indicates what capture field\nfrom the `pattern` should be extracted, or `0` for a static string. The `name` attribute\nis the key that will be reported in the case of a successful match and the `value`\nwill either be a static string for `pos` values of `0` or missing and taken from the\ncaptured field.\n\nThe `value` attribute supports interpolation of data from other fields. This is\noften useful when capturing the value for `hw.product` via regex and re-using this\nvalue in `os.product`.\n\nHere is an example from`http_servers.xml` where `hw.product` is captured and reused.\n\n```xml\n  \u003cfingerprint pattern=\"^Eltex (TAU-\\d+[A-Z]*(?:\\.IP)?)$\"\u003e\n    \u003cdescription\u003eEltex TAU model VoIP gateway\u003c/description\u003e\n    \u003cexample hw.product=\"TAU-72\"\u003eEltex TAU-72\u003c/example\u003e\n    \u003cexample hw.product=\"TAU-1.IP\"\u003eEltex TAU-1.IP\u003c/example\u003e\n    \u003cparam pos=\"0\" name=\"os.vendor\" value=\"Eltex\"/\u003e\n    \u003cparam pos=\"0\" name=\"os.product\" value=\"{hw.product} Firmware\"/\u003e\n    \u003cparam pos=\"0\" name=\"os.device\" value=\"VoIP Gateway\"/\u003e\n    \u003cparam pos=\"0\" name=\"hw.vendor\" value=\"Eltex\"/\u003e\n    \u003cparam pos=\"1\" name=\"hw.product\"/\u003e\n    \u003cparam pos=\"0\" name=\"hw.device\" value=\"VoIP Gateway\"/\u003e\n  \u003c/fingerprint\u003e\n```\n\nThere is special handling for temporary attributes that have a name starting with\n`_tmp.`. These attributes can be used for interpolation but are not emitted in the\noutput. This is useful when a particular product name is inconsistent in various\nbanners, vendor marketing, or with NIST values when trying to generate CPEs. In\nthese cases the useful parts of the banner can be extracted and a new value\ncrafted without cluttering the data emitted by a match.\n\n```xml\n\u003cfingerprint pattern=\"^foo baz switchThing-(\\d{4})$\"\u003e\n  \u003cdescription\u003eNetCorp NX series switches\u003c/description\u003e\n  \u003cexample hw.product=\"NX8200\"\u003efoo baz switchThing-8200\u003c/example\u003e\n  \u003cparam pos=\"0\" name=\"hw.vendor\" value=\"NetCorp\"/\u003e\n  \u003cparam pos=\"0\" name=\"hw.product\" value=\"NX{_tmp.001}\"/\u003e\n  \u003cparam pos=\"2\" name=\"_tmp.001\"/\u003e\n\u003c/fingerprint\u003e\n```\n\nThese temporary attributes are not tracked in the `identifiers/fields.txt`.\n\n[^back to top](#recog-a-recognition-framework)\n\n### Testing matches\n\nThe following examples make use of `bin/recog_match`, a simple Ruby command line tool that uses Recog's fingerprint data. Pre-processing is generally required before running Recog, i.e. extracting HTTP header values, etc. All fingerprint data can be found in `xml/*.xml`.\n\n#### ftp_banners\n\nFingerprint FTP servers based on the server's banner response after connecting:\n\n```\n# Example plaintext input\necho -n '220 Microsoft FTP Service' | bin/recog_match xml/ftp_banners.xml -\n\n# Example command using nmap\nnmap -sV -script=banner -p 21 192.168.123.13 | grep --color=never '_banner' | cut -d: -f2- | bin/recog_match xml/ftp_banners.xml -\n\n# Example output\nMATCH: {\"matched\"=\u003e\"Microsoft FTP Server on Windows XP, 2003 or later without version\", \"service.vendor\"=\u003e\"Microsoft\", \"service.product\"=\u003e\"IIS\", \"service.family\"=\u003e\"IIS\", \"service.cpe23\"=\u003e\"cpe:/a:microsoft:iis:-\", \"os.vendor\"=\u003e\"Microsoft\", \"os.family\"=\u003e\"Windows\", \"os.product\"=\u003e\"Windows\", \"os.cpe23\"=\u003e\"cpe:/o:microsoft:windows:-\", \"host.name\"=\u003e\"220\", \"service.protocol\"=\u003e\"ftp\", \"fingerprint_db\"=\u003e\"ftp.banner\", \"data\"=\u003e\"220 Microsoft FTP Service\"}\n```\n\n#### http_cookies\n\nUsing the HTTP `Set-Cookie` header value to fingerprint an HTTP server:\n\n```\n# Example plaintext input\necho 'laravel_session=eyJ...etc..etc...%3D; expires=Mon, 13-Mar-2023 16:48:58 GMT; Max-Age=7200; path=/; httponly; samesite=lax' | bin/recog_match xml/http_cookies.xml -\n\n# Example cURL command\ncurl --silent -I http://localhost:9001 | grep --color=never -i '^Set-Cookie:' | cut -d: -f2- | bin/recog_match xml/http_cookies.xml -\n\n# Example output\nMATCH: {\"matched\"=\u003e\"Laravel PHP web application framework\", \"service.vendor\"=\u003e\"Laravel\", \"service.product\"=\u003e\"Laravel\", \"service.cpe23\"=\u003e\"cpe:/a:laravel:laravel:-\", \"service.protocol\"=\u003e\"http\", \"fingerprint_db\"=\u003e\"http_header.cookie\", \"data\"=\u003e\"laravel_session=eyJ...etc..etc...%3D; expires=Mon, 13-Mar-2023 16:48:58 GMT; Max-Age=7200; path=/; httponly; samesite=lax\"}\n```\n\n#### http_servers\n\nUsing the HTTP `Server` header value to fingerprint an HTTP server:\n\n```\n# Example plaintext input\necho -n 'Apache/2.4.38 (Debian)' | bin/recog_match xml/http_servers.xml -\n\n# Example cURL command\ncurl --silent -I http://localhost:9001 | grep --color=never -i '^Server:' | cut -d: -f2- | bin/recog_match xml/http_servers.xml -\n\n# Example output\nMATCH: {\"matched\"=\u003e\"Apache\", \"service.vendor\"=\u003e\"Apache\", \"service.product\"=\u003e\"HTTPD\", \"service.family\"=\u003e\"Apache\", \"service.version\"=\u003e\"2.4.38\", \"service.cpe23\"=\u003e\"cpe:/a:apache:http_server:2.4.38\", \"apache.info\"=\u003e\"(Debian)\", \"service.protocol\"=\u003e\"http\", \"fingerprint_db\"=\u003e\"http_header.server\", \"data\"=\u003e\"Apache/2.4.38 (Debian)\"}\n```\n\n#### favicons\n\nUsing the md5sum of a favicon to identify a running service:\n\n```\n# Example plaintext input\necho -n fe22dd2bb09daccf58256611ac491469 | bin/recog_match xml/favicons.xml -\n\n# Example cURL command\ncurl --silent http://localhost:8000/favicon.ico | md5sum | awk '{ print $1 }' | bin/recog_match xml/favicons.xml -\n\n# Example output\nMATCH: {\"matched\"=\u003e\"Drupal CMS\", \"service.vendor\"=\u003e\"Drupal\", \"service.product\"=\u003e\"CMS\", \"service.certainty\"=\u003e\"0.5\", \"service.cpe23\"=\u003e\"cpe:/a:drupal:drupal:-\", \"service.protocol\"=\u003e\"\", \"fingerprint_db\"=\u003e\"favicon.md5\", \"data\"=\u003e\"fe22dd2bb09daccf58256611ac491469\"}\n```\n\n#### http_wwwauth\n\nUsing the HTTP `WWW-Authenticate` header value to fingerprint an HTTP server:\n\n```\n# Example plaintext input\necho -n 'Basic realm=\"monit\"' | bin/recog_match xml/http_wwwauth.xml -\n\n# Example cURL command\ncurl --silent -I http://localhost:9001 | grep --color=never -i '^WWW-Authenticate:' | cut -d: -f2- | bin/recog_match xml/http_wwwauth.xml -\n\n# Example output\nMATCH: {\"matched\"=\u003e\"Minot\", \"service.vendor\"=\u003e\"Tildeslash\", \"service.product\"=\u003e\"Monit\", \"service.cpe23\"=\u003e\"cpe:/a:tildeslash:monit:-\", \"service.protocol\"=\u003e\"http\", \"fingerprint_db\"=\u003e\"http_header.wwwauth\", \"data\"=\u003e\"Basic realm=\\\"monit\\\"\"}\n```\n\n#### tls_jarm\n\nFingerprint TLS servers based on the server's response to 10 TLS Client Hello packets. Fingerprint based on https://github.com/salesforce/jarm\n\n```\n# Example plaintext input\necho -n 07d14d16d21d21d07c42d43d000000f50d155305214cf247147c43c0f1a823 | bin/recog_match xml/jarm.xml -\n\n# Example command using Salesforce's JARM against a running Metasploit listener\npython3 $code/jarm/jarm.py -p 8443 192.168.123.1 | grep --color=never 'JARM: ' | awk -F: '{ print $2 }' | bin/recog_match xml/tls_jarm.xml -\n\n# Example output\nMATCH: {\"matched\"=\u003e\"Metasploit listener\", \"service.vendor\"=\u003e\"Rapid7\", \"service.product\"=\u003e\"Metasploit\", \"service.cpe23\"=\u003e\"cpe:/a:rapid7:metasploit:-\", \"service.protocol\"=\u003e\"tls\", \"fingerprint_db\"=\u003e\"tls.jarm\", \"data\"=\u003e\"07d14d16d21d21d07c42d43d000000f50d155305214cf247147c43c0f1a823\"}\n```\n\n## Contributing\n\nThe users and maintainers of Recog would greatly appreciate any contributions\nyou can make to the project. For guidelines and instructions please see\n[CONTRIBUTING.MD](CONTRIBUTING.md)\n\n[^back to top](#recog-a-recognition-framework)\n","funding_links":[],"categories":["Threat Detection and Hunting","Ruby","\u003ca id=\"9eee96404f868f372a6cbc6769ccb7f8\"\u003e\u003c/a\u003e工具"],"sub_categories":["Tools","\u003ca id=\"31185b925d5152c7469b963809ceb22d\"\u003e\u003c/a\u003e新添加的"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frapid7%2Frecog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frapid7%2Frecog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frapid7%2Frecog/lists"}