{"id":13773503,"url":"https://github.com/rusty-ferris-club/recon","last_synced_at":"2025-04-30T19:31:16.087Z","repository":{"id":63478260,"uuid":"557993509","full_name":"rusty-ferris-club/recon","owner":"rusty-ferris-club","description":"🕵️‍♀️ Find, locate, and query files for ops and security experts ⚡️⚡️⚡️","archived":false,"fork":false,"pushed_at":"2022-12-18T16:24:59.000Z","size":1109,"stargazers_count":26,"open_issues_count":2,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-05-15T15:09:02.653Z","etag":null,"topics":["devops","devops-tools","rust","security","security-tools"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rusty-ferris-club.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-26T17:35:47.000Z","updated_at":"2024-04-22T19:52:30.000Z","dependencies_parsed_at":"2023-01-29T19:31:09.853Z","dependency_job_id":null,"html_url":"https://github.com/rusty-ferris-club/recon","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rusty-ferris-club%2Frecon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rusty-ferris-club%2Frecon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rusty-ferris-club%2Frecon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rusty-ferris-club%2Frecon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rusty-ferris-club","download_url":"https://codeload.github.com/rusty-ferris-club/recon/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251769308,"owners_count":21640883,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["devops","devops-tools","rust","security","security-tools"],"created_at":"2024-08-03T17:01:16.391Z","updated_at":"2025-04-30T19:31:14.794Z","avatar_url":"https://github.com/rusty-ferris-club.png","language":"Rust","funding_links":[],"categories":["Web","Tools"],"sub_categories":["Scanning / Pentesting","IOC Scanner","Web Vulnerability Scanners"],"readme":"\u003ch1 align=\"center\"\u003e\n   \u003cimg src=\"media/recon.svg\" width=\"160\"/\u003e\n   \u003cbr/\u003e\n   Recon\n\u003c/h1\u003e\n\n\u003ch4 align=\"center\"\u003eFind, locate, and query files for ops and security experts\u003c/h4\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/rusty-ferris-club/recon/actions/workflows/build.yml/badge.svg\"/\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#key-features\"\u003eKey Features\u003c/a\u003e •\n  \u003ca href=\"#how-to-use\"\u003eHow To Use\u003c/a\u003e •\n  \u003ca href=\"#download\"\u003eDownload\u003c/a\u003e •\n  \u003ca href=\"#contributing\"\u003eContributing\u003c/a\u003e •\n  \u003ca href=\"#license\"\u003eLicense\u003c/a\u003e\n\u003c/p\u003e\n\n![screenshot](media/recon-terminal.gif)\n\n## Key Features\n\n* **Query with SQL over files** - find files using the full power of SQL queries\n* **Find content with digests** - use SHA256/512, md5, crc32 for duplicates, and other matchers for nontrivial matches to locate artifacts on hosts\n* **Find malware or binaries with YARA** - use YARA rules for matching against binary files efficiently     \n* **Finetune your search runtime** - you can choose only the processing you need in order to cover more files quickly \n* **Build your own scripts** - pipe `recon` results to your own scripts with `--xargs`\n* **Export** - use `--csv` or `--json`, or upload `recon.db` to your own servers for analysis\n\n## How to Use\n\n\n```\n$ recon --help\nSQL over files with security processing and tests\n\nUsage: recon [OPTIONS]\n\nOptions:\n  -c, --config \u003cCONFIG_FILE\u003e  Point to a configuration\n  -r, --root \u003cROOT\u003e           Target folder to scan\n  -q, --query \u003cSQL\u003e           Query with SQL\n  -f, --file \u003cDB_FILE\u003e        Use a specific DB file (file or :memory: for in memory) [default: recon.db]\n  -d, --delete                Clear data: delete existing cache database before running\n  -u, --update                Always walk files and update DB before query. Leave off to run query on existing recon.db.\n  -a, --all                   Walk all files (dont consider .gitignore)\n      --no-progress           Don't display progress bars\n  -m, --inmem                 Don't cache index to disk, run in-memory only\n      --xargs                 Output as xargs formatted list\n      --json                  Output as JSON\n      --csv                   Output as CSV\n      --no-style              Output as a table with no styles\n      --fail-some             Exit code failure if *some* files are found\n      --fail-none             Exit code failure if *no* files are found\n      --verbose               Show logs\n  -h, --help                  Print help information\n  -V, --version               Print version information\n```\n\n### Query files\n\nBy default `recon` walks the current directory and allows you to perform standard SQL queries against it:\n\n```\n$ recon -q 'select path,is_binary,mode from files limit 5'\n┌───────────────┬───────────┬────────────┐\n│ path          │ is_binary │ mode       │\n├───────────────┼───────────┼────────────┤\n│ ./os.rs       │ null      │ -rw-r--r-- │\n├───────────────┼───────────┼────────────┤\n│ ./out.rs      │ null      │ -rw-r--r-- │\n├───────────────┼───────────┼────────────┤\n│ ./bin/main.rs │ null      │ -rw-r--r-- │\n├───────────────┼───────────┼────────────┤\n│ ./config.rs   │ null      │ -rw-r--r-- │\n├───────────────┼───────────┼────────────┤\n│ ./lib.rs      │ null      │ -rw-r--r-- │\n└───────────────┴───────────┴────────────┘\n5 files in 16.872083ms\n```\n\n`recon` also caches the metadata it indexes in `recon.db` (an sqlite DB), to avoid re-indexing when you're experimenting with different queries.\n\nTo query against the cache (fastest, creates the cache if missing):\n\n```\n$ recon -q \u003cyour query\u003e\n```\n\nTo always update the cache before runs (good for incomplete index runs or changed folders between runs):\n\n```\n$ recon -u -q \u003cyour query\u003e\n```\n\nTo delete the cache and recreate it before runs (good for starting from scratch):\n\n```\n$ recon -d -q \u003cyour query\u003e\n```\n\nTo avoid any disk operations and run from memory (no cache):\n\n```\n$ recon -m -q \u003cyour query\u003e\n```\n### Add processors and/or matchers\n\nConfigure processors and matchers in case you need some compute-heavy data to be available for query or matching such as:\n\n* Digests or checksums (md5, SHAxxx, crc32)\n* Path, content matching\n* YARA rules\n* For more, see [config.yaml.example](config.yaml.example)\n\nHow to know which ones you need?\n\n* If there aren't a lot of files, you can use all of them. Generally - don't worry about it.\n* If you have a performance constraint (lots of files + limited time to run full index), pick only the fields you need to query on.\n\n\nLet's add binary detection:\n\n```yaml\nsource:\n  computed_fields:\n    is_binary: true\n```\n\n\u003e Did you know? The common way to detect if a file is binary, is to read a chunk of data from it (usually 1k bytes) and search for any non-text bytes in it. Largely: there's no built-in OS metadata \"flag\" that indicates if a file is a binary file.\n\nUpon updating configuration use `-u` to update the local cache database.\n\n```\n$ recon -u -c ../config.yaml -q 'select path,is_binary,mode from files limit 6'\n┌───────────────┬───────────┬────────────┐\n│ path          │ is_binary │ mode       │\n├───────────────┼───────────┼────────────┤\n│ ./os.rs       │ 0         │ -rw-r--r-- │\n├───────────────┼───────────┼────────────┤\n│ ./out.rs      │ 0         │ -rw-r--r-- │\n├───────────────┼───────────┼────────────┤\n│ ./bin/main.rs │ 0         │ -rw-r--r-- │\n├───────────────┼───────────┼────────────┤\n│ ./config.rs   │ 0         │ -rw-r--r-- │\n├───────────────┼───────────┼────────────┤\n│ ./lib.rs      │ 0         │ -rw-r--r-- │\n├───────────────┼───────────┼────────────┤\n│ ./recon.db    │ 1         │ -rw-r--r-- │\n└───────────────┴───────────┴────────────┘\n6 files in 145.81675ms\n```\n\n## Download\n\nFor macOS:\n\n```\nbrew tap rusty-ferris-club/tap \u0026\u0026 brew install recon\n```\n\nOtherwise, grab a release from [releases](https://github.com/rusty-ferris-club/recon/releases) and run `recon --help`:\n\n\n## Examples\n\nFully configured examples ([open a PR](https://github.com/rusty-ferris-club/recon/pulls) to add your own!):\n\n| Example                                             | Purpose                                                                |\n| --------------------------------------------------- | ---------------------------------------------------------------------- |\n| [all-processors.yaml](examples/all-processors.yaml) | Turn on all processors by default                                      |\n| [custom-walking.yaml](examples/custom-walking.yaml) | Compute fields only for part of the walked directory tree              |\n| [file-classes.yaml](examples/file-classes.yaml)     | Configure and classify your own file classes                           |\n| [find-log4shell.yaml](examples/find-log4shell.yaml) | An example for finding the log4shell vulnerability using known digests |\n| [using-yara.yaml](examples/using-yara.yaml)         | Using a simple YARA ruleset for matching                               |\n\n\n\n## FAQ\n\n\u003cdetails\u003e\u003csummary\u003e\u003cb\u003e\nWhat can I use this tool for?\n\u003c/b\u003e\u003c/summary\u003e\n\nRecon is a multi-purpose metadata indexing search and query tool with a focus on content types.\n\nYou can use it as:\n\n* A better `find`. Find files with a more expressive SQL query abilities.\n* A forensics tool: locate files matching advanced or simple criteria from machines that have issues.\n* A reconnaissance tool for security experts: explore infected / suspected machines and discover and capture malware or harmful content\n\n\n```\n$ DATABASE_URL=sqlite::memory: recon \u003c..args..\u003e\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003e\u003cb\u003e\nWhat are some example queries?\n\u003c/b\u003e\u003c/summary\u003e\n\nMake sure you have a config file which configures computed fields for digests:\n\n```yaml\n# ...\n  computed_fields:\n    byte_type: true\n    is_binary: true\n    file_magic: true\n    # crc32: true\n    sha256: true\n    sha512: true\n    simhash: true\n```\n\nYou can try any of these:\n\n\nShow binary files only:\n\n```\nrecon -c config.yaml -q 'select path from files where is_binary = 1'\n```\n\nCreate a list of file digests\n\n```\nrecon -c config.yaml -q 'select path,sha256,sha512 from files'\n```\n\n\nExport everything\n\n```\nrecon -c config.yaml -q 'select * from files' --csv\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003e\u003cb\u003e\nHow can I configure cache behavior?\n\u003c/b\u003e\u003c/summary\u003e\n\nRecon uses a sqlite database file for caching, query, and capture.\n\nThe default is to use a file called `recon.db`. You can pick your own database file name:\n\n```\n$ recon -f DB_FILE -q \u003cquery\u003e\n```\nTo use an inmemory database, use the special file name `:memory:`:\n\n```\n$ recon -f :memory: -q \u003cquery\u003e\n```\n\nOr, you can use an inmemory database by giving a full URL to your database via a `DATABASE_URL` environment variable. This environment variable takes precedence over specifying a file.\n\n```\n$ DATABASE_URL=sqlite::memory: recon \u003c..args..\u003e\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003e\u003cb\u003e\nHow can I see what fields to select from?\n\u003c/b\u003e\u003c/summary\u003e\n\nYou can view the full schema [here](recon/migrations/1_files.sql), or issue an ad-hoc sqlite query:\n\n```\n$ recon -q 'PRAGMA table_info(files)'\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\u003csummary\u003e\u003cb\u003e\nWhat are computed fields and how can I pick them?\n\u003c/b\u003e\u003c/summary\u003e\n\nComputed fields are fields that are either compute-intensive or not always needed. You have a fully annotated [example configuration](config.yaml.example).\n\nYou can run with a configuration file:\n\n```\n$ recon -c your-config.yaml\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003e\u003cb\u003e\nHow can I use this in shell scripts?\n\u003c/b\u003e\u003c/summary\u003e\n\nYou can use `xargs` to follow with extra actions for files that you discover:\n\n\n```\n$ recon -q \u003cyour query\u003e --xargs | xargs -I {} echo {}\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003e\u003cb\u003e\nHow can I capture remote state?\n\u003c/b\u003e\u003c/summary\u003e\n\nYou can send this out from a given machine using JSON:\n\n```\n$ recon -q \u003cyour query\u003e --json | your-http-post-command\n```\n\nOr you can ship out as CSV using `--csv` or even the `recon.db` file as-is out of the machine.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\u003csummary\u003e\u003cb\u003e\nHow do I run on large folders or complete disks?\n\u003c/b\u003e\u003c/summary\u003e\n\n`recon` can run without interruption on very large folders. Each run consists of two stages:\n\n1. Walking the file system. Here, we don't know how many files there are, so we're showing number of files indexed and time elapsed.\n2. Processing compute intensive fields. Here we have a list of target files to compute, so we're showing a progress bar.\n\nYou can interrupt walking the file system, and resume it later. Use the `-u` flag to make `recon` always update DB before query, this will also make it resume any non-processed file.\n\n```\n$ recon -u \u003cother args\u003e\n```\n\n\u003c/details\u003e\n\n# Contributing\n\nWe are accepting PRs. Feel free to [submit PRs](https://github.com/rusty-ferris-club/recon/pulls).\n\nTo all [Contributors](https://github.com/rusty-ferris-club/recon/graphs/contributors) - you make this happen, thanks!\n\n# License\n\nCopyright (c) 2022 [@jondot](http://twitter.com/jondot). See [LICENSE](LICENSE.txt) for further details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frusty-ferris-club%2Frecon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frusty-ferris-club%2Frecon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frusty-ferris-club%2Frecon/lists"}