{"id":13875896,"url":"https://github.com/Drew-Alleman/DataSurgeon","last_synced_at":"2025-07-16T10:32:05.532Z","repository":{"id":92353568,"uuid":"606695903","full_name":"Drew-Alleman/DataSurgeon","owner":"Drew-Alleman","description":" Quickly Extracts IP's, Email Addresses, Hashes, Files, Credit Cards, Social Security Numbers and a lot More From Text","archived":false,"fork":false,"pushed_at":"2024-02-28T17:43:25.000Z","size":27170,"stargazers_count":752,"open_issues_count":4,"forks_count":62,"subscribers_count":11,"default_branch":"main","last_synced_at":"2024-11-19T22:53:11.327Z","etag":null,"topics":["bug-bounty","ctf-tools","cybersecurity","email","file-search","hacking","incident-response","infosec","ip-address","osint","pentesting","recon","reconnaissance","regex","rust","rust-lang","search","search-tools","security-tools","windows"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Drew-Alleman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-26T09:32:09.000Z","updated_at":"2024-11-12T00:55:42.000Z","dependencies_parsed_at":"2024-10-30T02:34:48.202Z","dependency_job_id":null,"html_url":"https://github.com/Drew-Alleman/DataSurgeon","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Drew-Alleman%2FDataSurgeon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Drew-Alleman%2FDataSurgeon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Drew-Alleman%2FDataSurgeon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Drew-Alleman%2FDataSurgeon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Drew-Alleman","download_url":"https://codeload.github.com/Drew-Alleman/DataSurgeon/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225878124,"owners_count":17538400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bug-bounty","ctf-tools","cybersecurity","email","file-search","hacking","incident-response","infosec","ip-address","osint","pentesting","recon","reconnaissance","regex","rust","rust-lang","search","search-tools","security-tools","windows"],"created_at":"2024-08-06T06:00:49.182Z","updated_at":"2024-11-24T03:31:30.460Z","avatar_url":"https://github.com/Drew-Alleman.png","language":"Rust","readme":"# DataSurgeon v1.2.8\n![preview](media/main.gif)\nDataSurgeon (ds) is a versatile tool designed for incident response, DLP, penetration testing, and CTF challenges. It allows for the extraction of various types of sensitive information including emails, phone numbers, hashes, credit cards, URLs, IP addresses, MAC addresses, SRV DNS records and a lot more!\n\n* Supports Windows, Linux and MacOS\n* Support recursive file analysis within directories\n* Plugin Support\n* CSV Output\n\n# Quick Links\n* [Extraction Features](#extraction-features )\n* [Recent Updates](#recent-updates)\n* [Quick Install](#quick-install)\n* [Command Line Arguments](#command-line-arguments)\n* [Examples](#examples)\n* [Speed Tests](#speed-tests)\n* [Managing Plugins](https://github.com/Drew-Alleman/DataSurgeon/wiki/Plugin-Guide-%F0%9F%94%8C#managing-plugins)\n    - [Where to Submit Plugin Suggestions](https://github.com/Drew-Alleman/DataSurgeon/discussions/12)\n    - [Adding a New Plugin](https://github.com/Drew-Alleman/DataSurgeon/wiki/Plugin-Guide-%F0%9F%94%8C#adding-a-new-plugin)\n    - [Listing All Plugins](https://github.com/Drew-Alleman/DataSurgeon/wiki/Plugin-Guide-%F0%9F%94%8C#listing-all-plugins)\n    - [How to Use Your New Plugin](https://github.com/Drew-Alleman/DataSurgeon/wiki/Plugin-Guide-%F0%9F%94%8C#how-to-use-your-new-plugin)\n    - [Updating a Plugin](https://github.com/Drew-Alleman/DataSurgeon/wiki/Plugin-Guide-%F0%9F%94%8C#updating-a-plugin)\n    - [Removing a Plugin](https://github.com/Drew-Alleman/DataSurgeon/wiki/Plugin-Guide-%F0%9F%94%8C#removing-a-plugin)\n    - [Creating Your Own Plugin](https://github.com/DataSurgeon-ds/ds-winreg-plugin/blob/main/README.md#creating-your-own-plugin)\n* \u003cb\u003e[Reporting Issues](#reporting-issues)\u003c/b\u003e\n* [Project Goals](#project-goals)\n\n# Extraction Features \nTo learn how to manage plugins please follow the guide [here](#managing-plugins). \n## Personal Information\n* Emails\n* Phone numbers\n* Social Security Numbers\n\n## Financial Information\n* Credit Cards\n* Bitcoin wallets\n\n## Network Information\n* URL's\n* IPv4 Addresses and IPv6 addresses\n* MAC Addresses\n* SRV DNS Records\n\n## Security Information\n* Google API Private Key ID's\n* AWS Keys\n* [CVE Numbers](https://github.com/DataSurgeon-ds/ds-cve-plugin/) (PLUGIN)\n\n## System and File Information \n* Files\n* [Windows Registries](https://github.com/Drew-Alleman/ds-winreg-plugin/) (PLUGIN)\n  \n## Hashes\n* MD4 \u0026 MD5\n* SHA-1, SHA-224, SHA-256, SHA-384, SHA-512\n* SHA-3 224, SHA-3 256, SHA-3 384, SHA-3 512\n* MySQL 323, MySQL 41\n* NTLM\n* bcrypt\n\n# Recent Updates\n# 5/25/2023 | 1.2.0\nThis update brings a plugin management system to DataSurgeon. You can now add, remove, and list plugins using the ````--add````, ````--remove````, and ````--list```` options.\n\nI also added a new plugin, [ds-winreg-plugin](https://github.com/Drew-Alleman/ds-winreg-plugin/), that can find Windows registry paths.\n\nTo add a new plugin, use ````--add \u003cURL\u003e````, where the URL is a GitHub repository with a ````plugins.json```` file. To remove a plugin, use ````--remove \u003cURL\u003e````. To see all your plugins, use the ````--list```` option.\n\nOnce a plugin is added, you can use it as an argument in DataSurgeon. The argument's name is the \"Argument Long Name\" in the plugin's ````plugins.json```` file.\n# Quick Install\nThe quick installer can also be used to update DataSurgeon. \n\n## Video Guide\n[![Guide](https://img.youtube.com/vi/_Sq8Qgndi4M/0.jpg)](https://www.youtube.com/watch?v=_Sq8Qgndi4M)\n\nInstall [Rust](https://www.rust-lang.org/tools/install) and [Github](https://git-scm.com/downloads) then RESTART YOUR TERMINAL.\n\n### Linux\n```\nread -p \"Would you like to add 'ds' to your local bin? This will make 'ds' executable from any location in your terminal. (y/n) \" response \u0026\u0026 wget -q -O - https://raw.githubusercontent.com/Drew-Alleman/DataSurgeon/main/install/install.sh | bash -s -- \"$response\"\n```\n\n### Windows \nEnter the line below in an elevated powershell window. \n```\nIEX (New-Object Net.WebClient).DownloadString(\"https://raw.githubusercontent.com/Drew-Alleman/DataSurgeon/main/install/install.ps1\")\n```\nRelaunch your terminal and you will be able to use ```ds``` from the command line.\n\n### Mac\n```\ncurl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/Drew-Alleman/DataSurgeon/main/install/install.sh | sh\n```\n\n# Command Line Arguments\n![help](media/help_preview.PNG)\n## Video Guide\n[![Video Title](https://img.youtube.com/vi/LAb6kBOWHwg/0.jpg)](https://www.youtube.com/watch?v=LAb6kBOWHwg)\n# Examples\n## Extracting Files From a Remote Webiste\nHere I use ```wget``` to make a request to stackoverflow then I forward the body text to ```ds``` . The ```-F``` option will list all files found. ```--clean``` is used to remove any extra text that might have been returned (such as extra html). Then the result of is sent to ```uniq``` which removes any non unique files found. Ig you wanted you can remove the warning message at the top 'Reading standard input..' by using -S.\n```\n$ wget -qO - https://www.stackoverflow.com | ds -F --clean | uniq\n```\n![preview](media/wget_preview.gif)\n\n## Extracting Mac Addresses From an Output File\nHere I am pulling all mac addresses found in [autodeauth's](https://github.com/Drew-Alleman/autodeauth) log file using the ```-m``` query. The ```--hide``` option will hide the identifer string infront of the results. In this case 'mac_address: ' is hidden from the output. The ```-T``` option is used to check the same line multiple times for matches. Normallly when a match is found the tool moves on to the next line rather then checking again. \n```\n$ ./ds -m -T --hide -f /var/log/autodeauth/log     \n2023-02-26 00:28:19 - Sending 500 deauth frames to network: BC:2E:48:E5:DE:FF -- PrivateNetwork\n2023-02-26 00:35:22 - Sending 500 deauth frames to network: 90:58:51:1C:C9:E1 -- TestNet\n```\n\n## Reading all files in a directory\nYou can use the ```--directory``` option to read all files in the specified directory recursively. The ```-D``` option is used to display the filename where the match was found. ```-l``` or ```--line``` is used to display the line number the content was found on.\n```\n$ ds --directory test_dir/ -Dl\n```\n![preview](media/directory_search.gif)\n\n## CSV Output\nTo output your results to a CSV file, use the ```-o``` option followed by the name of the file you want to save your data to. The ```-D``` and ```-X``` are supported. The format is: ```ds -o \u003cFILENAME\u003e.csv``` (.csv is needed). \n```\n $ wget -qO - https://www.stackoverflow.com | ds -o output.csv -C\n```\n![preview](media/csv_output.gif)\n\n\n# Speed Tests\nWhen no specific query is provided, ```ds``` will search through all possible types of data, which is \u003cb\u003eSIGNIFICANTLY\u003c/b\u003e slower than using individual queries. The slowest query is ```--files```. Its also slightly faster to use ```cat``` to pipe the data to ```ds```. \n\nBelow is the elapsed time when processing a 5GB test file generated by [ds-test](https://github.com/Drew-Alleman/ds-test). Each test was ran 3 times and the average time was recorded.\n\n### Computer Specs \n```\nProcessor\tIntel(R) Core(TM) i5-10400F CPU @ 2.90GHz, 2904 Mhz, 6 Core(s), 12 Logical Processor(s)\nRam         12.0 GB (11.9 GB usable)\n```\n\n## Searching all data types\n\nCommand         | Speed         \n----------------|----------------\n`cat test.txt \\| ds -t` | 00h:02m:04s |\n`ds -t -f test.txt` | 00h:02m:05s\n`cat test.txt \\| ds -t -o output.txt` | 00h:02m:06s\n\n## Using specific queries\n\nCommand         | Speed          | Query Count\n----------------|----------------|----------------\n`cat test.txt \\| ds -t -6` | 00h:00m:12s | 1\n`cat test.txt \\| ds -t -i -m` | 00h:00m:22 | 2\n`cat test.txt \\| ds -tF6c` | 00h:00m:32s | 3\n\n# Reporting Issues\nWhen filling out a new issue please answer ALL questions on the  [bug template](https://github.com/Drew-Alleman/DataSurgeon/blob/main/.github/ISSUE_TEMPLATE/bug_report.md). Issues with not enough information will be closed. \n\n# Project Goals\n* JSON output\n* Untar/unzip and a directorty searching mode\n* Base64 Detection and decoding\n","funding_links":[],"categories":["Rust","1. [↑](#-content) OSINT","[](#table-of-contents) Table of contents","bug-bounty"],"sub_categories":["[](#dorkspentestvulnerabilities)Dorks/Pentest/Vulnerabilities"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDrew-Alleman%2FDataSurgeon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDrew-Alleman%2FDataSurgeon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDrew-Alleman%2FDataSurgeon/lists"}