{"id":13398400,"url":"https://github.com/USArmyResearchLab/Dshell","last_synced_at":"2025-03-14T02:31:19.328Z","repository":{"id":24723578,"uuid":"28135622","full_name":"USArmyResearchLab/Dshell","owner":"USArmyResearchLab","description":"Dshell is a network forensic analysis framework.","archived":false,"fork":false,"pushed_at":"2024-05-07T12:53:36.000Z","size":2686,"stargazers_count":5453,"open_issues_count":3,"forks_count":1145,"subscribers_count":589,"default_branch":"master","last_synced_at":"2024-10-29T15:04:09.873Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/USArmyResearchLab.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-12-17T12:04:51.000Z","updated_at":"2024-10-25T16:23:29.000Z","dependencies_parsed_at":"2022-08-07T11:01:17.776Z","dependency_job_id":"096f9203-3ca1-4171-97c1-4e14ca77b9ed","html_url":"https://github.com/USArmyResearchLab/Dshell","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/USArmyResearchLab%2FDshell","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/USArmyResearchLab%2FDshell/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/USArmyResearchLab%2FDshell/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/USArmyResearchLab%2FDshell/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/USArmyResearchLab","download_url":"https://codeload.github.com/USArmyResearchLab/Dshell/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243382924,"owners_count":20282049,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T19:00:24.848Z","updated_at":"2025-03-14T02:31:19.295Z","avatar_url":"https://github.com/USArmyResearchLab.png","language":"Python","readme":"# Dshell\nAn extensible network forensic analysis framework. Enables rapid development of plugins to support the dissection of network packet captures.\n\nKey features:\n* Deep packet analysis using specialized plugins\n* Robust stream reassembly\n* IPv4 and IPv6 support\n* Multiple user-selectable output formats and the ability to create custom output handlers\n* Chainable plugins\n* Parallel processing option to divide the handling of data source into separate Python processes\n* Enables development of external plugin packs to share and install new externally developed plugins without overlapping the core Dshell plugin directories\n\n## Guides\n* [Dshell User Guide](Dshell_User_Guide.pdf) \n  * A guide to installation as well as both basic and advanced analysis with examples\n  * Helps new and experienced end users with using and understanding the decoder-shell (Dshell) framework\n* [Dshell Developer Guide](Dshell_Developer_Guide.pdf) \n  * A guide to plugin development with basic examples, as well as core function and class definitions, and an overview of data flow\n  * Helps end users develop new, custom Dshell plugins as well as modify existing plugins\n  \n## Requirements\n* Linux (developed on Ubuntu 20.04 LTS)\n* Python 3 (developed with Python 3.8.10)\n* [pypacker](https://gitlab.com/mike01/pypacker)\n* [pcapy-ng](https://github.com/stamparm/pcapy-ng/)\n* [pyOpenSSL](https://github.com/pyca/pyopenssl)\n* [geoip2](https://github.com/maxmind/GeoIP2-python)\n  * [MaxMind GeoIP2 data sets](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data)\n    * Used to map IP addresses to country codes\n    * See Installation section for configuration \n\n## Optional\n* [oui.txt](http://standards-oui.ieee.org/oui.txt)\n  * used by some plugins that handle MAC addresses\n  * place in \u0026lt;dshell\u0026gt;/data/\n* [elasticsearch](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html)\n  * used in the elasticout output module\n  * only necessary if planning to use elasticsearch to store output\n* [pyJA3](https://github.com/salesforce/ja3/tree/master/python)\n  * used in the tls plugin\n\n## Installation\n\n1. Install Dshell with pip\n  * `python3 -m pip install Dshell/` OR `python3 -m pip install \u003cDshell-tarball\u003e`\n2. Configure geoip2 by placing the MaxMind GeoLite2 data set files (GeoLite2-ASN.mmdb, GeoLite2-City.mmdb, GeoLite2-Country.mmdb) in [...]/site-packages/dshell/data/GeoIP/\n3. Run `dshell`. This should drop you into a `Dshell\u003e ` prompt.\n\n## Basic Usage\n\n* `decode -l`\n  * This will list all available plugins, alongside basic information about them\n* `decode -h`\n  * Show generic command-line flags available to most plugins, such as the color blind friendly mode for all color output\n* `decode -p \u003cplugin\u003e`\n  * Display information about a plugin, including available command line flags\n* `decode -p \u003cplugin\u003e \u003cpcap\u003e`\n  * Run the selected plugin on a pcap or pcapng file\n* `decode -p \u003cplugin1\u003e+\u003cplugin2\u003e \u003cpcap\u003e`\n  * Chain two (or more) plugins together and run them on a pcap file\n* `decode -p \u003cplugin\u003e -i \u003cinterface\u003e`\n  * Run the selected plugin live on an interface (may require superuser privileges)\n\n## Usage Examples\nShowing DNS lookups in [sample traffic](http://wiki.wireshark.org/SampleCaptures#General_.2F_Unsorted)\n\n```\nDshell\u003e decode -p dns ~/pcap/dns.cap | sort\n[DNS] 2005-03-30 03:47:46    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 4146, TXT? google.com., TXT: b'\\x0fv=spf1 ptr ?all' **\n[DNS] 2005-03-30 03:47:50    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 63343, MX? google.com., MX: b'\\x00(\\x05smtp4\\xc0\\x0c', MX: b'\\x00\\n\\x05smtp5\\xc0\\x0c', MX: b'\\x00\\n\\x05smtp6\\xc0\\x0c', MX: b'\\x00\\n\\x05smtp1\\xc0\\x0c', MX: b'\\x00\\n\\x05smtp2\\xc0\\x0c', MX: b'\\x00(\\x05smtp3\\xc0\\x0c' **\n[DNS] 2005-03-30 03:47:59    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 18849, LOC? google.com. **\n[DNS] 2005-03-30 03:48:07    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 39867, PTR? 104.9.192.66.in-addr.arpa., PTR: 66-192-9-104.gen.twtelecom.net. **\n[DNS] 2005-03-30 03:49:18    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 30144, A? www.netbsd.org., A: 204.152.190.12 (ttl 82159s) **\n[DNS] 2005-03-30 03:49:35    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 61652, AAAA? www.netbsd.org., AAAA: 2001:4f8:4:7:2e0:81ff:fe52:9a6b (ttl 86400s) **\n[DNS] 2005-03-30 03:50:35    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 32569, AAAA? www.netbsd.org., AAAA: 2001:4f8:4:7:2e0:81ff:fe52:9a6b (ttl 86340s) **\n[DNS] 2005-03-30 03:50:44    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 36275, AAAA? www.google.com., CNAME: 'www.l.google.com.' **\n[DNS] 2005-03-30 03:50:54    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 56482, AAAA? www.l.google.com. **\n[DNS] 2005-03-30 03:51:35    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 48159, AAAA? www.example.com. **\n[DNS] 2005-03-30 03:51:46    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 9837, AAAA? www.example.notginh., NXDOMAIN **\n[DNS] 2005-03-30 03:52:17    192.168.170.8:32795 --   192.168.170.20:53    ** ID: 65251, AAAA: 2001:4f8:0:2::d (ttl 600s), A: 204.152.184.88 (ttl 600s) **\n[DNS] 2005-03-30 03:52:17    192.168.170.8:32796 --   192.168.170.20:53    ** ID: 23123, PTR? 1.0.0.127.in-addr.arpa., PTR: localhost. **\n[DNS] 2005-03-30 03:52:17    192.168.170.8:32797 --   192.168.170.20:53    ** ID: 8330, NS: b'\\x06ns-ext\\x04nrt1\\xc0\\x0c', NS: b'\\x06ns-ext\\x04sth1\\xc0\\x0c', NS: b'\\x06ns-ext\\xc0\\x0c', NS: b'\\x06ns-ext\\x04lga1\\xc0\\x0c' **\n[DNS] 2005-03-30 03:52:17   192.168.170.56:1707  --      217.13.4.24:53    ** ID: 12910, SRV? _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs.utelsystems.local., NXDOMAIN **\n[DNS] 2005-03-30 03:52:17   192.168.170.56:1708  --      217.13.4.24:53    ** ID: 61793, SRV? _ldap._tcp.dc._msdcs.utelsystems.local., NXDOMAIN **\n[DNS] 2005-03-30 03:52:17   192.168.170.56:1709  --      217.13.4.24:53    ** ID: 33633, SRV? _ldap._tcp.05b5292b-34b8-4fb7-85a3-8beef5fd2069.domains._msdcs.utelsystems.local., NXDOMAIN **\n[DNS] 2005-03-30 03:52:17   192.168.170.56:1710  --      217.13.4.24:53    ** ID: 53344, A? GRIMM.utelsystems.local., NXDOMAIN **\n[DNS] 2005-03-30 03:52:25   192.168.170.56:1711  --      217.13.4.24:53    ** ID: 30307, A? GRIMM.utelsystems.local., NXDOMAIN **\n```\n\nFollowing and reassembling a stream in [sample traffic](http://wiki.wireshark.org/SampleCaptures#General_.2F_Unsorted)\n\n```\nDshell\u003e decode -p followstream ~/pcap/v6-http.cap \nConnection 1 (TCP)\nStart: 2007-08-05 15:16:44.189851\nEnd:   2007-08-05 15:16:44.219460\n2001:6f8:102d:0:2d0:9ff:fee3:e8de: 59201 -\u003e 2001:6f8:900:7c0::2:    80 (300 bytes)\n2001:6f8:900:7c0::2:    80 -\u003e 2001:6f8:102d:0:2d0:9ff:fee3:e8de: 59201 (2379 bytes)\n\nGET / HTTP/1.0\nHost: cl-1985.ham-01.de.sixxs.net\nAccept: text/html, text/plain, text/css, text/sgml, */*;q=0.01\nAccept-Encoding: gzip, bzip2\nAccept-Language: en\nUser-Agent: Lynx/2.8.6rel.2 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.8b\n\n\n\nHTTP/1.1 200 OK\nDate: Sun, 05 Aug 2007 19:16:44 GMT\nServer: Apache\nContent-Length: 2121\nConnection: close\nContent-Type: text/html\n\n\u003c!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\"\u003e\n\u003chtml\u003e\n \u003chead\u003e\n  \u003ctitle\u003eIndex of /\u003c/title\u003e\n \u003c/head\u003e\n \u003cbody\u003e\n\u003ch1\u003eIndex of /\u003c/h1\u003e\n\u003cpre\u003e\u003cimg src=\"/icons/blank.gif\" alt=\"Icon \"\u003e \u003ca href=\"?C=N;O=D\"\u003eName\u003c/a\u003e                    \u003ca href=\"?C=M;O=A\"\u003eLast modified\u003c/a\u003e      \u003ca href=\"?C=S;O=A\"\u003eSize\u003c/a\u003e  \u003ca href=\"?C=D;O=A\"\u003eDescription\u003c/a\u003e\u003chr\u003e\u003cimg src=\"/icons/folder.gif\" alt=\"[DIR]\"\u003e \u003ca href=\"202-vorbereitung/\"\u003e202-vorbereitung/\u003c/a\u003e       06-Jul-2007 14:31    -   \n\u003cimg src=\"/icons/layout.gif\" alt=\"[   ]\"\u003e \u003ca href=\"Efficient_Video_on_demand_over_Multicast.pdf\"\u003eEfficient_Video_on_d..\u0026gt;\u003c/a\u003e 19-Dec-2006 03:17  291K  \n\u003cimg src=\"/icons/unknown.gif\" alt=\"[   ]\"\u003e \u003ca href=\"Welcome%20Stranger!!!\"\u003eWelcome Stranger!!!\u003c/a\u003e     28-Dec-2006 03:46    0   \n\u003cimg src=\"/icons/text.gif\" alt=\"[TXT]\"\u003e \u003ca href=\"barschel.htm\"\u003ebarschel.htm\u003c/a\u003e            31-Jul-2007 02:21   44K  \n\u003cimg src=\"/icons/folder.gif\" alt=\"[DIR]\"\u003e \u003ca href=\"bnd/\"\u003ebnd/\u003c/a\u003e                    30-Dec-2006 08:59    -   \n\u003cimg src=\"/icons/folder.gif\" alt=\"[DIR]\"\u003e \u003ca href=\"cia/\"\u003ecia/\u003c/a\u003e                    28-Jun-2007 00:04    -   \n\u003cimg src=\"/icons/layout.gif\" alt=\"[   ]\"\u003e \u003ca href=\"cisco_ccna_640-801_command_reference_guide.pdf\"\u003ecisco_ccna_640-801_c..\u0026gt;\u003c/a\u003e 28-Dec-2006 03:48  236K  \n\u003cimg src=\"/icons/folder.gif\" alt=\"[DIR]\"\u003e \u003ca href=\"doc/\"\u003edoc/\u003c/a\u003e                    19-Sep-2006 01:43    -   \n\u003cimg src=\"/icons/folder.gif\" alt=\"[DIR]\"\u003e \u003ca href=\"freenetproto/\"\u003efreenetproto/\u003c/a\u003e           06-Dec-2006 09:00    -   \n\u003cimg src=\"/icons/folder.gif\" alt=\"[DIR]\"\u003e \u003ca href=\"korrupt/\"\u003ekorrupt/\u003c/a\u003e                03-Jul-2007 11:57    -   \n\u003cimg src=\"/icons/folder.gif\" alt=\"[DIR]\"\u003e \u003ca href=\"mp3_technosets/\"\u003emp3_technosets/\u003c/a\u003e         04-Jul-2007 08:56    -   \n\u003cimg src=\"/icons/text.gif\" alt=\"[TXT]\"\u003e \u003ca href=\"neues_von_rainald_goetz.htm\"\u003eneues_von_rainald_go..\u0026gt;\u003c/a\u003e 21-Mar-2007 23:27   31K  \n\u003cimg src=\"/icons/text.gif\" alt=\"[TXT]\"\u003e \u003ca href=\"neues_von_rainald_goetz0.htm\"\u003eneues_von_rainald_go..\u0026gt;\u003c/a\u003e 21-Mar-2007 23:29   36K  \n\u003cimg src=\"/icons/layout.gif\" alt=\"[   ]\"\u003e \u003ca href=\"pruef.pdf\"\u003epruef.pdf\u003c/a\u003e               28-Dec-2006 07:48   88K  \n\u003chr\u003e\u003c/pre\u003e\n\u003c/body\u003e\u003c/html\u003e\n```\n\nChaining plugins to view flow data for a specific country code in [sample traffic](http://wiki.wireshark.org/SampleCaptures#General_.2F_Unsorted) (note: TCP handshakes are not included in the packet count)\n\n```\nDshell\u003e decode -p country+netflow --country_code=JP ~/pcap/SkypeIRC.cap\n2006-08-25 15:32:20.766761       192.168.1.2 -\u003e  202.232.205.123  (-- -\u003e JP)   UDP   60583   33438     1      0       64        0  0.0000s\n2006-08-25 15:32:20.634046       192.168.1.2 -\u003e  202.232.205.123  (-- -\u003e JP)   UDP   60583   33435     1      0       64        0  0.0000s\n2006-08-25 15:32:20.747503       192.168.1.2 -\u003e  202.232.205.123  (-- -\u003e JP)   UDP   60583   33437     1      0       64        0  0.0000s\n2006-08-25 15:32:20.651501       192.168.1.2 -\u003e  202.232.205.123  (-- -\u003e JP)   UDP   60583   33436     1      0       64        0  0.0000s\n```\n\nCollecting DNS traffic from several files and storing it in a new pcap file.\n\n```\nDshell\u003e decode -p dns+pcapwriter --pcapwriter_outfile=test.pcap ~/pcap/*.cap \u003e /dev/null\nDshell\u003e tcpdump -nnr test.pcap | head\nreading from file test.pcap, link-type EN10MB (Ethernet)\n15:36:08.670569 IP 192.168.1.2.2131 \u003e 192.168.1.1.53: 40209+ A? ui.skype.com. (30)\n15:36:08.670687 IP 192.168.1.2.2131 \u003e 192.168.1.1.53: 40210+ AAAA? ui.skype.com. (30)\n15:36:08.674022 IP 192.168.1.1.53 \u003e 192.168.1.2.2131: 40209- 1/0/0 A 212.72.49.131 (46)\n15:36:09.011208 IP 192.168.1.1.53 \u003e 192.168.1.2.2131: 40210 0/1/0 (94)\n15:36:10.171350 IP 192.168.1.2.2131 \u003e 192.168.1.1.53: 40210+ AAAA? ui.skype.com. (30)\n15:36:10.961350 IP 192.168.1.1.53 \u003e 192.168.1.2.2131: 40210* 0/1/0 (85)\n15:36:10.961608 IP 192.168.1.2.2131 \u003e 192.168.1.1.53: 40211+ AAAA? ui.skype.com. (30)\n15:36:11.294333 IP 192.168.1.1.53 \u003e 192.168.1.2.2131: 40211 0/1/0 (94)\n15:32:21.664798 IP 192.168.1.2.2130 \u003e 192.168.1.1.53: 39862+ A? ui.skype.com. (30)\n15:32:21.664913 IP 192.168.1.2.2130 \u003e 192.168.1.1.53: 39863+ AAAA? ui.skype.com. (30)\n```\n\nCollecting TFTP data and converting alerts to JSON format using [sample traffic](https://wiki.wireshark.org/SampleCaptures#TFTP)\n\n```\nDshell\u003e decode -p tftp -O jsonout ~/pcap/tftp_*.pcap\n{\"ts\": 1367411051.972852, \"sip\": \"192.168.0.253\", \"sport\": 50618, \"dip\": \"192.168.0.10\", \"dport\": 3445, \"readwrite\": \"read\", \"filename\": \"rfc1350.txt\", \"plugin\": \"tftp\", \"pcapfile\": \"/home/pcap/tftp_rrq.pcap\", \"data\": \"read  rfc1350.txt (24599 bytes) \"}\n{\"ts\": 1367053679.45274, \"sip\": \"192.168.0.1\", \"sport\": 57509, \"dip\": \"192.168.0.13\", \"dport\": 2087, \"readwrite\": \"write\", \"filename\": \"rfc1350.txt\", \"plugin\": \"tftp\", \"pcapfile\": \"/home/pcap/tftp_wrq.pcap\", \"data\": \"write rfc1350.txt (24599 bytes) \"}\n```\n\nRunning a plugin within a separate Python script using [sample traffic](https://wiki.wireshark.org/SampleCaptures#TFTP)\n\n```\n# Import required Dshell libraries\nimport dshell.decode as decode\nimport dshell.plugins.tftp.tftp as tftp\n\n# Instantiate plugin\nplugin = tftp.DshellPlugin()\n# Define plugin-specific arguments, if needed\ndargs = {plugin: {\"rip\": True, \"outdir\": \"/tmp/\"}}\n# Add plugin(s) to plugin chain\ndecode.plugin_chain = [plugin]\n# Run decode main function with all other arguments\ndecode.main(\n    debug=True,\n    files=[\"/home/user/pcap/tftp_rrq.pcap\", \"/home/user/pcap/tftp_wrq.pcap\"],\n    plugin_args=dargs\n)\n```\n","funding_links":[],"categories":["Tools","The United States of America","Network","Python","Automation","Python (1887)","自动化工具","Network Tools","Forensics","Automation and Convention"],"sub_categories":["Network Tools","United States Army Research Lab","Full Packet Capture / Forensic","零碎的","Protocol Analyzers and Sniffers","Protocol Analyzers / Sniffers","Steganography"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FUSArmyResearchLab%2FDshell","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FUSArmyResearchLab%2FDshell","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FUSArmyResearchLab%2FDshell/lists"}