{"id":13774569,"url":"https://github.com/armenb/sharktools","last_synced_at":"2025-05-11T06:33:19.968Z","repository":{"id":1003147,"uuid":"817513","full_name":"armenb/sharktools","owner":"armenb","description":"Tools for programmatic parsing of packet captures using Wireshark functionality","archived":false,"fork":false,"pushed_at":"2013-09-30T13:14:45.000Z","size":365,"stargazers_count":92,"open_issues_count":9,"forks_count":20,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-11-17T09:39:29.057Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/armenb.png","metadata":{"files":{"readme":"README","changelog":"ChangeLog","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2010-08-04T18:02:07.000Z","updated_at":"2024-08-12T19:19:26.000Z","dependencies_parsed_at":"2022-07-14T21:17:04.418Z","dependency_job_id":null,"html_url":"https://github.com/armenb/sharktools","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/armenb%2Fsharktools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/armenb%2Fsharktools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/armenb%2Fsharktools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/armenb%2Fsharktools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/armenb","download_url":"https://codeload.github.com/armenb/sharktools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253528415,"owners_count":21922623,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T17:01:28.019Z","updated_at":"2025-05-11T06:33:19.589Z","avatar_url":"https://github.com/armenb.png","language":"C","funding_links":[],"categories":["\u003ca id=\"6fa0e0d1f898fba299b2566a33602841\"\u003e\u003c/a\u003eWireshark"],"sub_categories":[],"readme":"\n= Introduction =\n\n\"Sharktools\" is the name given to a small set of tools that allow use of\nWireshark's deep packet inspection capabilities in interpreted\nprogramming languages.  The two currently supported interpreted\nprogramming languages are Python and Matlab; \"pyshark\" is the name\nof the tool for Python, and \"matshark\" is the name of the tool for\nMatlab.\n\nSharktools is written in C.\n\n= Basic Operational Concept =\n\n1) A user collects packets using a packet sniffer (e.g. Wireshark or tcpdump)\nand saves them in a pcap file.\n\n2) Given an arbitrary pcap file, Sharktools uses Wireshark's Display Filter\ntechnology (which knows how to parse thousands of common and obscure\nnetwork protocols) to cherry-pick packet fields of interest.  See the\nAppendix of this document for details.\n\n3) Sharktools then provides this data as a cell array of structs in Matlab, or\na list of dictionaries in Python.\n\n4) A user can then plot packet fields with respect to time or carry out more\ncomplicated analysis of packet captures in their favorite programming\nenvironment.\n\n= Authors =\n\nArmen Babikyan of MIT Lincoln Laboratory \u003carmenb@mit.edu\u003e, for\n* Sharktools core\n* matshark\n* pyshark\n* bug fixes\n\nNathaniel Jones of MIT Lincoln Laboratory \u003cnjones@ll.mit.edu\u003e, for\n* bug fixes to Sharktools core and matshark\n\n= Links = \n\nSharktools makes use of the following third-party programs:\n\n* Wireshark, http://www.wireshark.org\n* libpcap, http://en.wikipedia.org/wiki/Pcap#libpcap\n* Matlab, http://www.mathworks.com/products/matlab\n* Python, http://www.python.org\n\n= Known-working Platforms/Environments =\n\nThis software should work on both 32-bit and 64-bit Linux systems, with\nrelatively new versions of Matlab (R2007+), with relatively new versions of\nPython (\u003e 2.4) and relatively new versions of Wireshark (\u003e 1.0).\n\nSpecifically, as of this writing, this software has been tested and is\nconfirmed working on:\n - RHEL5.5 + Matlab R2010a + Wireshark 1.0.11\n - RHEL5.5 + Matlab R2010a + Wireshark 1.0.15\n - RHEL5.5 + Matlab R2010a + Wireshark 1.2.7\n - RHEL5.5 + Python 2.4.3 + Wireshark 1.2.7\n - RHEL5.5 + Python 2.4.3 + Wireshark 1.4.0\n - Ubuntu 10.04.1 LTS + Python 2.6.5 + Wireshark 1.2.7\n - MacOSX 10.6 + Python 2.4.3 + Wireshark 1.4.0 (see README.MacOSX)\n - MacOSX 10.6 + Matlab R2010a + Wireshark 1.4.0 (see README.MacOSX)\n - MacOSX 10.8 + Python 2.7.3 + Wireshark 1.8.6 (see README.MacOSX)\n - MacOSX 10.8 + Python 2.7.3 + Wireshark 1.10.0 (see README.MacOSX)\nSee the FAQ below for answers to common problems/questions. For more\ninformation, contact Armen Babikyan (armenb@static.net).\n\n= Features and Usage =\n\n== Example Usage in Matlab ==\n\nPass matshark a pcap file, a list of wireshark fields of interest, and a\ndisplay filter string. matshark will return a cellarray of structs.\n\nExample usage:\n\n\u003e\u003e b = matshark('capture1.pcap', {'frame.number', 'ip.version', 'tcp.seq', 'udp.dstport', 'frame.len'}, 'ip.version eq 4')\n\nb =\n\n1x76 struct array with fields:\n    frame_number\n    ip_version\n    tcp_seq\n    udp_dstport\n    frame_len\n\n\u003e\u003e b(3)\n\nans =\n\n    frame_number: 6\n      ip_version: 4\n         tcp_seq: []\n     udp_dstport: 60000\n       frame_len: 60\n\n\u003e\u003e\n\nAnother example, showing usage of time fields and conversion of struct members\nto an array:\n\n\u003e\u003e c = matshark('capture1.pcap', {'frame.number', 'frame.time', 'frame.time_relative', 'frame.len', 'frame.protocols'}, '' )\n\nc =\n\n1x100 struct array with fields:\n    frame_number\n    frame_time\n    frame_time_relative\n    frame_len\n    frame_protocols\n\n\u003e\u003e c(9)\n\nans =\n\n           frame_number: 9\n             frame_time: 1.0664e+09\n    frame_time_relative: 0.0228\n              frame_len: 60\n        frame_protocols: ''\n\n\u003e\u003e t = [c.frame_time_relative];\n\u003e\u003e t = t - t(1);\n\u003e\u003e t(9)                        \n\nans =\n\n    0.0228\n\n\u003e\u003e \n\nSometimes you can request pieces of data that are impossible to find in\npackets.  For example, you should never have a tcp.seq and udp.dstport in the\nsame packet.  In this case, matshark will insert an empty list in its place;\npyshark will insert a None object in its place.\n\n== Example Usage in Python ==\n\n\u003e\u003e\u003e import pyshark\n\u003e\u003e\u003e b = pyshark.read('capture1.pcap', ['frame.number', 'ip.version', 'tcp.seq', 'udp.dstport', 'frame.len'], 'ip.version eq 4')\n\u003e\u003e\u003e b = list(b)\n\u003e\u003e\u003e b[2]\n{'frame.number': 6, 'tcp.seq': None, 'frame.len': 60, 'udp.dstport': 60000, 'ip.version': 4}\n\u003e\u003e\u003e c = pyshark.read('capture1.pcap', ['frame.number', 'frame.time', 'frame.time_relative', 'frame.len', 'frame.protocols'], '' )\n\u003e\u003e\u003e c = list(c)\n\u003e\u003e\u003e c[8]\n{'frame.number': 9, 'frame.len': 60, 'frame.time': 1066402442.768941, 'frame.time_relative': 0.022801999999999999, 'frame.protocols': None}\n\u003e\u003e\u003e\n\n== Python Iterators ==\n\nPyshark now uses Python iterators to reduce memory footprint.  To quickly adapt\nolder code, execute \"foo = list(foo)\" on the data structure \"foo\" returned by\npyshark.read().  Read up on how iterators can make your life easier here:\n\nhttp://www.ibm.com/developerworks/library/l-pycon/index.html\n\nYou may also wish to search for Normal Matloff's \"PyIterGen.pdf\" file on the web.\n\n\n\n== Using Wireshark's \"Decode As\" feature ==\n\nWireshark's packet dissection engine uses a combination of heuristics and\nconvention to determine what dissector to use for a particular packet. For\nexample, IP packets with TCP port 80 are, by default, parsed as HTTP packets.\nIf you wish to have TCP port 800 packets parsed as HTTP packets, you need to\ntell the Wireshark engine your explicit intent.\n\nWireshark adds a \"decode as\" feature in its GUI that allows for users to\nspecify this mapping (Analyze Menu -\u003e Decode As...).  Sharktools attempts to\nprovide a basic interface to this feature as well.  By adding a 4th (optional)\nargument to both the matshark and pyshark commands, a user can achieve the\ndesired effect.  For example, the following \"decode as\" string will parse TCP\nport 60000 packets as HTTP packets: 'tcp.port==60000,http'\n\n= Building/Installation instructions =\n\nYou'll need a few things:\n\n1) Install Wireshark, the packet capture tool:\n     Ubuntu 10.04.1 LTS: apt-get install wireshark wireshark-dev\n     RedHat Enterprise Linux 5: yum install wireshark\n\n   See the FAQ below for MacOSX installation instructions.\n\n   Make note of the version number.\n\n   NB: The wireshark-dev package on Ubuntu creates the /usr/lib/wireshark/libwireshark.so\n   symlink (among doing other things); this is necessary so pyshark and matshark\n   can be built.\n\n2) Install Glib-2.0 development package, which contains headers and libraries\n   necessary for sharktools.  Practically all Linux distributions have glib-2.0,\n   named something like glib2-devel (rpm-based systems) or libglib2.0-dev (deb-\n   based systems).  On MacOSX, you will need Macports (preferred) or fink, but\n   either distribution's glib2 should work fine.\n\n   NB: glib 1.* and glib 2.* usaully coexist on Linux and MacOSX systems; you need\n   the latter.\n\n3) Install bison, flex, and libpcap-dev packages on your system:\n      apt-get install bison flex libpcap-dev\n      yum install bison flex libpcap-devel\n\n   NB: These are only needed for the next step\n\n4) Download, unpack, and run ./configure on the Wireshark source from\n   http://www.wireshark.org.\n\n   Be sure that you download the version of Wireshark that is roughly(*) the\n   same as the version of Wireshark installed by your package management\n   system.  The source to Wireshark is needed because your distribution's\n   wireshark-dev package is generally not sufficient(**) to build sharktools.\n\n   Unpack the tarball by running:\n       tar -zxf wireshark-\u003cversion\u003e.tar.gz\n   Change into the soruce directory and run the following command(***):\n       ./configure --disable-wireshark\n\n   (*) Make sure the Major, Minor, and Sub-Minor numbers are the same.  For\n        example, if you have the wireshark-1.0.8-1.el5_3.1 RPM package\n        installed, you should download wireshark-1.0.8.tar.gz.  1.0.7 or 1.0.9\n        won't cut it.\n\n   (**) sharktools uses some data structures in wireshark's headers that are\n        unfortunately not packaged with wireshark-dev package (e.g. cfile.h,\n        file.h, print.h).  You only need these headers to build the software,\n        and you can remove them afterwards.\n\n   (***) Since we aren't actually building Wireshark, we need the\n        \"--disable-wireshark\" argument to instruct the configure script to\n        ignore the lack of gtk2 development headers and libraries on your\n        system.  The word \"wireshark\" in --disable-wireshark is referring to\n        the GUI frontend program. If you insist on leaving this argument off,\n        note that you'll probably have to install the gtk2-dev(el) package\n        on your system, or the configure script will thrown an error.\n\n5) (Semi-Required - needed for matshark) Install Matlab.  You will need its \"mex\"\n   (Matlab EXternal) tool, which allows Matlab-accessible functions to be\n   written in C.  For the most part, \"mex\" just wraps your C code, links\n   in the proper libraries and headers, and calls gcc on your behalf.\n   Make sure the \"mex\" program is in your path.\n\n6) (Semi-Required - needed for pyshark) Install Python and Python development\n   packages on your system.  It is likely that python is already installed\n   on your system.  The development packages can be downloaded and installed\n   as simply as:\n     apt-get install python-dev\n     yum install python-devel\n\nClearly, you will want select at least one of Step 5 or Step 6. By default,\nneither are created.  You can selectively enable your choice by\npassing --enable-{py,mat}shark to Sharktool's ./configure.\n\nOnce Glib, Wireshark, Matlab and Python have been installed:\n\n  :~$ cd /path/to/wireshark-x.y.z\n\n  :/path/to/wireshark-x.y.z$ ./configure --disable-wireshark\n\n  :/path/to/wireshark-x.y.z$ cd /path/to/sharktools\n\n  :/path/to/sharktools$ ./configure --with-wireshark-src=/path/to/wireshark-x.y.z --enable-pyshark --enable-matshark\n\n  :/path/to/sharktools$ make\n\n  :/path/to/sharktools$ mv matshark.\u003csuffix\u003e /path/to/your/matlab/path\n\nWhere \u003csuffix\u003e is:\n* \"mexglx\" on 32-bit Linux\n* \"mexa64\" on 64-bit Linux\n* \"mexmaci\" on 32-bit MacOSX\n* \"mexmaci64\" on 64-bit MacOSX\n\nYou can add an arbitrary directory to your matlab path by adding the\nfollowing lines to your ~/matlab/startup.m file and restarting Matlab:\n\n        % Add matshark to Matlab's path\n        addpath /path/to/matshark\n\n  :/path/to/sharktools$ mv pyshark.so /path/to/your/pythonmodules\n\nThe PYTHONPATH environment variable is searched by the python interpreter for\nexternal Python modules.  Be sure to run:\n\n  $ export PYTHONPATH=/path/to/your/pythonmodules\n\nTo test this out, run:\n\n  % matlab\n  \u003e\u003e matshark\n  ??? Must provide filename, cell array of fieldnames, and display filter\n  \n  \u003e\u003e\n\n= FAQ/Troubleshooting =\n\nQ: When I try to run matshark in Matlab, I get an error about \n   libwireshark/libwiretap not being found! What's wrong?\n\nA: Make sure you have Wireshark libraries installed.  Usually a distribution\n   will put them in /usr/lib, but if you know they are definitely somewhere\n   else, set your LD_LIBRARY_PATH before running Matlab.\n\n   NB: pyshark may have this same problem, and the solution is the same.\n\nQ: mex is giving me an error:\n    Warning: You are using gcc version \"3.4.6\".  The earliest gcc version\n    supported with mex is \"4.0.0\".  The latest version tested for use with mex\n    is \"4.2.0\". How do I fix this problem?\n\nA: Either upgrade your gcc or downgrade your Matlab, because the binary output\n   might not work.\n\n   In particular, keep in mind the latest version of gcc that RHEL4 provides\n   is 3.4.6.  RHEL5 provides gcc 4.*\n\nQ: MacOSX support?\n\nA: There has been a successful port of sharktools to MacOSX; see the\n   README.MacOSX file for details.\n\nQ: Windows support?\n\nA: No effort has been made to port this tool to Windows.  Sticking to a\n   un*x-like Operating system is probably your best bet, but patches are\n   certainly welcome.\n\n= Notes and General Design Information =\n\nThis tool is comprised of two pieces:\n\n1) A \"core\" which exports the functionality of libwireshark into a simple API.\n   This core is compiles into libsharktools.a, a static library which dynamically\n   links to libwireshark.so and libwiretap.so.\n\n2) An environment-specific portion which links to either the Matlab or Python\n   environments.\n\n   In Matlab, the output of this is matshark.mex{glx, a64}, which is the Matlab\n   module that is the final product. This Matlab module staticly links to\n   libsharktools.a. The glx vs. a64 extension is Matlab's way of identifying\n   32-bit vs. 64-bit code on Linux.  Other OS's will have different extensions.\n\n   In Python, the output is pyshark.so, which is a shared library that is\n   dynamically loaded by the python interpreter\n\nIn original revisions of this tool, matshark operation was as follows:\n\n   1) Open a (potentially giant) pcap file\n   2) Read the whole thing into memory as a linked list\n   3) Close the pcap file\n   4) Create a memory structure for the interpreted programming language\n   5) Copy from the giant linked list to the interpreted language's native structure\n   6) Delete the internal linked list\n   7) Return control back to the interpreter.\n\nThis approach was simple, but was very memory inefficient.  Since then,\nsharktools has evolved to implement a set of callbacks that are registered by\nthe environment-specific portion of the code, and run by the \"core\".  This\napproach reduces the overall memory footprint of the tool at the cost of some\ncomplexity (and in the case of Matlab, time; see matshark.c for notes).\n\nThis tool attempts to create native objects in the host environment and\nefficiently copy data to them.  For this reason, we have different copy\nconversion routines for different data types (e.g. ints vs. doubles). Some\ntypes, e.g. MAC addresses, have no native type, so they are simply copied\nas strings.\n\n== Versioning nightmare? ==\n\nYou may have noticed that this tool seems very particular about the versions\nof gcc, Matlab, and Wireshark that are used.  Unfortunately this is necessary.\n\n=== Wireshark versioning problem ===\n\nThe first problem is that Wireshark is provided as a monolithic package with\n1) Executable binaries (\"wireshark\", \"tshark\", \"rawshark\", etc), and\n2) Some dynamic libraries (\"libwireshark.so\", \"libwiretap.so\", etc.) that are\n   used by those executables.\n\nUnfortunately, the Wireshark project does this for memory efficiency and not\nfor modularity: the API between the executables and the libraries has the\npotential to change with every release of Wireshark, which makes dealing with\nlibwireshark.so itself a version-dependent effort.  The configure script\nknows how to deal with some specific versions, and tries to figure out\nwhat version of Wireshark is being used, and passes appropriate -D tags to\nthe compiler to include and exclude chunks of code based on whats needed for\neach of these versions.  This technique may not be possible in the future\nif/when the Wireshark decides to radically change their API.  Hopefully\nthey'll eventually come to a stable API and commit to providing some backwards\ncompatibility in the future.\n\nSee above for the versions we've tested on.\n\n=== Matlab versioning problem ===\n\nEach version of Matlab comes with a version of mex, which is its external\nmodule building script/tool.  mex started requiring newer versions of gcc\nbetween R2006* and R2007*, and RHEL4 does not provide these, whereas RHEL5\ndoes.\n\n== Future work ==\n\nPython:\n* More/better unit tests for pyshark\n\nMatlab:\n* Add support for Matlab iterators in matshark\n* At least some unit tests for matshark (there's an MUNIT test framework)\n\nPerl:\n* Integrate perlshark code that someone else already wrote\n\nGeneral:\n* Fixing memory leaks, of course.\n* Integrate into main Wireshark package as extshark\n* ??? Suggestions are welcome!\n\nThese past TODO items have already been addressed:\n\n*The Python implementation currently does not use Python iterators.  By doing\n*so, we could be much more memory efficient. More information about python\n*iterators is at:\n*\n*  http://www.ibm.com/developerworks/library/l-pycon.html\n*  http://heather.cs.ucdavis.edu/~matloff/Python/PyIterGen.pdf\n\n*At the moment, if a dissector field name appears more than once in a packet,\n*only one of the fields is displayed (usually the latter field).  Future work\n*could involve returning an ordered list of these processed packet fields.\n\n*Identified inefficiency:  some data types are rendered as strings, only to be\n*rendered back by pyshark or matshark.  This definitely does not need to happen\n*for certain classes of data types (integers and floats in particular).\n\n*The wireshark engine uses callbacks to get information. These callbacks could\n*call back into the interpreted language modules and create native data\n*structures, and this technique could greatly reduce the amount of memory taken\n*used by the module.\n\n*The Matlab module uses a superlinear amount of memory (with respect to pcap\n*file size). This is probably a fault of this module, but apparently Matlab\n*could be the problem (since it has a reputation for leaking memory).  Nathan\n*has a hack around this right now (tshark_read_block), but fixing here would\n*be the best idea.\n\n= Other Notes =\n\nsharktools generally runs CPU-bound and not IO-bound.\n\nAs previously mentioned, mex calls gcc on a user's behalf.  Unfortunately,\nfor whatever reason, mex passes the -ansi flag to gcc by default, which\nprevents the use of //-style comments.  If a user really wants to use\n//-style comments, they will have to edit $MATLAB_HOME/bin/mexopts.sh.\nThe magic incantation can also be done via command line, via the MEXFLAGS\nenvironment variable:\n\nMEXFLAGS=\"-v -g CFLAGS='-fPIC -D_GNU_SOURCE -pthread -fexceptions -m32'\"\n\nNote that the default MEXFLAGS may change from version to version of Matlab;\nconsider basing your MEXFLAGS off the one contained in your MATLAB's\nmexopts.sh file.\n\n= Appendix: Finding Display Filter Names =\n\nThe easiest way to find Wireshark's dissector field names is by opening a\npacket capture in Wireshark, clicking on the field of interest, and looking\nat the status bar at the bottom of the wireshark window - the dissector\nfield name is the text in parenthesis.  This is true in Linux, anyway,\nand I'm pretty sure in MacOSX as well.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farmenb%2Fsharktools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farmenb%2Fsharktools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farmenb%2Fsharktools/lists"}