{"id":13453985,"url":"https://github.com/cisco/mercury","last_synced_at":"2025-05-15T23:06:09.504Z","repository":{"id":41142708,"uuid":"205461953","full_name":"cisco/mercury","owner":"cisco","description":"Mercury: network metadata capture and analysis","archived":false,"fork":false,"pushed_at":"2024-08-26T19:07:54.000Z","size":330209,"stargazers_count":443,"open_issues_count":14,"forks_count":75,"subscribers_count":31,"default_branch":"main","last_synced_at":"2024-10-29T15:48:17.847Z","etag":null,"topics":["fingerprinting","linux","network-security","packet-capture","packet-capture-and-analysis"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cisco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-30T21:58:25.000Z","updated_at":"2024-10-29T11:24:15.000Z","dependencies_parsed_at":"2024-11-28T14:04:23.590Z","dependency_job_id":"2d2577f0-1ed7-4dd0-8b8b-48e73df24725","html_url":"https://github.com/cisco/mercury","commit_stats":{"total_commits":2050,"total_committers":26,"mean_commits":78.84615384615384,"dds":"0.38195121951219513","last_synced_commit":"8c58000fd59af15a541887a82e2e2cfddb168a72"},"previous_names":[],"tags_count":29,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisco%2Fmercury","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisco%2Fmercury/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisco%2Fmercury/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cisco%2Fmercury/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cisco","download_url":"https://codeload.github.com/cisco/mercury/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254436944,"owners_count":22070946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fingerprinting","linux","network-security","packet-capture","packet-capture-and-analysis"],"created_at":"2024-07-31T08:00:50.057Z","updated_at":"2025-05-15T23:06:04.481Z","avatar_url":"https://github.com/cisco.png","language":"C++","funding_links":[],"categories":["Threat Detection and Hunting","C++","linux","\u003ca name=\"cpp\"\u003e\u003c/a\u003eC++"],"sub_categories":["Tools"],"readme":"# Mercury: network metadata capture and analysis\n\u003cimg align=\"right\" src=\"./mercury.png\" width=\"200\"\u003e\n\nThis package contains two programs for fingerprinting network traffic and capturing and analyzing packet metadata: **mercury**, a Linux application that leverages the modern Linux kernel's high-performance networking capabilities (AF_PACKET and TPACKETv3), which is described below, and [**pmercury**](python/README.md), a portable python application.  There is also a [User's Guide](https://github.com/cisco/mercury/wiki/Using-Mercury).  While mercury is used in some production applications, please consider this software as a 'beta'.  The [CHANGELOG](doc/CHANGELOG.md) itemizes changes across different versions.\n\n\n## Overview\n\nMercury reads network packets, identifies metadata of interest, and writes out the metadata in JSON format.  Alternatively, mercury can write out the packets that contain the metadata in the PCAP file format.  Mercury can scale up to high data rates (40Gbps on server-class hardware); it uses zero-copy ring buffers to acquire packets, and packets are processed by independent worker threads.  The amount of memory consumed by the ring buffers, and the number of worker threads, are configurable; this makes it easy to scale up (but be wary of using too much memory).\n\nMercury produces fingerprint strings for TLS, DTLS, SSH, HTTP, TCP, and other protocols; these fingerprints are formed by carefully selecting and normalizing metadata extracted from packets (as documented [here](doc/npf.md)).  Fingerprint strings are reported in the \"fingerprint\" object in the JSON output.  Optionally, mercury can perform process identification based on those fingerprints and the destination context; these results are reported in the \"analysis\" object.\n\n# Contents\n\n* [Building and installing mercury](#building-and-installing-mercury)\n   * [Installation](#installation)\n   * [Compile-time options](#compile-time-options)\n* [Running mercury](#running-mercury)\n  * [Details](#details)\n  * [System](#system)\n  * [Examples](#examples)\n* [Ethics](#ethics)\n* [Credits](#credits)\n* [Acknowledgements](#acknowledgments)\n\n\n## Building and installing mercury\n\nMercury itself has minimal dependencies other than a g++ or llvm build environment, but to run the automated tests and ancillary programs in this package, you will need to install additional packages, as in the following Debian/Ubuntu example:\n```\nsudo apt install g++ jq git zlib1g-dev tcpreplay valgrind python3-pip libssl-dev clang\npip3 install jsonschema\n```\nTo build mercury, in the root directory, run\n```\n./configure\nmake\n```\nto build the package (and check for the programs and python modules required to test it).  TPACKETv3 is present in Linux kernels newer than 3.2.\n\n### Installation\nIn the root directory, edit mercury.cfg with the network interface you want to capture from, then run\n```\n./configure\nmake\n\nsudo make install MERCURY_CFG=mercury.cfg\n```\nto install mercury and create and start a systemd service.  If you don't want the mercury systemd service to be installed, then instead run\n```\nsudo make install-nosystemd\n```\nThe default file and directory locations are\n   * __/usr/local/bin/mercury__ for the executable\n   * __/usr/local/share/mercury__ for the resource files\n   * __/usr/local/var/mercury__ for the output files\n   * __/etc/mercury/mercury.cfg__ for the configuration file\n   * __/etc/systemd/system/mercury.service__ for the systemd unit file\n\nThe output file directory is owned by the user **mercury**; this user is created by the 'make install' target, which must be run as root.  The installation prefix **/usr/local/** can be changed by running ./configure with the --prefix argument, for instance `--prefix=$HOME'.  If you want to install the program somewhere in your home directory, you probably don't want to create the user mercury; you should use the 'make install-nonroot' target, which does not create a user, does not install anything into /etc, and does not install a systemd unit.\n\nThe easiest way to run mercury in capture mode is using systemd; the OS automatically starts the mercury systemd unit after each boot, and halts it when the OS is shut down.  To check its status, run\n```\nsystemctl status mercury\n```\nand the output should contain 'active (running)'.  To view the log (stderr) output from the mercury unit, run\n```\nsudo journalctl -u mercury\n```\n\nTo uninstall mercury, run\n```\nsudo make uninstall\n```\nwhich will remove the mercury program, resources directory, user, group, and systemd related files.  The directory containing capture files will be retained, but its owner will be changed to root, to avoid unintentional data loss.  All captured data files are retained across successive installs and uninstalls, and must be manually deleted.\n\n### Compile-time options\nTo create a debugging version of mercury, use the **make debug-mercury** target in the src/ subdirectory.  Be sure to run **make clean** first.\n\nThere are compile-time options that can tune mercury for your hardware.  Each of these options is set via a C/C++ preprocessor directive, which should be passed as an argument to \"make\" through the OPTFLAGS variable.   Frst run **make clean** to remove the previous build, then run **make \"OPTFLAGS=\u003cDIRECTIVE\u003e\"**.   This runs make, telling it to pass \u003cDIRECTIVE\u003e to the C/C++ compiler.  The available compile time options are:\n\n   * -DDEBUG, which turns on debugging, and\n   * -FBUFSIZE=16384, which sets the fwrite/fread buffer to 16,384 bytes (for instance).\nIf multiple compile time options are used, then they must be passed to make together in the OPTFLAGS string, e.g. \"OPTFLAGS=-DDEBUG -DFBUFSIZE=16384\".\n\n## Running mercury\n```\nmercury: packet metadata capture and analysis\n./src/mercury [INPUT] [OUTPUT] [OPTIONS]:\nINPUT\n   [-c or --capture] capture_interface   # capture packets from interface\n   [-r or --read] read_file              # read packets from file\n   no input option                       # read packets from standard input\nOUTPUT\n   [-f or --fingerprint] json_file_name  # write JSON fingerprints to file\n   [-w or --write] pcap_file_name        # write packets to PCAP/MCAP file\n   no output option                      # write JSON fingerprints to stdout\n--capture OPTIONS\n   [-b or --buffer] b                    # set RX_RING size to (b * PHYS_MEM)\n   [-t or --threads] [num_threads | cpu] # set number of threads\n   [-u or --user] u                      # set UID and GID to those of user u\n   [-d or --directory] d                 # set working directory to d\nGENERAL OPTIONS\n   --config c                            # read configuration from file c\n   [-a or --analysis]                    # analyze fingerprints\n   --resources=f                         # use resource file f\n   --format=f                            # report fingerprints with formats(s) f\n   --stats=f                             # write stats to file f\n   --stats-time=T                        # write stats every T seconds\n   --stats-limit=L                       # limit stats to L entries\n   [-s or --select] filter               # select traffic by filter (see --help)\n   --nonselected-tcp-data                # tcp data for nonselected traffic\n   --nonselected-udp-data                # udp data for nonselected traffic\n   --reassembly                          # reassemble protocol messages over multiple transport segments\n   [-l or --limit] l                     # rotate output file after l records\n   --output-time=T                       # rotate output file after T seconds\n   --dns-json                            # output DNS as JSON, not base64\n   --certs-json                          # output certs as JSON, not base64\n   --metadata                            # output more protocol metadata in JSON\n   --raw-features                        # select protocols to write out raw features string(see --help)\n   [-v or --verbose]                     # additional information sent to stderr\n   --license                             # write license information to stdout\n   --version                             # write version information to stdout\n   [-h or --help]                        # extended help, with examples\n\nDETAILS\n   \"[-c or --capture] c\" captures packets from interface c with Linux AF_PACKET\n   using a separate ring buffer for each worker thread.  \"[-t or --thread] t\"\n   sets the number of worker threads to t, if t is a positive integer; if t is\n   \"cpu\", then the number of threads will be set to the number of available\n   processors.  \"[-b or --buffer] b\" sets the total size of all ring buffers to\n   (b * PHYS_MEM) where b is a decimal number between 0.0 and 1.0 and PHYS_MEM\n   is the available memory; USE b \u003c 0.1 EXCEPT WHEN THERE ARE GIGABYTES OF SPARE\n   RAM to avoid OS failure due to memory starvation.\n\n   \"[-f or --fingerprint] f\" writes a JSON record for each fingerprint observed,\n   which incorporates the flow key and the time of observation, into the file f.\n   With [-a or --analysis], fingerprints and destinations are analyzed and the\n   results are included in the JSON output.\n\n   \"[-w or --write] w\" writes packets to the file w, in PCAP format.  With the\n   option [-s or --select], packets are filtered so that only ones with\n   fingerprint metadata are written.\n\n   \"[r or --read] r\" reads packets from the file r, in PCAP format.\n\n   if neither -r nor -c is specified, then packets are read from standard input,\n   in PCAP format.\n\n   \"[-s or --select] f\" selects packets according to the metadata filter f, which\n   is a comma-separated list of the following strings:\n      arp               ARP message\n      bittorrent        Bittorrent Handshake Message, LSD message, DHT message\n      cdp               CDP message\n      dhcp              DHCP discover message\n      dnp3              DNP3 industrial control message\n      dns               DNS messages\n      dtls              DTLS clientHello, serverHello, and certificates\n      gre               GRE message\n      http              HTTP request and response\n      http.request      HTTP request\n      http.response     HTTP response\n      icmp              ICMP message\n      iec               IEC 60870-5-104\n      lldp              LLDP message\n      mdns              multicast DNS\n      mysql             MySQL Client/Server Protocol\n      nbns              NetBIOS Name Service\n      nbds              NetBIOS Datagram Service\n      nbss              NetBIOS Session Service\n      openvpn_tcp       OpenVPN over TCP\n      ospf              OSPF message\n      quic              QUIC handshake\n      sctp              SCTP message\n      ssh               SSH handshake and KEX\n      smb               SMB v1 and v2\n      smtp              SMTP client and server messages\n      stun              STUN messages\n      ssdp              SSDP (UPnP)\n      socks             SOCKS4,SOCKS5 messages\n      tcp               TCP headers\n      tcp.message       TCP initial message\n      tcp.syn_ack       TCP syn ack message\n      tls               TLS clientHello, serverHello, and certificates\n      tls.client_hello  TLS clientHello\n      tls.server_hello  TLS serverHello\n      tls.certificates  TLS serverCertificates\n      tofsee            Tofsee malware communication\n      wireguard         WG handshake initiation message\n      all               all of the above\n      \u003cno option\u003e       all of the above\n      none              none of the above\n\n   --nonselected-tcp-data writes the first TCP Data field in a flow with\n   nonzero length, for *non*-selected traffic, into JSON.  This option provides\n   a view into the TCP data that the --select option does not recognize. The\n   --select filter affects the TCP data written by this option; use\n   '--select=none' to obtain the TCP data for each flow.\n\n   --nonselected-udp-data writes the first UDP Data field in a flow with\n   nonzero length, for *non*-selected traffic, into JSON.  This option provides\n   a view into the UDP data that the --select option does not recognize. The\n   --select filter affects the UDP data written by this option; use\n   '--select=none' to obtain the UDP data for each flow.\n\n   --reassembly enables reassembly\n   This option allows mercury to keep track of tcp or udp segment state and\n   and reassemble these segments based on the application in payload\n \n   \"[-u or --user] u\" sets the UID and GID to those of user u, so that\n   output file(s) are owned by this user.  If this option is not set, then\n   the UID is set to SUDO_UID, so that privileges are dropped to those of\n   the user that invoked sudo.  A system account with username mercury is\n   created for use with a mercury daemon.\n\n   \"[-d or --directory] d\" sets the working directory to d, so that all output\n   files are written into that location.  When capturing at a high data rate, a\n   high performance filesystem and disk should be used, and NFS partitions\n   should be avoided.\n\n   \"--config c\" reads configuration information from the file c.\n\n   [-a or --analysis] performs analysis and reports results in the \"analysis\"\n   object in the JSON records.   This option only works with the option\n   [-f or --fingerprint].\n\n   \"--format=f\" reports fingerprints with formats(s) f, where f is either a\n   fingerprint protocol and format like \"tls/1\", or is a comma separated\n   list of below fingerprint protocol and format strings.\n       tls\n       tls/1\n       tls/2\n       quic\n       quic/1\n\n   \"[-l or --limit] l\" rotates output files so that each file has at most\n   l records or packets; filenames include a sequence number, date and time.\n\n   --dns-json writes out DNS responses as a JSON object; otherwise,\n   that data is output in base64 format, as a string with the key \"base64\".\n\n   --certs-json writes out certificates as JSON objects; otherwise,\n    that data is output in base64 format, as a string with the key \"base64\".\n\n   --metadata writes out additional metadata into the protocol JSON objects.\n\n   --raw-features selects protocols to write out raw features string into the protocol\n    JSON object which can be comma separated list of following strings:\n       bittorrent      Bittorrent Handshake Message, LSD message, DHT message\n       smb             SMB v2, v3 messages\n       ssdp            SSDP (UPnP)\n       stun            Stun messages\n       tls             TLS clientHello\n       all             All of the above\n       none            None of the above\n      \u003cno option\u003e     None of the above\n\n   [-v or --verbose] writes additional information to the standard error,\n   including the packet count, byte count, elapsed time and processing rate, as\n   well as information about threads and files.\n\n   --license and --version write their information to stdout, then halt.\n\n   [-h or --help] writes this extended help message to stdout.\n```\n\n### SYSTEM\nThe directories used by the default install are as follows.  Run **mercury --help** to\nsee if the directories on your system differ.\n```\n   Resource files used in analysis: /usr/local/share/mercury\n   Systemd service output:          /usr/local/var/mercury\n   Systemd service configuration    /etc/mercury/mercury.cfg\n```\n\n### EXAMPLES\n```\n   mercury -c eth0 -w foo.pcap           # capture from eth0, write to foo.pcap\n   mercury -c eth0 -w foo.pcap -t cpu    # as above, with one thread per CPU\n   mercury -c eth0 -w foo.mcap -t cpu -s # as above, selecting packet metadata\n   mercury -r foo.mcap -f foo.json       # read foo.mcap, write fingerprints\n   mercury -r foo.mcap -f foo.json -a    # as above, with fingerprint analysis\n   mercury -c eth0 -t cpu -f foo.json -a # capture and analyze fingerprints\n```\n\n## Ethics\nMercury is intended for defensive network monitoring, security research and forensics.  Researchers, administrators, penetration testers, and security operations teams can use these tools to protect networks, detect vulnerabilities, and benefit the broader community through improved awareness and defensive posture. As with any packet monitoring tool, Mercury could potentially be misused. **Do not run it on any network of which you are not the owner or the administrator**.\n\n## Credits\nMercury and this package was developed by David McGrew, Brandon Enright, Blake Anderson, Lucas Messenger, Adam Weller, Andrew Chi, Shekhar Acharya, Anastasiia-Mariia Antonyk, Oleksandr Stepanov, Vigneshwari Viswanathan, and Apoorv Raj, with input from Brian Long, Bill Hudson, and others.  Pmercury was developed by Blake Anderson, with input from others.\n\n## Acknowledgments\n\nThis package includes GeoLite2 data created by MaxMind, available from [https://www.maxmind.com](https://www.maxmind.com).\n\nWe make use of Mozilla's [Public Suffix List](https://publicsuffix.org/list/) which is subject to the terms of the [Mozilla Public License, v. 2.0](https://mozilla.org/MPL/2.0/).\n\nThis package directly incorporates some software made by other\ndevelopers, to make the package easier to build, deploy, and run.  We\nare grateful to the copyright holders for making their excellent\nsoftware available under licensing terms that allow its\nredistribution.\n   * RapidJSON\n      [https://github.com/cisco/mercury/src/rapidjson/license.txt](src/rapidjson/license.txt);\n      this package is Copyright 2015 THL A29 Limited, a Tencent company,\n      and Milo Yip.\n   * lctrie [https://github.com/cisco/mercury/src/lctrie](src/lctrie);\n      this package is copyright 2016-2017 Charles Stewart\n      \u003cchuckination_at_gmail_dot_com\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcisco%2Fmercury","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcisco%2Fmercury","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcisco%2Fmercury/lists"}