{"id":13531046,"url":"https://github.com/secureworks/dalton","last_synced_at":"2026-01-16T23:39:40.286Z","repository":{"id":26902263,"uuid":"100408000","full_name":"secureworks/dalton","owner":"secureworks","description":"Suricata, Snort and Zeek IDS rule and pcap testing system","archived":false,"fork":false,"pushed_at":"2025-09-18T20:31:35.000Z","size":2719,"stargazers_count":495,"open_issues_count":22,"forks_count":96,"subscribers_count":38,"default_branch":"master","last_synced_at":"2025-09-18T22:11:05.029Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/secureworks.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2017-08-15T18:41:49.000Z","updated_at":"2025-09-18T20:30:17.000Z","dependencies_parsed_at":"2024-01-07T01:13:46.102Z","dependency_job_id":"a7707990-668b-4e12-8a23-faa4fca18b96","html_url":"https://github.com/secureworks/dalton","commit_stats":null,"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"purl":"pkg:github/secureworks/dalton","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/secureworks%2Fdalton","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/secureworks%2Fdalton/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/secureworks%2Fdalton/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/secureworks%2Fdalton/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/secureworks","download_url":"https://codeload.github.com/secureworks/dalton/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/secureworks%2Fdalton/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28487812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T22:54:02.790Z","status":"ssl_error","status_checked_at":"2026-01-16T22:50:10.344Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T07:00:59.337Z","updated_at":"2026-01-16T23:39:40.270Z","avatar_url":"https://github.com/secureworks.png","language":"Python","funding_links":[],"categories":["\u003ca id=\"0abd611fc3e9a4d9744865ca6e47a6b2\"\u003e\u003c/a\u003e工具","\u003ca id=\"946d766c6a0fb23b480ff59d4029ec71\"\u003e\u003c/a\u003e防护\u0026\u0026Defense","\u003ca id=\"f13469c9891173804423be4403b2c4ff\"\u003e\u003c/a\u003epcap","Simulation and Testing"],"sub_categories":["\u003ca id=\"ff3e0b52a1477704b5f6a94ccf784b9a\"\u003e\u003c/a\u003eIDS\u0026\u0026IPS","\u003ca id=\"eb49514924c3f4bf2acf6f3a4436af13\"\u003e\u003c/a\u003e未分类"],"readme":"======\nDalton\n======\n\nDalton is a system that allows a user to quickly and easily run network\npacket captures (\"pcaps\") against an intrusion detection system (\"IDS\")\nsensor of his choice (e.g. Snort, Suricata) using defined rulesets\nand/or bespoke rules.\n\nDalton also includes a wizard-like web interface for\n`Flowsynth \u003chttps://github.com/secureworks/flowsynth\u003e`__ to facilitate\ncustom pcap creation.\n\n.. image:: app/static/images/dalton.png\n\n**Quickstart**:\n\n.. code:: bash\n\n    ./start-dalton.sh\n\nor this which does the same thing:\n\n.. code:: text\n\n    docker-compose build \u0026\u0026 docker-compose up -d\n\nThen navigate to ``http://\u003cdocker-host\u003e/dalton/``\n\nTo configure what rulesets are available, see \n`Adding Rulesets \u003c#adding-rulesets\u003e`__.\n\nTo configure what sensors are available, see \n`Adding Sensors \u003c#adding-sensors\u003e`__.\n\nIf Dalton is being built behind a proxy, see\n`Building Behind A Proxy \u003c#building-behind-a-proxy\u003e`__\n\nContents\n========\n\n-  `Use Cases \u003c#use-cases\u003e`__\n-  `Design \u003c#design\u003e`__\n-  `Requirements \u003c#requirements\u003e`__\n-  `Installing and Running Dalton \u003c#installing-and-running-dalton\u003e`__\n\n   -  `Building Behind A Proxy \u003c#building-behind-a-proxy\u003e`__\n\n-  `Using Dalton \u003c#using-dalton\u003e`__\n\n   -  `Launching A New Job \u003c#launching-a-new-job\u003e`__\n   -  `Suricata Socket Control Mode \u003c#suricata-socket-control-mode\u003e`__\n   -  `Job Settings \u003c#job-settings\u003e`__\n   -  `Config Files \u003c#config-files\u003e`__\n   -  `Job Results \u003c#job-results\u003e`__\n   -  `Job Queue \u003c#job-queue\u003e`__\n   -  `Sensors \u003c#sensors\u003e`__\n\n-  `Dalton API \u003c#dalton-api\u003e`__\n\n   -  `Job API \u003c#job-api\u003e`__\n   -  `Controller API \u003c#controller-api\u003e`__\n\n-  `Teapot Jobs \u003c#teapot-jobs\u003e`__\n-  `Adding Rulesets \u003c#adding-rulesets\u003e`__\n-  `Adding Sensors \u003c#adding-sensors\u003e`__\n\n   -  `Docker Sensors \u003c#docker-sensors\u003e`__\n   -  `Non-Docker Sensors \u003c#non-docker-sensors\u003e`__\n   \n-  `Adding Sensor Configs \u003c#adding-sensor-configs\u003e`__\n-  `Logging and Debugging \u003c#logging-and-debugging\u003e`__\n-  `Flowsynth WebUI \u003c#flowsynth-webui\u003e`__\n-  `Zeek \u003c#zeek\u003e`__\n-  `CyberChef \u003c#cyberchef\u003e`__\n-  `Frequently Asked Questions \u003c#frequently-asked-questions\u003e`__\n-  `Authors \u003c#authors\u003e`__\n\nUse Cases\n=========\n\nThese are the most common use cases for Dalton:\n\n-  | **Testing Rulesets/Ruleset Coverage**\n   | User-provided pcaps can be run thru an IDS engine loaded with a \n     particular ruleset.\n\n-  | **Troubleshooting and Developing Signatures**\n   | User-provided pcaps can be tested against user-provided ad hoc IDS\n     rules to quickly and easily see the IDS alerts and/or test for rule\n     syntax errors.\n\n-  | **Testing Variable Changes**\n   | The ruleset variables used by the engine can easily be modified\n     for submitted jobs; this can be used to determine the impact that a\n     variable change may have on a specific detection.\n\n-  | **Testing Configuration Changes**\n   | Customized engine configuration files can included with submitted\n     jobs; this can be used to determine the impact that an engine\n     configuration change may have on a specific detection.\n\n-  | **Testing specific IDS engine behavior**\n   | Dalton supports the ability to apply the above use cases on\n     specific sensors. The Dalton architecture is designed to accommodate\n     and support sundry sensor engines and engine versions.\n\n-  | **Crafting custom packet captures**\n   | As part of the Web interface, Dalton has a module that provides a\n     wizard-like web interface for Flowsynth. This allows for quick and\n     easy network flow definition and pcap creation for popular protocols\n     and traffic patterns.\n\nDesign\n======\n\nDalton consists of a “controller” (dalton.py) and “agents”\n(dalton-agent.py). The controller provides a web interface as well as a\nHTTP API for agent communication and programmatic job results retrieval.\nFrom a web interface, a user submits a job to be run on a particular\nagent or agent platform. A Dalton job consists of one or more pcaps, a\npre-defined ruleset and/or custom rules, agent engine configuration\noptions (e.g. configuration to apply to Suricata when running a job),\nand a manifest file specifying other options for the job (e.g. return\nrule performance logs).\n\nThe Dalton Agent code (dalton-agent.py) runs on an IDS sensor and\nprovides an interface between the Dalton controller and an IDS engine.\nDalton agents grab pending jobs from the Dalton controller, run them\nlocally, and report the results back. These results are then displayed\nin the web GUI provided by the Dalton controller. Jobs are submitted to\nspecific sensor engines (e.g. Suricata) and versions (e.g. 4.0.0).\n\nCode for the Dalton agent and controller webapp are written in Python\nand leverage `Flask \u003chttp://flask.pocoo.org/\u003e`__ and\n`Jinja2 \u003chttp://jinja.pocoo.org/\u003e`__. On the Dalton controller,\n`Redis \u003chttp://www.redis.io\u003e`__ is used to manage the job queue, store\nresults, and maintain a list of active Dalton agents.\n\nThe Dalton controller includes a\n`Flowsynth \u003chttps://github.com/secureworks/flowsynth\u003e`__ WebUI module\nthat provides a user interface to assist with rapid Flowsynth language\nprototyping and development of network flow definitions that are then\ncompiled into network pcaps by the Flowsynth script. This is basically a\nGUI to facilitate input and output to Flowsynth. There is the option to\neasily send Flowsynth WebUI generated pcaps to Dalton for testing.\n\nWhile all the above systems could be independent physical (or virtual)\nmachines (and in fact this setup has been done), for ease of install and\nuse, everything has also been architected as a group of\n`Docker \u003chttps://www.docker.com/\u003e`__ containers. The Dalton codebase\nincludes Dockerfiles, “docker-compose.yml”, and associated\nconfiguration files to facilitate easy application launch using a set of\nDocker containers.\n\nRequirements\n============\n\n-  `Docker \u003chttps://www.docker.com/get-docker\u003e`__\n-  `Docker Compose \u003chttps://docs.docker.com/compose/install/\u003e`__\n-  Internet connection (to build)\n\nInstalling and Running Dalton\n=============================\n\nThe easiest way to get Dalton up and running is to use the Docker files\nprovided and launch the system as a group of Docker containers. From\nthe root of the repository, run:\n\n.. code:: bash\n\n    ./start-dalton.sh\n\nor this which does the same thing:\n\n.. code:: bash\n\n    docker-compose build \u0026\u0026 docker-compose up -d\n\nTo specify or add what agents (specific sensors and versions) are built\nand run, edit the docker-compose.yml file as appropriate. See also\n`Adding Sensors \u003c#adding-sensors\u003e`__.\n\nThe HTTP listen port can be changed if desired by editing the\n``DALTON_EXTERNAL_PORT`` value in the .env file in the root of the\nrepository.\n\nConfiguration options for the Dalton Controller can be found in ``dalton.conf``; \nConfiguration options for Dalton Agents can be found in \n``dalton-agent/dalton-agent.conf``.  See the inline comments in those files for \nmore details.\n\n\nBuilding Behind A Proxy\n-----------------------\n\nIt is recognized that getting systems to work behind a corporate proxy can be an endless source of\nacute frustration and ongoing consternation.  However, a small attempt\nhas been made to make it easier for Dalton to be built behind a proxy. Note that\nit comes with no guarantees.\n\nTo build Dalton behind a proxy, most likely Docker and\nthe containers will need to be set up to use the proxy.\n\nConfiguring Docker to use a proxy will vary depending on the platform\nDocker is run on.  For Linux, it usually involves editing the\n``/etc/default/docker`` file, or if systemd is used (as it is in Ubuntu 16.04),\nsee `https://docs.docker.com/engine/admin/systemd/ \u003chttps://docs.docker.com/engine/admin/systemd/\u003e`__.\nThis is for *Docker*, not the\nDocker containers.  This allows Docker to do things like pull (external) images\nfrom the Docker Hub Registry.\n\nTo build the Dalton containers behind a proxy, edit the ``.env`` file\nin the Dalton repository root and set the ``http_proxy``, ``https_proxy``, and/or ``no_proxy``\nvariables accordingly.  Example:\n\n.. code:: bash\n\n    http_proxy=http://192.168.1.50:3128\n    https_proxy=http://192.168.1.50:3128\n    no_proxy=\n\nBe aware that DNS may not work in which case the IP of the\nproxy will need to be used.\n\nThese environment variables will be used when containers are\n*built*.  This will allow the container to do things like\n'apt-get install...'; they are used *inside* the container,\nnot by docker to pull (external) images.\n\nNote that these environment variables do not persist after the\ncontainer is built.  This means that if there are no rulesets,\nand Dalton attempts to download default rulesets, it will most\nlikely fail and result in an empty file.  In this case rulesets\nwill need to be added (and the empty files removed);\nsee `Adding Rulesets \u003c#adding-rulesets\u003e`__.\n\nEnabling SSL/TLS on the Controller\n----------------------------------\nThe Dalton Controller web interface supports SSL/TLS.  To enable,\nset the ``DALTON_EXTERNAL_PORT_SSL`` variable in the ``.env`` file\nto the desired SSL/TLS listen port; by default it is 443.  Then,\nmodify the \"nginx\" section of the ``docker-compose.yml`` and uncomment\n(or add if it is missing) the line:\n\n.. code:: bash\n\n             - DALTON_EXTERNAL_PORT_SSL=${DALTON_EXTERNAL_PORT_SSL}\n\nThe Dalton Controller comes with a default certificate and key but\nthese should be replaced.  The certificate and key files should be\nplaced in the ``nginx-conf/tls/`` directory and named ``dalton.crt``\nand ``dalton.key``, respectively.\n\n\nUsing Dalton\n============\n\nLaunching A New Job\n-------------------\n\nThe job submission page can be navigated to via the \"New\" menu on the\ntoolbar, or by clicking the ``[Go \u003e\u003e]`` button on the homepage below a given\nsensor technology. The user will be prompted to select the sensor to be\nused, supply a packet capture and ruleset (pre-defined and/or custom),\nand given the ability to configure other options using the vertical\ntab(s) on the submission page. On the 'Config Files' tab a user can\nmodify the sensor configuration file.\n\nPlease be aware that in most rulesets, almost all rules looking at TCP\ntraffic are set to inspect established sessions. This means that if a\npcap is supplied that only contains a single packet (e.g. from a sensor\nor firewall technology that only logs a single packet), it will not\nalert on these rules because the sensor will not see it as an\nestablished session because of the lack of a TCP 3-way handshake. If\ntesting such a packet is desired, it will need to be incorporated into a\nnew pcap that includes a 3-way handshake and the server and client IPs\nset correctly. This can be done fairly easily using Flowsynth; the\n`Flowsynth Web UI \u003c#flowsynth-webui\u003e`__ makes this easy.\n\nSuricata Socket Control Mode\n----------------------------\n\nDalton Agents running Suricata 3.0 and later are capable of using the\n`Suricata Socket Control \u003chttps://suricata.readthedocs.io/en/latest/manpages/suricatasc.html\u003e`__\nmode to process pcaps instead of starting up a new Suricata process for each job\nand using pcap replay mode.  Leveraging the socket control feature of Suricata\noffers significant job performance gains (reduced job runtime) when the\nruleset and config do not change between jobs on an agent, since the overhead\nof starting up Suricata and processing the ruleset is eliminated.\n\nTo enable Suricata Socket Control select ``Use Suricata Socket Control Pcap Processing Mode``\non the job submission page, located in the ``Sensor Version`` section of the ``Job Settings``\nvertical tab.\n\nIf the Dalton agent is unable to use Suricata Socket Control for a job, it will\nuse the classic read pcap mode.\n\nIf ``Rule profiling`` is enabled, then Suricata Socket Control\nmode will be disabled for that job since the rule profiling and\nkeyword profiling logs do not get populated (or usually do not have\nenough time to be populated) for socket control pcap runs.\n\nThe Suricata Socket Control mode leverages the ``suricatasc`` Python\nmodule included with the Suricata source.  If the agent was built\nas a Docker container using the Dockerfile(s) provided, then the\n``suricatasc`` Python file(s) should already be there and the\nagent aware of them.  If not, or if the module is not in PYTHONPATH,\nthen the ``SURICATA_SC_PYTHON_MODULE`` config item in the\n``dalton-agent.conf`` file can be set to point to correct location.\n\nWhile Socket Control is supported by Suricata in versions 1.4 and later,\nthe ``suricatasc`` module was not Python 3 compatible until Suricata\n3.0 so that is the earliest version Dalton supports.\n\n-  | **Problems with Suricata Socket Control Mode**\n   | There are some known issues with Suricata Socket Control, not related to Dalton.\n     If problems are encountered\n     with it, try running the job with this option disabled.\n\n   -  | **Sample Issues**\n      | `Docker Suricata Socket Control crashing using command 'reopen-log-files \u003chttps://redmine.openinfosecfoundation.org/issues/3436\u003e`__\n\n      | `Suricata 4.1 Seg Fault: Socket Control pcap-file and corrupt pcap \u003chttps://redmine.openinfosecfoundation.org/issues/3448\u003e`__\n\n      | `Alert metadata not present in EVE output when using Socket Control Pcap Processing Mode \u003chttps://redmine.openinfosecfoundation.org/issues/3467\u003e`__\n\nJob Settings\n------------\n\nOn the job submission page, the \"Job Settings\" vertical tab provides a\nnumber of user-configurable options:\n\n-  | **Packet Captures**\n   | Specify packet captures (libpcap format) to be run across the\n     sensor. Depending on the engine, pcapng format may be supported as\n     well. Archive files that contain pcaps can be submitted and the files\n     will be extracted and used. Supported extensions (and their inferred\n     formats) are .zip, .gz, .gzip, .bz2, .tar, .tgz, and .tar.gz. Since\n     zip and tar files can contain multiple files, for those formats only\n     members that have the \".pcap\", \".pcapng\", or \".cap\" extensions will\n     be included; the other files will be ignored. Password protected zip\n     files will be attempted to be decrypted with the password 'infected'.\n\n   | If multiple pcaps are submitted for a Suricata job, they will be \n     combined into a single pcap on job submission since (older versions of) Suricata can\n     only read a single pcap in read pcap mode.\n\n   -  | **Create separate jobs for each pcap**\n      | If selected, each pcap file submitted (or found in an archive) will be\n        submitted as its own job.  When all the jobs are submitted, Dalton will\n        redirect the user to the Queue page.  If this is a `Teapot job \u003c#teapot-jobs\u003e`__,\n        then a comma separated list of JIDs is returned.\n\n-  | **Sensor Version**\n   | The specific sensor version to use to run the specified pcap(s)\n     and rule(s).\n\n   -  | **Use Suricata Socket Control Pcap Processing Mode**\n      | See `Suricata Socket Control Mode \u003c#suricata-socket-control-mode\u003e`__ section.\n\n-  **Ruleset**\n\n   -  | **Use a production ruleset**\n      | Select which \"production\" (pre-defined) ruleset to use if this\n        option is checked. See also `Adding\n        Rulesets \u003c#adding-rulesets\u003e`__.\n\n      -  | **Enable disabled rules**\n         | Enable all disabled rules. This may cause engine errors if\n           variables in disabled rules are not defined.\n      -  | **Show all flowbit alerts**\n         | Rules that have, ``flowbit:noalert`` will have that directive\n           removed so that they show up in the sensor alerts.\n\n   -  | **Use custom rules**\n      | This allows a user to specify specific ad hoc rules to include\n        when testing the pcap(s). The user will need to ensure that any\n        custom rules are valid since very little rule syntax validation is\n        done on the Dalton controller; submitting invalid rules will\n        result in verbose errors from the Dalton Agent (sensor engine)\n        being used, which can facilitate rule syntax troubleshooting.\n        Custom rules are added to a ``dalton-custom.rules`` file and included in the job\n        so valid format is supported such as multiple rules (one on\n        each line), and comments (ignored lines) beginning with a pound\n        ('#') sign. If a ``sid`` is not provided for a custom rule, one will be added\n        when the job is submitted.\n\n-  **Logs**\n\n   -  | **Pcap records from alerts (unified2)**\n      | This tells the agent to process unified2 alert data and if alerts\n        are generated by the job, this information will show up under the \n        \"Alert Details\" tab on the job results page. Information returned\n        includes hex/ASCII output from packets that generated alerts as\n        well as \"Extra\" data from the unified2 file such as \"Original\n        Client IP\" from packets with \"X-Forwarded-For\" or \"True-Client-IP\"\n        HTTP headers (if enable\\_xff is configured on the sensor).\n        Note that Suricata version 6 and later does not support unified2\n        output so this option is unavailable for jobs to such agents.\n   -  | **EVE Log**\n      | *Suricata only*, version 2 and later.  Turn on (or off, if not checked)\n        EVE logging and return the results.\n        The specific EVE log types, settings, etc. are determined by\n        (and can be set in) the config file.\n        Since Suricata version \u003c 3.1\n        doesn't support multiple TLS loggers, TLS logging in the EVE log\n        is disabled for jobs submitted to such agents.\n        The maximum supported\n        size for the EVE log is 512MB; see note about 512MB limit for\n        'Other logs'.\n   -  | **Other logs (Alert Debug, HTTP, TLS, DNS, etc.)**\n      | *Suricata only*.  This will return other logs generated by the\n        engine that can be useful for analysis and debugging.\n        Depending on the version\n        of Suricata running on the agent, some logs may not be supported.\n        Like all results, the 'Other logs' data is stored in Redis as a\n        string and the maximum size this can be is 512MB. If these logs\n        exceed that size, there may be data loss and/or other issues.\n        Currently the following logs are returned, each in it's own tab,\n        and if the log file is empty, the tab won't be shown:\n\n      -  | **Engine Stats** (*always returned even if this option is not\n           checked*)\n         | Statistics from the engine including numbers about memory,\n           flows, sessions, reassembly, etc.\n      -  | **Packet Stats** (*always returned even if this option is not\n           checked*)\n         | Statistics from the pcap including network protocols,\n           application layer protocols, etc.\n      -  | **Alert Debug**\n         | Detailed information on what particular rules matched on for\n           each alert.  Useful for seeing why an alert fired and/or\n           troubleshooting false positives.\n      -  | **HTTP Log**\n         | A log of HTTP requests and responses, showing time, IPs and\n           ports, HTTP method, URI, HTTP version, Host, User-Agent,\n           Referer, response code, response size, etc.  By default, each\n           line represents the HTTP request and response all in one.\n      -  | **DNS Log**\n         | A log of DNS requests and responses as provided by Suricata.\n           This won't be available if Suricata is compiled with Rust support\n           or if the version of Suricata is 5.0 or later.\n      -  | **TLS Log**\n         | A log of SSL/TLS traffic as provided by Suricata.\n   -  | **Dump buffers (alerts only)**\n      | This will display the contents of buffers used by the detection\n        engines, which can be useful for troubleshooting signature creation with traffic\n        that may not be parsing as expected. Since such output can be voluminous,\n        only buffer content associated with alerts are returned.  To see buffer content from\n        more traffic, use rule(s) that match on more traffic (or even\n        a generic rule that matches on all traffic).\n        Snort will output buffer contents into a \"Buffer Dump\" log output.\n        Suricata works differently and will place contents into \"HTTP Buffers\",\n        \"TLS Buffers\" and/or \"DNS Buffers\". These are Lua script outputs\n        intended to be visually similar than the Snort buffer dump output.\n        However on Suricata the protocol must be specified for the buffer dump\n        to work. Examples: ``alert http``, ``alert tls``, ``alert dns``.\n   -  | **Rule profiling**\n        Return per-rule performance statistics. This is data from the\n        engine's rule performance profiling output. This data will show up\n        under the \"Performance\" tab on the job results page.\n   -  | **Fast pattern info**\n\n      -  *Suricata only*. Return fast pattern data about the submitted\n         rules.  The Dalton Suricata agent will return a file (displayed\n         in the \"Fast Pattern\" tab) with details on what the engine is\n         using for the fast pattern match.  To generate this, Suricata\n         must do two runs – one to generate the fast pattern info and\n         one to actually run the submitted job so this will approximately\n         double the job run time. Unless fast pattern info is needed for\n         some reason, there isn't a need to check this. Fast pattern\n         data can be voluminous so it is not recommended that this be\n         selected for a large production/pre-defined ruleset.\n\nConfig Files\n------------\n\nOn the job submission page, the \"Config Files\" vertical tab provides the\nability to edit the configuration file(s) for the sensor:\n\n-  | **Configuration File**\n   | The engine configuration file, including variables, that the\n     Dalton agent uses for the job.\n\nIf the ``Override EXTERNAL_NET (set to 'any')`` option is selected\n(on by default), then the ``EXTERNAL_NET`` IP variable will be set to\n``any`` when the job is submitted.\n\nSee also `Updating Sensor Configs \u003c#updating-sensor-configs\u003e`__. \n\nJob Results\n===========\n\nThe job results page allows users to download the job zip file and also\npresents the results from the job run in a tabulated interface:\n\n-  | **Alerts**\n     These are the raw alerts from the sensor.\n-  | **Alert Details**\n   | If ``Include Detailed Alerts`` is selected for a job, detailed output\n     from processing unified2 alert files will be shown here.\n-  | **EVE JSON** (Suricata only)\n   | The EVE log, with syntax highlighting, if EVE logging is enabled.\n     The ``Format`` checkbox\n     \"pretty-prints\" the EVE data; the ``Dark Mode`` checkbox applies\n     a dark mode theme/coloring to the EVE data.  The UI also dynamically\n     presents checkboxes based on the event types present in the EVE log.\n     These can be used to filter the displayed EVE data.\n     If the EVE data is more than 2000000 bytes, then by default the\n     ``Dark Mode`` option is\n     disabled and syntax coloring is turned off, for performance reasons.\n-  | **IDS Engine**\n   | This the raw output from the IDS engine. For Snort jobs, the engine\n     statistics will be in this tab, at the bottom.\n-  | **Performance**\n   | If ``Rule profiling`` is enabled, those results will be\n     included here.\n-  | **Debug**\n   | This is the Debug output from the agent.\n-  | **Error**\n   | If any errors are encountered by the Dalton agent running the job,\n     they will be returned and displayed in this tab and the tab will be\n     selected by default. If there are no errors, this tab will not be\n     shown.\n-  | **Other logs**\n   | If other logs are returned by the agent they will each be displayed\n     in their own tab if they are non-empty.  ``Engine Stats`` and ``Packet\n     Stats`` are always returned for Suricata jobs.  See discussion in the\n     above \"Configuration Options\" discussion for more details.\n\nJob Queue\n=========\n\nSubmitted jobs can be viewed on the \"Queue\" page. Each test is assigned\na quasi-unique sixteen byte Job ID, which is based on the job's runtime\nparameters. Each recent Job ID is included on the 'Queue' page as a\nhyperlink for easy access. Queued jobs will be cleared out periodically \nif an agent has not picked them up; this should not happen unless\nall agents are down or are unreasonably backlogged.  There is additional\nlogic in the Dalton controller to respond appropriately when jobs have\ntimed out or have been interrupted; this should happen rarely, if ever.\n\nJob results are cleared out periodically as well; this option is\nconfigurable with the ``redis_expire`` parameter in the ``dalton.conf`` file.\n`Teapot jobs \u003c#teapot-jobs\u003e`__ expire timeouts are \nconfigured with the ``teapot_redis_expire`` option.\nAfter a job has completed, the original job can always be viewed (if it\nhasn't expired) by accessing the following url::\n\n  /dalton/job/\u003cjobid\u003e\n\nA job zip file, which includes the packet capture file(s) submitted\nalong with rules and variables associated with the job, is stored on\ndisk, by default in the ``/opt/dalton/jobs`` directory; this location is\nconfigurable via the ``job_path`` parameter in the ``dalton.conf`` file.\nThese files are cleaned up by Dalton based on the ``redis_expire`` and\n``teapot_redis_expire``. Visiting a job's share link increases the expire\ntime for the job zip file. How long the expire time is extended can be\nconfigured in the ``dalton.conf`` file as well with the ``share_expire``\nconfiguration option. Dalton only cleans up job zip files from disk when the\n``Queue`` page is loaded. To force the clean up job to run on demand, send\na HTTP GET request to::\n\n  /dalton/controller_api/delete-old-job-files\n\nA job zip file can be accessed from the appropriate link on the job results \npage or directly downloaded using the following URL::\n\n  /dalton/sensor_api/get_job/\u003cjobid\u003e.zip\n\nSensors\n=======\n\nAgents (a.k.a. \"Sensors\") check into the Dalton server frequently\n(about every second but configurable in the ``dalton-agent.conf`` file). The \nlast time an agent checked in can be viewed on the ``Sensors`` page. Agents\nthat have not checked in recently will be pruned based on the \n``agent_purge_time`` value in the ``dalton.conf`` config file. When an\nexpired or new agent checks into the Dalton Controller it will be\nautomatically (re)added and made available for job submissions.\n\nDalton API\n==========\n\nJob API\n-------\n\nThe Dalton controller provides a RESTful API to retrieve data about\nsubmitted jobs.  API responses use JSON or the raw (\"RAW\") data, and\nthe data returned in the values is, \nin most cases, just the raw text that is displayed in the Dalton web interface.\n\n**JSON API**\n\nThe JSON API can be utilized via HTTP GET requests in this format::\n\n  GET /dalton/controller_api/v2/\u003cjobid\u003e/\u003ckey\u003e\n\nFor requests, ``\u003cjobid\u003e`` is the Job ID and::\n\n    \u003ckey\u003e : [alert|alert_debug|alert_detailed|all|debug|dns_log|\n             error|engine_stats|eve|fast_pattern|http_log|ids|\n             keyword_perf|other_logs|packet_stats|perf|start_time|\n             statcode|status|submission_time|tech|time|tls_log|user]\n\nA JSON API request returns JSON with three root elements:\n\n-  | **data**\n   | The requested data.  If the key is invalid for the\n     job, then an error is returned, along with an error message stating\n     as such. If there is no data for the requested Job ID and key, then\n     this ``data`` parameter value is an empty string and ``error`` is set\n     to false..\n\n-  | **error**\n   | [true\\|false] depending if the API request generated an error. This is\n     not returned as a quoted string.  \\ **This** **indicates an error with\n     the API request, not an error running the job**.  Errors running the job\n     can be found by querying for the 'error' key (see above).\n\n-  | **error_msg**\n   | null if error is false, otherwise this is a quoted string with the error\n     message.\n\n**RAW API**\n\nThe RAW API can be utilized via the same HTTP GET requests appended with \"/raw\"::\n\n  GET /dalton/controller_api/v2/\u003cjobid\u003e/\u003ckey\u003e/raw\n\nThe ``\u003cjobid\u003e`` and ``\u003ckey\u003e`` are the same as the JSON API but a\nRAW API request returns the raw data from the Redis database, in the response body.\nThis is basically what is returned from the JSON API but not encapsulated or encoded as JSON.  For\nRAW API responses, the Content-Type header is set to \"text/plain\" with the exception of\nthe \"eve\" and \"all\" logs which\nuse \"application/json\".  A RAW request for the \"all\" key returns a string representation\nof a Python dictionary with all the key-value pairs.\nThe RAW responses also include \"attachment\" and \"filename\"\nin the Content-Disposition header that prompt browsers to download/save the file.\n\n**Valid Keys**\n\n-  **alert** - Alert data from the job. This is the same as what is\n   displayed in the \"Alerts\" tab in the job results page.\n\n-  **alert\\_debug** - A full alert log containing much information for\n   signature writers or for investigating suspected false positives (Suricata only).\n   This is the same as what is displayed in the \"Alert Debug\" tab in the job\n   results page.\n\n-  **alert\\_detailed** - Detailed alert data from the job. This is the\n   same as what is displayed in the \"Alert Details\" tab in the job\n   results page.\n\n-  **all** - Returns data from all keys (except for \"all\" of course).\n\n-  **debug** - Debug data from the job.  This is the same as what is\n   displayed in the \"Debug\" tab in the job results page.\n\n-  **dns\\_log** - A line based log of DNS requests and responses (Suricata only).\n   This is the same as what is displayed in the \"DNS Log\" tab in the job\n   results page.\n\n-  **engine\\_stats** - Contains data from various counters of the Suricata\n   engine (Suricata only).  This is the same as what is displayed in\n   the \"Engine Stats\" tab in the job results page.\n\n-  **error** - Error data from the job.  This is the same as what is\n   displayed in the \"Error\" tab in the job results page.\n\n-  **eve** - EVE JSON output from the job (Suricata only).  This is the same as what is\n   displayed in the \"EVE JSON\" tab in the job results page.\n\n-  **fast\\_pattern** - Fast pattern details for the submitted rules (Suricata only).\n   This is the same as what is displayed in the \"Fast Pattern\" tab in the job\n   results page.\n\n-  **http\\_log** - A line based log of HTTP requests (Suricata only).  This is the\n   same as what is displayed in the \"HTTP Log\" tab in the job results page.\n\n-  **ids** - IDS Engine output from the job.  This is the same as what\n   is displayed in the \"IDS Engine\" tab in the job results page.  \n   For Snort Agents, engine statistics output at the end of the job \n   run are populated here.\n\n-  **keyword\\_perf** - Contains data of per keyword profiling (Suricata only).\n   This is the same as what is displayed in the \"Keyword Perf\" tab in the job\n   results page.\n\n-  **other\\_logs** - *deprecated* - Other logs from the job (Suricata only).\n   This is returned as key/value pairs with the key being the\n   name of the log and the value being the contents of the log. This key\n   is deprecated and is not included in the ``all`` key response. The contents\n   of ``other_logs``, e.g. \"http_log\", \"tls_log\", etc., can and should be\n   accessed directly.\n\n-  **packet\\_stats** - Statistics from the pcap(s) (Suricata only).  This is the\n   same as what is displayed in the \"Engine Stats\" tab in the job results page.\n\n-  **perf** - Performance data from the job (if the job generated\n   performance data).   This is the same as what is displayed in the\n   \"Performance\" tab in the job results page.\n\n-  **start\\_time** - The time (epoch) the job was requested by a Dalton\n   agent.  This is returned as a string.\n\n-  **statcode** - Status code of the job.  This is a number returned as\n   a string.  If a job doesn't exist, the API will return an error (see\n   below) instead of an \"Invalid\" statcode.  Here is how to interpret\n   the status code:\n\n   +-------+-------------+\n   | Code  |   Meaning   |\n   +=======+=============+\n   |  -1   |   Invalid   |\n   +-------+-------------+\n   |   0   |    Queued   |\n   +-------+-------------+\n   |   1   |   Running   |\n   +-------+-------------+\n   |   2   |     Done    |\n   +-------+-------------+\n   |   3   | Interrupted |\n   +-------+-------------+\n   |   4   |   Timeout   |\n   +-------+-------------+\n\n-  **status** - A string corresponding to the current status of a job. \n   This is used in the Dalton Controller web UI and is what is displayed\n   in the browser when a job is submitted via the web interface to\n   inform the user of the current progress/state of the job.  When a job\n   is done, this will actually be a hyperlink saying \"Click here to view\n   your results\".  Unless there is a specific use case, 'statcode' is \n   usually used instead of 'status' for determining the status of a job.\n\n-  **submission\\_time** - The time (formatted as \"%b %d %H:%M:%S\") the\n   job was submitted to the Dalton Controller.\n\n-  **tech** - The sensor technology (i.e. engine and version) the job was submitted\n   for, in the format ``\u003cengine\u003e/\u003cversion\u003e``.\n   For example, ``suricata/4.0.0`` is Suricata v4.0.0.\n   If a custom config is used, it will be added on the end, also separated by a\n   forward slash.  For example, ``suricata/4.0.7/mycustomconfigname``.  A Suricata 4\n   sensor compiled with Rust support will have \"rust\\_\" prepended to the version,\n   for example, ``suricata/rust_4.1.5``.\n\n-  **time** - The time in seconds the job took to run, as reported by\n   the Dalton Agent (this includes job download time by the agent). \n   This is returned as a string and is the same as the \"Processing Time\"\n   displayed in the job results page.\n\n-  **tls\\_log** - A line based log of TLS handshake parameters (Suricata only).\n   This is the same as what is displayed in the \"TLS Log\" tab in the job\n   results page.\n\n-  **user** - The user who submitted the job. This will always be \"undefined\" \n   since authentication is not implemented in this release.\n\n**Examples:**\n\nJSON API Request::\n\n    GET /dalton/controller_api/v2/d1b3b838d41442f6/alert\n\nJSON API Response:\n\n.. code::\n\n    {\n    \"data\": \"06/26/2017-12:08:13.255103  [**] [1:180043530:4] Nemucod Downloader\n            Trojan Request Outbound [**] [Classification: \n            A Network Trojan was detected] [Priority: 1] {TCP} 192.168.1.201:65430 \n            -\u003e 47.91.93.208:80\\n\\n06/26/2017-12:08:13.255103  [**] [1:180056733:3] \n            Suspicious HTTP Request to a *.top TLD - Outbound [**] [Classification: Potentially \n            Bad Traffic] [Priority: 2] {TCP} 192.168.1.201:65430 -\u003e 47.91.93.208:80\\n\n            \\n06/26/2017-12:08:13.646674  [**] [1:180043530:4] Nemucod Downloader\n            Trojan Request Outbound [**] [**] [Classification: \n            A Network Trojan was detected] [Priority: 1] {TCP} 192.168.1.201:65430 \n            -\u003e 47.91.93.208:80\\n\\n\",\n    \"error_msg\": null,\n    \"error\": false\n    }\n\nJSON API Request::\n\n    GET /dalton/controller_api/v2/ae42737ab4f52862/ninjalevel\n\nJSON API Response:\n\n.. code:: javascript\n\n    {\"data\": null, \"error_msg\": \"No data found for 'ninjalevel' for Job ID ae42737ab4f52862\", \"error\": true}\n\nRAW API Request::\n\n    GET /dalton/controller_api/v2/ae42737ab4f52862/alert/raw\n\nRAW API Response:\n\n.. code::\n\n    12/16/2019-20:03:24.094114  [**] [1:806421601:0] MyMalware C2 Request Outbound [**]\n    [Classification: (null)] [Priority: 3] {TCP} 192.168.102.203:45661 -\u003e 172.16.31.41:80\n\nController API\n--------------\n\nIn addition to providing information on submitted jobs, the Dalton API includes\nthe ability to pull information from, and perform limited actions on, the Controller.\nThe following routes can be accessed via HTTP GET requests.  Full examples are not\nprovided here but can be easily obtained by making the request in a web browser.\n\n-  | **/dalton/controller_api/request_engine_conf?sensor=\u003csensor\u003e**\n   | Returns the requested configuration file as text.  The \u003csensor\u003e value\n     is going to be the engine, version, and, if applicable, the custom config\n     filename, separated by forward slashes.  For example:\n     ``suricata/5.0.0`` or ``suricata/5.0.0/mycustomconfig.yaml``.\n     Suricata version 4.x compiled with Rust support will have\n     the prefix \"rust\\_\" before the version, e.g. ``suricata/rust_4.1.5``.\n\n   | If no exact match is found for a config file on disk, the closest file\n     that matches is returned.\n\n-  | **/dalton/controller_api/delete-old-job-files**\n   | Deletes old job files from disk. Returns the number of\n     files deleted.\n     For more info see the `Job Queue \u003c#job-queue\u003e`__ section.\n\n-  | **/dalton/controller_api/job_status/\u003cjobid\u003e**\n   | Returns a string corresponding to the current status of a job.\n     This is used by the web browser primarily when a job is running.\n     See the 'status' key information in\n     the `Job API \u003c#job-api\u003e`__ section.\n\n-  | **/dalton/controller_api/job_status_code/\u003cjobid\u003e**\n   | Returns the job status code for the given jobid.\n     This is the job status code number, returned as string.\n\n   | For more details, see the information about 'statcode' in\n     the `Job API \u003c#job-api\u003e`__ section.\n\n-  | **/dalton/controller_api/get-current-sensors/\u003cengine\u003e**\n   | Returns a JSON response with 'sensor_tech' as the root element containing\n     an array of current active sensors, sorted descending based on ruleset\n     filename (just like the list in the web interface).\n\n   | \u003cengine\u003e should be ``suricata``, ``snort``, or ``zeek``.\n\n   | Example response:\n\n.. code:: javascript\n\n    {\"sensor_tech\": [\"suricata/4.0.1\", \"suricata/3.2.4\", \"suricata/2.0.9\"]}\n\n-  | **/dalton/controller_api/get-current-sensors-json-full**\n   | Response is a JSON payload with details about\n     all the current active sensors (agents). Info includes agent IP,\n     last check-in time, tech (e.g. ``suricata/4.0.1``), etc.\n\n-  | **/dalton/controller_api/get-prod-rulesets/\u003cengine\u003e**\n   | Returns a list of current available production rulesets on the\n     Controller for the given engine. The list contains the full path of\n     the rules files on the Controller.\n\n   | \u003cengine\u003e should be ``suricata`` or ``snort``\n\n   | Example response:\n\n.. code:: javascript\n\n    {\"prod-rulesets\": [\n        \"/opt/dalton/rulesets/suricata/SCWX-20171024-suricata-security.rules\",\n        \"/opt/dalton/rulesets/suricata/SCWX-20171024-suricata-malware.rules\",\n        \"/opt/dalton/rulesets/suricata/ET-20171023-all-suricata.rules\"\n        ]\n    }\n\n-  | **/dalton/controller_api/get-max-pcap-files**\n   | Returns the maximum number of pcap (or archive) files the controller is configured\n     to process per job submission.  This is set by the ``max_pcap_files`` option\n     in dalton.conf and knowing this can be useful to ensure that all pcaps programmatically\n     submitted are going to be processed.\n     A single archive file, even though it can contain multiple\n     pcaps, is only considered a single file in this context.\n\n-  | **/dalton/sensor_api/get_job/\u003cjobid\u003e**\n   | Returns the job zip file which includes the pcap(s), rule(s),\n     config file, and manifest used by the job referenced by \u003cjobid\u003e.\n     If the \u003cjobid\u003e is invalid or an error occurs, a HTML error page\n     is returned.\n\n\nDalton API Client\n-----------------\n\nAn API Client has been added in ``api/dalton.py`` that performs API calls with Python requests. \nThe client is limited to GET and POST requests.\n\nSubmit Job using the API\n------------------------\n\nThere is an option to programmatically submit jobs using HTTP POST requests. \nThe endpoint to submit a job is ``/dalton/coverage/summary``. \n\nAdditional parameters that are mandatory and will need to be included in the json payload of the POST request are listed below:\n\n.. code:: javascript\n    data = {\n        \"sensor_tech\": \u003cstring that has the sensor technology\u003e,\n        \"optionProdRuleset\": \"prod\",\n        \"prod_ruleset\": \u003crules path\u003e,\n        \"custom_engineconf\": \u003cstring with the complete confguration yaml file\u003e,\n        \"teapotJob\": 1\n    }\n\nThe above example indicates the minimum data payload to submit a job. \nYou need to make sure that you have the proper sensor tech name. \nYou may use the API call: ``GET /dalton/controller_api/v2/\u003cjobid\u003e/tech/raw`` to retrieve the specific sensor tech.\nThe rules path is ``/opt/dalton/rulesets/\u003csensor_name\u003e/\u003crule_file_name\u003e`` where sensor can be: suricata, zeek, snort, and the file name is the name of the file that has all the rules of this sensor.\n\nIt is also necessary to submit a file using the following format:\n\n.. code:: javascript\n    files = {\"coverage-pcap*\": (\u003cpcap_filename\u003e, \u003cpcap_bytes\u003e)}\n\nYou can upload up to 10 files with one job, so substitute * with a number from 0-9.\nYou will need to read the filebytes in the pcap_bytes var and optially you can include the ``pcap_filename``.\nSubmit the job as a shortlived ``teapotJob`` if you plan to make multiple calls in a short amount of time for better performance.\n\nOther useful arguments to submit a job are: \n\n- ``custom_rules`` in which you may include the custom rules you may want to test with your job,\n- ``optionAlertDetailed``, ``optionEveLog``, ``optionOtherLogs``: this can be set to ``True`` if you want to generate additional logs with your job.\n\nAn example script can be found in ``api/examples/submit_job.py``.\n\nTeapot Jobs\n===========\n\nDalton has the concept and capability of what is called a \"teapot\" job. \nA teapot job is one that is short lived in the Redis database and\n(usually) on disk.\n\nTeapot jobs are useful when submitting large number of jobs and/or jobs\nwhere the results are immediately processed and there isn't a need to\nkeep them around after that.  Often this is utilized in the programmatic\nsubmission of jobs combined with using the `Dalton API \u003c#dalton-api\u003e`__\nto automatically and/or quickly process the results.\n\nSuch job submissions are fleeting and voluminous in number.  In other \nwords, short and stout.  *Like a little teapot.*\n\nTeapot jobs differ from regular jobs in a few main ways:\n\n-  Results kept for a shorter period of time than regular jobs. \n   Teapot job expire timeouts are  configured with the ``teapot_redis_expire`` \n   option in ``dalton.conf``.\n-  Teapot jobs are submitted using the 'teapotJob' POST parameter (with\n   any value).  This parameter is not set or available when submitting\n   jobs via the Dalton web UI.\n-  Teapot jobs have a job id (\"JID\") that starts with 'teapot\\_'.\n-  The submission of a teapot job results in the JID being returned\n   instead of a redirect page.\n\nCurrently, if teapot jobs have not expired, they will show up in the Dalton\nQueue in the web UI although it would be fairly trivial to change the code to\nexclude them from the list.\n\nAdding Rulesets\n===============\n\nFor each Dalton job, a single 'defined ruleset' file can be used and/or 'custom rules'. \nCustom rules are entered in the Web UI but defined rulesets are stored on disk.\n\nOn the Dalton Controller, defined rulesets must be in the directory \nspecified by the ``ruleset_path`` variable in ``dalton.conf``.  By default this is  \n``/opt/dalton/rulesets``.  Inside that directory there must be a ``suricata`` \ndirectory where Suricata rules must be placed and a ``snort`` directory where \nSnort rules must be placed.  The ruleset files must end in\n``.rules``.\n\nIf the default ``ruleset_path`` value is not changed from \n``/opt/dalton/rulesets`` then the ``rulesets`` directory \n(and subdirectories) on the host running the Dalton \nController container is shared with the container so '.rules' files can be easily \nadded from the host machine.\n\nPopular open source rule download and management tools such as \n`rulecat \u003chttps://github.com/jasonish/py-idstools\u003e`__,\n`PulledPork \u003chttps://github.com/shirkdog/pulledpork\u003e`__, and\n`Suricata-Update \u003chttps://github.com/OISF/suricata-update\u003e`__ make it trivial to download\nrulesets, combine all rules into a single ``.rules`` file, and then store it \nin the necessary location.\n\nThe Dalton Controller container includes rulecat (see the ``rulecat_script`` variable \nin ``dalton.conf``) and when the Dalton Controller first starts up, if there \nare no existing rulesets, it will attempt to download the latest Suricata and Snort rulesets \nfrom `rules.emergingthreats.net \u003chttps://rules.emergingthreats.net\u003e`__.\n\nAdding Sensors\n==============\n\nAdding sensors to Dalton is a fairly simple process.  If there isn't already \na corresponding or compatible configuration file for the new sensor, that \nwill also need to be added; see `Adding Sensor Configs \u003c#adding-sensor-configs\u003e`__\nfor more information and to use custom config files for specific sensors.\n\nUnless a custom configuration is used, (see `Adding Sensor Configs \u003c#adding-sensor-configs\u003e`__),\nsensors (Agents) request jobs based on\ntheir particular engine (Suricata or Snort) and version (e.g. 5.0.0, 2.9.9.0).\nSubmitted jobs are queued based on the (corresponding) \"Sensor Version\" specified in the user\ninterface.  All applicable sensors pull jobs from the Controller from their respective queue, meaning\nthat there can be multiple Agents of the same type (engine and version) and\nthey will all pull from the appropriate shared queue on the Controller and\nreceive/run jobs on a first-come-first-served basis.\n\nDocker Sensors\n--------------\nThe ``docker-compose.yml`` file includes directives to build Dalton Agents for\na variety of Suricata and Snort versions.  The sensor engines (Suricata or\nSnort) are built from source.  To add a new or different version, just copy \none of the existing specifications and change the version number(s) as necessary.\n\nFor example, here is the specification for Suricata 3.2.3:\n\n.. code:: yaml\n\n      agent-suricata-3.2.3:\n        build:\n          context: ./dalton-agent\n          dockerfile: Dockerfiles/Dockerfile_suricata\n          args:\n            - SURI_VERSION=3.2.3\n            - http_proxy=${http_proxy}\n            - https_proxy=${https_proxy}\n            - no_proxy=${no_proxy}\n        image: suricata-3.2.3:latest\n        container_name: suricata-3.2.3\n        environment:\n          - AGENT_DEBUG=${AGENT_DEBUG}\n        restart: always\n\nTo add a specification for Suricata 4.0.2 (if it exists) just change the\n``SURI_VERSION`` arg value from '3.2.3' to '4.0.2'.  This will cause that version\nof Suricata to be downloaded and built.  The service name (e.g. 'agent-suricata-3.2.3')\ncontainer name, and image name should also be updated to be unique.  Multiple Agents with\nthe same engine/version can be run by keeping the ``SURI_VERSION`` and image name\nthe same but using different service and container names.\n\nExample Suricata 4.0.2 specification:\n\n.. code:: yaml\n\n      agent-suricata-4.0.2:\n        build:\n          context: ./dalton-agent\n          dockerfile: Dockerfiles/Dockerfile_suricata\n          args:\n            - SURI_VERSION=4.0.2\n            - http_proxy=${http_proxy}\n            - https_proxy=${https_proxy}\n            - no_proxy=${no_proxy}\n        image: suricata-4.0.2:latest\n        container_name: suricata-4.0.2\n        environment:\n          - AGENT_DEBUG=${AGENT_DEBUG}\n        restart: always\n\nRust support was added in Suricata 4.0 but is optional.  Starting with Suricata 5.0.0,\nRust is mandatory.  To turn on Rust support for a Suricata 4.x Agent, set the\n``ENABLE_RUST`` arg in the docker-compose file to ``--enable-rust`` for that\nparticular Agent specification (see below example).  Suricata 4.x Agents that have\nRust support will show up in the Web UI alongside the string, \"with Rust support\".\n\nExample Suricata 4.1.4 specification with Rust support:\n\n.. code:: yaml\n\n      agent-suricata-4.1.4-rust:\n        build:\n          context: ./dalton-agent\n          dockerfile: Dockerfiles/Dockerfile_suricata\n          args:\n            - SURI_VERSION=4.1.4\n            - http_proxy=${http_proxy}\n            - https_proxy=${https_proxy}\n            - no_proxy=${no_proxy}\n            - ENABLE_RUST=--enable-rust\n        image: suricata-4.1.4-rust:latest\n        container_name: suricata-4.1.4-rust\n        environment:\n          - AGENT_DEBUG=${AGENT_DEBUG}\n        restart: always\n\nSuricata can also have ``SURI_VERSION=current`` in which case the latest\nSuricata version will be used to build the Agent.  Having a 'current' Suricata \nversion specification in the ``docker-compose.yml`` file is especially convenient \nsince when a new version comes out, all that has to be done is run the\n``start-dalton.sh`` script and a new Dalton Agent with the latest Suricata \nversion will be built and available.\n\nSnort agents are the same way but the args to customize are ``SNORT_VERSION`` and, \nif changed, ``DAQ_VERSION``.  Example Snort specification:\n\n.. code:: yaml\n\n      # Snort 2.9.11 from source\n      agent-snort-2.9.11:\n        build:\n          context: ./dalton-agent\n          dockerfile: Dockerfiles/Dockerfile_snort\n          args:\n            - SNORT_VERSION=2.9.11\n            - DAQ_VERSION=2.0.6\n            - http_proxy=${http_proxy}\n            - https_proxy=${https_proxy}\n            - no_proxy=${no_proxy}\n        image: snort-2.9.11:latest\n        container_name: snort-2.9.11\n        environment:\n            - AGENT_DEBUG=${AGENT_DEBUG}\n          restart: always\n\nSuricata Agents should build off the Suricata Dockerfile,\n``Dockerfiles/Dockerfile_suricata_rust``.\n\nSnort Agents should build off the\nSnort Dockerfile at ``Dockerfiles/Dockerfile_snort``.\n\nNon-Docker Sensors\n------------------\nSensors don't have to be Docker containers or part of the docker-compose\nnetwork to be used by the Dalton Controller; they just have to be able to \naccess and talk with the Docker Controller webserver.\n\nA Suricata or Snort machine can be turned into a Dalton Agent fairly easily. \nRequirements:\n\n-  Engine (Suricata or Snort)\n-  Python 3.6 or later\n-  ``dalton-agent.py``\n-  ``dalton-agent.conf``\n\nThe ``dalton-agent.conf`` file must be modified to point to the Docker \nController (see ``DALTON_API`` option).\n\nFor more details on the Dalton Agent configuration options, see the inline \ncomments in the ``dalton-agent.conf`` file.\n\nTo start the Dalton Agent, run dalton-agent.py::\n        \n        Usage: dalton-agent.py [options]\n\n        Options:\n        -h, --help            show this help message and exit\n        -c CONFIGFILE, --config=CONFIGFILE\n                              path to config file [default: dalton-agent.conf]\n\n\nAdding Sensor Configs\n=====================\n\nSensor configuration files (e.g. ``suricata.yaml`` or ``snort.conf``) are \nstored on the Dalton Controller.  When a sensor checks into the Controller, \nit is registered in Redis and when that sensor is selected for a Dalton job, \nthe corresponding config file is loaded, populated under the ``Config Files`` vertical tab \nin the Web UI, and submitted with the Dalton job.\n\nThe Dalton Controller uses the ``engine_conf_path`` variable from ``dalton.conf`` \nto use as a starting location on the filesystem to find sensor configuration files to use.  \nInside that directory there must be \na ``suricata`` directory where the Suricata ``.yaml`` files go and a ``snort`` \ndirectory where the Snort ``.conf`` files go.\n\nBy default, on the Controller, ``engine_conf_path`` is set to ``/opt/dalton/app/static/engine-configs`` \nwhich is symlinked from ``/opt/dalton/engine-configs``.  The Dalton Controller and host also \nshare the ``engine-configs`` directory to make it easy to add config files as needed \nfrom the host.\n\nIt is recommended that the ``engine_conf_path`` not be changed since Flask looks in \nthe ``static`` directory to serve the config files and changing it will \nmostly like break something.\n\nSensor configuration files \nare not automatically added when Agents are built or the Controller is run; \nthey must be manually added. \nHowever, the Dalton Controller already comes with the default (from source) config files \nfor Suricata versions 0.8.1 and later, and for Snort 2.9.0 and later. \nDuplicate config files are not included.  For example, since all the Suricata \n1.4.x versions have the same (default) .yaml file, only \"suricata-1.4.yaml\" \nis included.\n\nThe Controller attempts to find a config file to load/use based off\nthe sensor engine (Suricata or Snort) and version number (e.g. 5.0.0, 2.9.9.0).\n\nFor example, if an Agent is running Suricata version 5.0.0, then the Controller will \nlook for a file with the name \"suricata-5.0.0.yaml\" in the \n``engine-configs/suricata/`` directory.  If it can't find an \nexact match, it will attempt to find the closest match it can based off the\nversion number.\n\nIf a custom config is desired to be used by a particular sensor, set\nthe ``SENSOR_CONFIG`` variable in the Agent's ``dalton-agent.conf`` file\nand place a file with the same name on the Controller in the\n``engine-configs/suricata/`` directory (for Suricata) or\n``engine-configs/snort/``  directory (for Snort).  If the ``SENSOR_CONFIG`` value\ndoes not exactly match a config file on the Controller, the Controller\nwill look for filenames with the SENSOR_CONFIG value and extensions \".yaml\", \".yml\",\nand \".conf\".\n\nFor new Suricata releases, the ``.yaml`` file from source should just \nbe added to the ``engine-configs/suricata`` directory and named \nappropriately.  For new Snort releases, it is recommended that the \ndefault ``.conf`` file be run thru  the ``clean_snort_config.py`` \nscript located in the ``engine-configs/`` directory::\n\n    Usage:\n    \n    python clean_snort_config.py \u003cin-file\u003e \u003cout-file\u003e\n\n\n\nLogging and Debugging\n=====================\n\nBy default, the Dalton Controller logs to ``/var/log/dalton.log`` and Dalton \nAgents log to ``/var/log/dalton-agent.log``.  The nginx container logs to \nthe ``/var/log/nginx`` directory (``dalton-access.log`` and \n``dalton-error.log``).  The (frequent) polling that Dalton Agents do to the \nnginx container to check for new jobs is intentionally not logged since it is \nconsidered too noisy.\n\nFor the Dalton Controller, debugging can be enabled in ``dalton.conf`` file or \nby setting the ``CONTROLLER_DEBUG`` environment variable (e.g. \n``CONTROLLER_DEBUG=1``.  This can also be passed during the container build \nprocess and set in the ``.env`` file.  If either the config file or environment \nvariable has debugging set, debug logging will be enabled.\n\nFor the Dalton Agents, debugging can be enabled in ``dalton-agent.conf`` file or \nby setting the ``AGENT_DEBUG`` environment variable (e.g. \n``AGENT_DEBUG=1``.  This can also be passed during the container build \nprocess and set in the ``.env`` file.  If either the config file or environment \nvariable has debugging set, debug logging will be enabled.\n\nFlowsynth WebUI\n===============\n\nDalton includes a Web UI for\n`Flowsynth \u003chttps://github.com/secureworks/flowsynth\u003e`__ , a tool that \nfacilitates network packet capture creation. The flowsynth Web UI makes it trivial\nto model network traffic and test it against a Dalton Agent.\n\nAccessing the Flowsynth WebUI can be done via the 'Flowsynth' link in the Dalton toolbar, or directly\nusing the '/flowsynth' URI path.\nThe flowsynth UI has two modes of\noperation: Build and Compile. The build mode provides a wizard-like interface for\ncreating certain types of pcaps. The compile mode provides a direct interface to\nthe flowsynth compiler, allowing for the building of synth files directly in the UI.\n\nBuild Mode\n----------\nThe Flowsynth Build mode allows for quick pcap generation using some sensible\ndefaults. On the 'Network Layer' vertical tab, the source and destination IP ranges can be selected.\nAn IP address is chosen at random from these ranges. On the 'Transport Layer' vertical tab\nis the ability to choose between TCP and UDP, and optionally establish the TCP connection\nwith a three-way handshake. Destination and Source ports are chosen at random,\nor can be set explicitly. The 'Payload' vertical tab allows the user to easily build some common\npayloads. The wizards generate flowsynth syntax language, and populate the 'Compile'\ntab with the content to allow for any last minute changes prior to compilation.\n\nBinary, non-printable, and printable bytes can be represented using Hexadecimal escape sequences \n(\\xhh).  Such encoding are converted to their representative bytes when the pcap is compiled. \nFor example, '\\x41' becomes 'A'.\n\n\nRaw Payload\n```````````\nThe raw payload wizard allows a user to rapidly model two-way communication between\na client and server.  \nIt is often useful for modeling custom protocols and/or binary protocols.\n\nHTTP Payload\n````````````\nThe HTTP  wizard makes it simple to build HTTP client requests and HTTP\nserver responses. The payload prompts for two types of input, an HTTP header section\nand a HTTP body section.\n\nIf the 'Autocompute request Content-Length header' and/or \n'Autocompute response Content-Length header' is selected, the wizard will compute and add a\nContent-Length header based on the HTTP body data.  If a Content-Length header already\nexists in the HTTP Header data, it will be updated to reflect the correct size of the \ncorresponding HTTP body.  If the request body is empty, a \"Content-Length: 0\" header \nwill *not* be added; if a response body is empty, a \"Content-Length: 0\" header *will* be \nadded.\n\nCertificate Payload\n```````````````````\nThe Certificate wizard makes it trivial to generate a partial SSL/TLS handshake\nusing a user-supplied certificate.\n\nCompile Mode\n------------\nCompile mode provides a direct interface to the flowsynth compiler, allowing for \nthe building of synth files directly in the UI. The compile mode UI is populated by the\nbuild mode wizards. After the synth has been submitted, a pcap will be generated\nand a download link provided. The pcap can also be directly submitted from the web interface \nto Dalton, to be used in a Suricata or Snort job.\n\nIt's also possible to pre-populate the compile page, either via a GET or POST request.\n\nExample 1:\n\n.. code:: text\n\n    http://127.0.0.1/flowsynth/compile?flowsynth=_flowsynth_code_goes_here_\n\nExample 2:\n\n.. code-block:: html\n\n    \u003cform action=\"http://127.0.0.1/flowsynth/compile\" method=\"POST\"\u003e\n      \u003cdiv\u003e\n        \u003clabel for=\"synth\"\u003eWhat do you want to synth?\u003c/label\u003e\n        \u003ctextarea rows=20 class=\"field span11\" name=\"synth\"\u003e\u003c/textarea\u003e\n      \u003c/div\u003e\n      \u003cinput type=\"submit\"\u003e\n    \u003c/form\u003e\n\n\nZeek\n====\n\nStarting with Dalton version 3.2.0, Zeek as a sensor is supported. There is limited support in the API and\nconfigurations/rulesets cannot be changed at runtime from the UI. However, Zeek scripts can be\nadded in the rulesets directory and will be executed with every run.\n\n\nCyberChef\n=========\n\nFor convenience, Dalton has the ability to easily build and run a\n`CyberChef \u003chttps://gchq.github.io/CyberChef/\u003e`__ container.  This is enabled by default in the\n``docker-compose.yml`` file.  Accessing CyberChef can be done via the 'CyberChef' link\nin the Dalton toolbar, or directly using the '/cyberchef' URI path.\n\n\nFrequently Asked Questions\n==========================\n\n1. | **Why is it named 'Dalton'?**\n   | Dalton is the name of Patrick Swayze's character in the movie \n     \"Road House\".\n\n#. | **How do I configure the Dalton Controller to listen on a different port?**\n   | The external listen port of the Dalton Controller can be set in the ``.env``\n     file in the repository root.  The Dalton Controller and nginx containers\n     must be rebuilt for the change to take effect (just run ``start_dalton.sh``).\n\n#. | **Is SSL/TLS supported?**\n   | SSL/TLS can be configured for the Web UI.\n     See `Enabling SSL/TLS on the Controller \u003c#Enabling-SSL-TLS-on-the-Controller\u003e`__.\n   \n#. | **Will this work on Windows?**\n   | The native Dalton code won't work as expected on Windows without non-trivial \n     code changes. \n     However, if the Linux containers can run on Windows, then \n     it should be possible to get containers working on a Windows host.  But\n     this has not been tested.\n   \n#. | **What is the difference between an \"engine\", \"sensor\", and \"agent\"?**\n   | In this context those terms, for the most part, mean the same thing.\n     Technically, you can think of \"engine\" as the IDS engine, in this\n     case Suricata or Snort; \"sensor\" as the system running the engine; and\n     \"agent\" as a specific system running the Dalton Agent code and checking into\n     the Dalton Controller.  \"Sensor\" and \"Agent\" are very often used\n     interchangeably.\n\n#. | **Is there Dalton Agent support for Snort version \u003c 2.9?**\n   | Currently no.  Dalton Agents that run Snort utilize the 'dump' DAQ to replay pcaps\n     and DAQ wasn't introduced until Snort 2.9.  Dalton Agents for older Snort\n     versions (e.g. 2.4) have been written in the past but are not part of this \n     open source release.  However, if there is a demand for such support, then\n     adding support for older Snort versions will be reconsidered.\n\n#. | **So then is Snort 3 supported?**\n   | Not at this time.  Snort 3 support is certainly possible and is being\n     considered.\n\n#. | **Does Dalton support authentication such as username/password/API tokens or \n     authorization enforcement like discretionary access control?**\n   | No, not in this open source release although such additions have been done\n     before, including single sign on integration.  However, such enhancements \n     would require non-trivial code additions. There are some authentication \n     decorators commented out and scattered throughout the code and the Dalton \n     Agents do send an API token as part of their requests but the Dalton \n     Controller doesn't validate it.  The lack of authentication and \n     authorization does mean that it isn't difficult for malicious actors to \n     flood the Controller, submit malformed jobs, corrupt job results, dequeue\n     jobs, and DoS the application.\n     \n#. | **How can I programmatically submit a job to Dalton?**\n   | Right now, a programmatic submission must mimic a Web UI submission. In the\n     future, a more streamlined and easier to use submission API may be exposed.\n     Feel free to submit a pull request with this feature.\n\n#. | **When I submit jobs to Suricata Agents with multiple pcaps, the job zipfile\n     only has one pcap. What's going on?**\n   | In read pcap mode, which is how the Suricata and Snort engines process pcaps,\n     older version of Suricata only support the reading of a single pcap.  Therefore,\n     *for jobs submitted to such older Suricata Agents*, to support\n     multiple pcaps in the same Suricata job, the Dalton Controller will combine \n     the pcaps into a single file before making the job available for Agents to\n     grab. By default, the pcap merging is done with\n     `mergecap \u003chttps://www.wireshark.org/docs/man-pages/mergecap.html\u003e`__.\n     For more details see `Packet Captures \u003c#Packet-Captures\u003e`__.\n\n#. | **Can I have more than one Agent with the same engine/version? For example, can\n     I have multiple Agents running Suricata 4.0.1?**\n   | Of course.  If you use the Agent containers and Docker Compose, make sure that\n     the service and container name are unique between sensors.  Agents poll a\n     queue on the Dalton controller for jobs based on their \"TECHNOLOGY\"\n     (typically engine and version) and multiple Agents can poll the same queue.\n     Pending jobs are given to the first Agent that requests them.\n\n#. | **Why is it that when I try to build a Snort 2.9.0 or 2.9.0.x container, it fails when\n     configuring Snort saying it can't find the 'dnet' files?**\n   | Attempting to build Snort 2.9.0 and 2.9.0.x  will fail because \n     Autoconf can't find the dnet files. This was apparently fixed in \n     Snort 2.9.1 and later. If \n     you really want a Snort 2.9.0 or 2.9.0.x Agent, you will have to build \n     one out yourself.  The Dalton Agent code should work\n     fine on it.  If it turns out that there is a lot of demand for \n     Snort 2.9.0.x Agents, adding native support for it will be reconsidered.\n\n#. | **Regarding the code ... why did you do that like that? What were you \n     thinking? Do you even know about object-oriented programming?**\n   | These are valid questions.  Much of the code was written many years ago \n     when the author was new to Python, never having written any Python code\n     before other than tweaking a few lines of code in existing projects, and\n     unaware of Python's object-oriented support.  While such code could be\n     cleaned up and refactored, a lot of it was left as-is since it already \n     worked and it was decided that time and effort should be spent elsewhere.\n     Additionally, the Dalton Agent code was originally written to run on \n     restricted/custom systems that only had Python 2.4 support and couldn't use \n     non-standard libraries.  This is especially noticeable (painful?) with \n     the use of urllib2 instead of urllib3 or Requests.  Therefore, if you \n     do review the code, it is requested that you approach it with a spirit of\n     charity.\n\n#. | **I found a bug in Dalton.  What should I do?**\n   | Feel free to report it and/or fix it and submit a pull request.\n   \n \n\nAuthors\n=======\n\n-  David Wharton\n   \n-  Will Urbanski\n   \nContributors\n------------\n\n-  Rob Vinson\n-  George P. Burdell\n-  Adam Mosesso\n-  Donald Campbell\n \n\nFeedback including bug reports, suggestions, improvements, questions,\netc. is welcome.\n\n \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsecureworks%2Fdalton","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsecureworks%2Fdalton","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsecureworks%2Fdalton/lists"}