{"id":13585857,"url":"https://github.com/bitsofinfo/testssl.sh-processor","last_synced_at":"2025-04-13T07:55:48.725Z","repository":{"id":146759356,"uuid":"154894966","full_name":"bitsofinfo/testssl.sh-processor","owner":"bitsofinfo","description":"Wrapper for concurrent batch processing of testssl.sh commands","archived":false,"fork":false,"pushed_at":"2019-03-15T14:24:27.000Z","size":1830,"stargazers_count":13,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-13T07:55:44.034Z","etag":null,"topics":["batch","continuous-integration","security","ssl","test","testssl","tls"],"latest_commit_sha":null,"homepage":"https://bitsofinfo.wordpress.com/2018/12/21/batch-processing-testssl-sh-commands/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bitsofinfo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-26T21:11:25.000Z","updated_at":"2023-02-02T10:40:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"f157d927-9689-4bf4-9aad-ee223382d558","html_url":"https://github.com/bitsofinfo/testssl.sh-processor","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitsofinfo%2Ftestssl.sh-processor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitsofinfo%2Ftestssl.sh-processor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitsofinfo%2Ftestssl.sh-processor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitsofinfo%2Ftestssl.sh-processor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bitsofinfo","download_url":"https://codeload.github.com/bitsofinfo/testssl.sh-processor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248681491,"owners_count":21144700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","continuous-integration","security","ssl","test","testssl","tls"],"created_at":"2024-08-01T15:05:11.271Z","updated_at":"2025-04-13T07:55:48.706Z","avatar_url":"https://github.com/bitsofinfo.png","language":"Python","readme":"# testssl.sh-processor\n\nThis project is intended to serve as part of a larger pipeline for mass concurrent invocations of the great SSL/TLS\ntesting tool https://github.com/drwetter/testssl.sh via command files.\n\nSee the sibling project [testssl.sh-alerts](https://github.com/bitsofinfo/testssl.sh-alerts) for a engine that can react to `testssl.sh` JSON result output files to send notifications or perform other actions.\n\n![](docs/diag1.png)\n\n## testssl_processor.py\n\nProvides a long lived watchdog process that monitors a directory (via [watchdog](https://github.com/gorakhargosh/watchdog))\nfor `testssl.sh` command files. As new files appear within the `--input-dir` containing the `--filename-filter`\nthey are consumed and evaluated for `testssl.sh` commands, one per line. Each `testssl.sh` command is processed in a separate thread and processing results are logged to a YAML or JSON result file under the `--output-dir`. The actual output from each invoked `testssl.sh` invocation (i.e. via `--*file` arguments) is also written to disk scoped within a timestamped output directory under the `--output-dir`\n\n# Requirements\n\n**Python 3**\n\nDependencies:\n```\npip install twisted pyyaml python-dateutil watchdog\n```\n\n# Usage\n\n```bash\n./testssl_processor.py --help                                                                                       \n\nusage: testssl_processor.py [-h] [-i INPUT_DIR] [-O OUTPUT_DIR]\n                            [-m TESTSSL_PATH_IF_MISSING] [-f FILENAME_FILTER]\n                            [-o RESULT_FILENAME_PREFIX] [-q RESULT_FORMAT]\n                            [-l LOG_FILE] [-x LOG_LEVEL] [-w WATCHDOG_THREADS]\n                            [-t TESTSSL_THREADS]\n                            [-W OUTPUT_DIR_HTTPSERVER_PORT]\n                            [-u RETAIN_OUTPUT_DAYS]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -i INPUT_DIR, --input-dir INPUT_DIR\n                        Directory path to recursively monitor for new\n                        `--filename-filter` testssl.sh command files. Default\n                        './input'\n  -O OUTPUT_DIR, --output-dir OUTPUT_DIR\n                        Directory path to place all processor output, and\n                        testssl.sh output files to if relative paths are in\n                        command files. If absoluate paths are in testssl.sh\n                        command files they will be respected and only\n                        processor output will go into --output-dir. Default\n                        './output'\n  -m TESTSSL_PATH_IF_MISSING, --testssl-path-if-missing TESTSSL_PATH_IF_MISSING\n                        If the testssl.sh commands in the command files do not\n                        reference an absolute path to the testssl.sh command,\n                        it assumes its already on the PATH or in the current\n                        working directory of the processor. Otherwise you can\n                        specify the PATH to it with this argument. Default\n                        './testssl.sh'\n  -f FILENAME_FILTER, --filename-filter FILENAME_FILTER\n                        Only react to filenames in --input-dir that contain\n                        the string --filename-filter, default 'testssl_cmds'\n  -o RESULT_FILENAME_PREFIX, --result-filename-prefix RESULT_FILENAME_PREFIX\n                        processor execution result filename prefix. Default\n                        'testssl_processor_result'\n  -q RESULT_FORMAT, --result-format RESULT_FORMAT\n                        processor result filename format, json or yaml.\n                        Default 'json'\n  -l LOG_FILE, --log-file LOG_FILE\n                        Path to log file, default None which means STDOUT\n  -x LOG_LEVEL, --log-level LOG_LEVEL\n                        log level, default DEBUG\n  -w WATCHDOG_THREADS, --watchdog-threads WATCHDOG_THREADS\n                        max threads for watchdog file processing, default 1\n  -t TESTSSL_THREADS, --testssl-threads TESTSSL_THREADS\n                        for each watchdog file event, the maximum number of\n                        commands to be processed concurrently by testssl.sh\n                        invocations, default 5\n  -W OUTPUT_DIR_HTTPSERVER_PORT, --output-dir-httpserver-port OUTPUT_DIR_HTTPSERVER_PORT\n                        Default None, if a numeric port is specified, this\n                        will startup a simple twisted http server who's\n                        document root is the --output-dir\n  -u RETAIN_OUTPUT_DAYS, --retain-output-days RETAIN_OUTPUT_DAYS\n                        Optional, default 7, the number of days of data to\n                        retain that ends up under `--output-dir`, purges\n                        output dirs older than this time threshold\n```\n\n# Docker\n```\ndocker build -t testssl-processor .\n\ndocker run \\\n  -v /pathto/input:/input -v /pathto/output:/output \\\n  testssl-processor:latest \\\n  testssl_processor.py \\\n    --input-dir /input \\\n    --output-dir /output \\\n    --output-dir-httpserver-port 8888 \\\n    --testssl-threads 5 \\\n    --retain-output-days .1\n```\n\n## Example:\n\nRun the command:\n```\ngit clone https://github.com/drwetter/testssl.sh\n\nmkdir input\nmkdir output\n\n./testssl_processor.py \\\n  --input-dir ./input \\\n  --testssl-path-if-missing ./testssl.sh \\\n  --output-dir ./output \\\n  --filename-filter testssl_cmds \\\n  --result-format json \\\n  --output-dir-httpserver-port 8888\n```\n\nGiven a `testssl_cmds` file (see `sample/` dir) dropped into directory `input/`.\n\n```\ncp sample/www.google.com-testssl_cmds input/\n```\n\nNow the `testssl_processor.py` output shows:\n\n```\n2018-11-13 19:49:10,022 - root - INFO - Monitoring for new testssl_cmds files at: ./input with filename filter: testssl_cmds\n2018-11-13 19:49:10,023 - root - INFO - Starting HTTP server listening on: 8888 and serving up: ./output\n2018-11-13 19:49:12,311 - root - INFO - Responding to creation of file: ./input/www.google.com-testssl_cmds\n2018-11-13 19:49:17,317 - root - INFO - Processing testssl_cmds: './input/www.google.com-testssl_cmds'\n2018-11-13 19:49:17,340 - root - INFO - Processing testssl_cmd: 'testssl.sh -S -P -p --fast --logfile www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.log --jsonfile-pretty www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.json --csvfile www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.csv --htmlfile www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.html https://www.google.com'\nmaking dir: ./output/20181113_194917-www.google.com-testssl_cmds/www.google.com/20181108120000/public/search\n2018-11-13 19:50:03,392 - root - DEBUG - Command finished: exit code: 0 stdout.len:5752 stderr.len:0 cmd: /home/bitsofinfo/code/github.com/bitsofinfo/testssl.sh-processor/testssl.sh/testssl.sh -S -P -p --fast --logfile www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.log --jsonfile-pretty www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.json --csvfile www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.csv --htmlfile www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.html https://www.google.com\n2018-11-13 19:50:03,394 - root - DEBUG - Event 20181113_194917 Testssl processor result written to: ./output/20181113_194917-www.google.com-testssl_cmds/testssl_processor_result_20181113_194917.json\n2018-11-13 19:50:03,495 - root - DEBUG - Pool closed and terminated\n```\n\nThe contents of our `input/` and `output/` dirs is now as follows.\nThe actual output of the `testssl.sh` commands `--*file` directives are in the respective html/json files etc, while the output from the processor itself that invokes all the `testssl.sh` commands is in the `testssl_processor_result_*.json` files.\n\n![](docs/dirs.png)\n\nContents of `testssl_processor_result_*.json`:\n\n```\n[\n    {\n        \"success\": true,\n        \"orig_cmd\": \"testssl.sh -S -P -p --fast --logfile www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.log --jsonfile-pretty www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.json --csvfile www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.csv --htmlfile www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.html https://www.google.com\",\n        \"timestamp\": \"20181113_194917\",\n        \"testssl_path_if_missing\": \"./testssl.sh\",\n        \"actual_cmd\": \"/home/bitsofinfo/testssl.sh-processor/testssl.sh/testssl.sh -S -P -p --fast --logfile www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.log --jsonfile-pretty www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.json --csvfile www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.csv --htmlfile www.google.com/20181108120000/public/search/20181108120000_testssl_www.google.com.html https://www.google.com\",\n        \"cwd\": \"./output/20181113_194917-www.google.com-testssl_cmds\",\n        \"returncode\": 0,\n        \"stdout\": \"\\u001b[1m\\n###########################################################\\n    testssl.sh       3.0rc2 from \\u001b[m\\u001b[1mhttps://testssl.sh/dev/.................. ... .. . .172.217.6.68:443 (www.google.com) \u003c\u003c--\\u001b[m\\n\\n\\n\",\n        \"stderr\": \"\",\n        \"exec_ms\": 46051.146\n    }\n]\n```\n\nHitting http://localhost:8888 in a browser:\n\n![](docs/httpd.png)\n\n## Related\n\n* If you would like to react and send alerts or copy files around based on the results of the `testssl.sh` JSON result file output produced by this, take a look at [testssl.sh-alerts](https://github.com/bitsofinfo/testssl.sh-alerts) at https://github.com/bitsofinfo/testssl.sh-alerts\n\n* This tool was originally developed to consume `testssl.sh` command files generated as part of the [swarm-traefik-state-analyzer](https://github.com/bitsofinfo/swarm-traefik-state-analyzer/blob/master/docs/tlsssltools.md) project's [testsslcmdsgeneratory.py](https://github.com/bitsofinfo/swarm-traefik-state-analyzer/blob/master/docs/tlsssltools.md) script as part of a larger pipeline of scripts that continously monitors the state of applications deployed on Docker Swarm clusters.\n\n* You can also use [testssl.sh-masscan's generate_scan_file.py](https://github.com/TKCERT/testssl.sh-masscan/blob/master/generate_scan_file.py) to generate command files that are compatible with the `testssl_processor.py` watchdog processor\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitsofinfo%2Ftestssl.sh-processor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbitsofinfo%2Ftestssl.sh-processor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitsofinfo%2Ftestssl.sh-processor/lists"}