{"id":23792027,"url":"https://github.com/EC-DIGIT-CSIRC/sysdiagnose","last_synced_at":"2025-09-06T09:32:58.343Z","repository":{"id":173733993,"uuid":"611836591","full_name":"EC-DIGIT-CSIRC/sysdiagnose","owner":"EC-DIGIT-CSIRC","description":"Forensic toolkit for iOS sysdiagnose feature","archived":false,"fork":false,"pushed_at":"2024-12-17T09:32:43.000Z","size":822,"stargazers_count":155,"open_issues_count":34,"forks_count":13,"subscribers_count":12,"default_branch":"main","last_synced_at":"2024-12-17T10:26:44.589Z","etag":null,"topics":["forensic-analysis","incident-response-tooling","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"eupl-1.2","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EC-DIGIT-CSIRC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-09T16:37:38.000Z","updated_at":"2024-12-17T09:32:47.000Z","dependencies_parsed_at":"2024-01-14T19:56:43.797Z","dependency_job_id":"12e385c4-bae9-45b5-9626-8a78b321131d","html_url":"https://github.com/EC-DIGIT-CSIRC/sysdiagnose","commit_stats":null,"previous_names":["ec-digit-csirc/sysdiagnose"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EC-DIGIT-CSIRC%2Fsysdiagnose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EC-DIGIT-CSIRC%2Fsysdiagnose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EC-DIGIT-CSIRC%2Fsysdiagnose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EC-DIGIT-CSIRC%2Fsysdiagnose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EC-DIGIT-CSIRC","download_url":"https://codeload.github.com/EC-DIGIT-CSIRC/sysdiagnose/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232110610,"owners_count":18474011,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["forensic-analysis","incident-response-tooling","python"],"created_at":"2025-01-01T18:01:41.077Z","updated_at":"2025-09-06T09:32:58.318Z","avatar_url":"https://github.com/EC-DIGIT-CSIRC.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Sysdiagnose analysis framework\n\n![sysdiagnose-512x512](https://github.com/EC-DIGIT-CSIRC/sysdiagnose/assets/750019/2742ca75-758e-4393-a2d1-5c94d09b0eb3)\n\n# Installation\n\nNote that you will need Python 3.11 or higher.\n\nCreate a virtual environment and install dependencies:\n\n```bash\n python3 -m venv venv\n source venv/bin/activate\n pip install .\n sudo apt install graphviz\n ```\n\n On MacOS (Apple Silicon):\n```bash\n python3 -m venv venv\n source venv/bin/activate\n pip3 install .\n brew install graphviz  # Or your prefered way of installing graphviz binaries\n brew install libmagic\n ```\n\n\nOn linux systems you may wish to install the [unifiedlogs](#unifiedlogs) parser. See below for instructions how to do this.\n\n# Quickstart\n\n## Case management\n\nCreating a new case, with the optional `-c` parameter if you want to specify the case number yourself. (such as an uuid)\n\n```bash\n$ sysdiag create test-data/iOS12/sysdiagnose_2019.02.13_15-50-14+0100_iPhone_OS_iPhone_16C101.tar.gz\n\nSysdiagnose file has been processed\nCase ID: 1\n```\n\nListing existing cases can be done easily:\n\n```bash\n$ sysdiag cases\nCase ID              acquisition date           Serial number    Unique device ID                          iOS Version    Tags\n-------------------  -------------------------  ---------------  ----------------------------------------  -------------  ------\npublic               2023-05-24T13:29:15-07:00  F4GT2K24HG7K     e22f7f830e5dcc1287a1690a2622c2b12afaa33c  \u003cunknown\u003e\n```\n\nThe `cases` folder is the current folder by default.\nYou can change this using the environment variable `SYSDIAGNOSE_CASES_PATH`, for example.\n\n```bash\n$ export SYSDIAGNOSE_CASES_PATH='/path/to/folder'\n$ sysdiag list cases\n```\n\n## Parsing data and converting it to a usable format\n\nData of sysdiagnose is not always usable directly, use parsers to convert them to a nice json file.\n\nRun parsers:\n\n```bash\n$ sysdiag -c 1 parse ps\nExecution success, output saved in: cases/1/parsed_data/ps.json\n\n$ sysdiag -c 1 parse sys\nExecution success, output saved in: cases/1/parsed_data/sys.json\n```\n\nTo run on all cases do not specify a case number or use `-c all`.\n\nList available parsers :\n\n```bash\n$ sysdiag list parsers\nParser Name            Parser Description\n---------------------  ---------------------------------------------------------------------\nall                    Run all parsers\naccessibility_tcc      Parsing Accessibility TCC logs\nappinstallation        Parsing app installation logs\nbrctl                  Parsing brctl files\ncontainermanager       Parsing containermanagerd logs file\ncrashlogs              Parsing crashes folder\ndemo_parser            Demo parsers\nitunesstore            Parsing iTunes store logs\nlockdownd              Parsing lockdownd logs file\nlogarchive             Parsing system_logs.logarchive folder\nmobileactivation       Parsing mobileactivation logs file\nmobileinstallation     Parsing mobile_installation logs file\nnetworkextension       Parsing networkextension plist file\nnetworkextensioncache  Parsing networkextensioncache plist file\nolddsc                 Parsing olddsc files\nplists                 Parsing any pslist into json\npowerlogs              Parsing powerlogs database\nps                     Parsing ps.txt file\npsthread               Parsing ps_thread.txt file\nremotectl_dumpstate    Parsing remotectl_dumpstate file containing system information\nsecurity_sysdiagnose   Parsing security-sysdiagnose.txt file containing keychain information\nshutdownlogs           Parsing shutdown.log file\nspindumpnosymbols      Parsing spindump-nosymbols file\nswcutil                Parsing swcutil_show file\nsys                    Parsing SystemVersion plist file\ntaskinfo               Parsing taskinfo txt file\nuuid2path              Parsing UUIDToBinaryLocations plist file\nwifi_known_networks    Parsing Known Wifi Networks plist file\nwifinetworks           Parsing com.apple.wifi plist files\nwifiscan               Parsing wifi_scan files\nwifisecurity           Parsing WiFi Security logs\n```\n\n## Analysers to process parsed data\n\nList analysers:\n\n```bash\n$ sysdiag list analysers\nAnalyser Name         Analyser Description\n--------------------  -------------------------------------------------------------------------------\nall                   Run all analysers\napps                  Get list of Apps installed on the device\ndemo_analyser         Do something useful (DEMO)\nps_everywhere         List all processes we can find a bit everywhere.\nps_matrix             Makes a matrix comparing ps, psthread, taskinfo\ntimesketch            Generate a Timesketch compatible timeline\nwifi_geolocation      Generate GPS Exchange (GPX) of wifi geolocations\nwifi_geolocation_kml  Generate KML file for wifi geolocations\nyarascan              Scan the case folder using YARA rules ('./yara' or SYSDIAGNOSE_YARA_RULES_PATH)\n```\n\nRun analyser (make sure you run `parse all` before)\n\n```bash\n$ sysdiag -c 1 analyse timesketch\nExecution success, output saved in: cases/1/parsed_data/timesketch.jsonl\n```\n\n# Using the output\n\nMost of the parsers and analysers save their results in `jsonl` or `json` format. A few analysers use `txt` and more.\nExported data is stored in the `\u003ccases\u003e/\u003ccase_id\u003e/parsed_data` folder. You can configure your ingestion tool to automatically monitor and all that data.\n\nThe JSONL files are event based and (most often) structured with a a `timestamp` (unixtime) and `datetime` (isoformat) field. These can be used to build timelines.\n\n## Timesketch\n\nYou might want to visualise timelines which you can extract via sysdiagnose in [Timesketch](https://timesketch.org/guides/admin/install/). Do note that timesketch expects timestamps in microseconds, that's why we made the `timesketch` analyser.\n\nNote that for a reasonable sysdiagnose log output, we recommend the following base requirements:\n\n- Ubuntu 20.04 or higher\n- 128GB of RAM\n- 4-8 virtual CPUs\n- Minimum 64 GB of HDD space just for timesketch data (add some more GBs for the OS and OS upgrades, etc.)\n- SSDs (NVMEs) for the data.\n\n## Yarascan\n\nUsing YARA rules are an easy and flexible way of spotting 'evil', the Yarascan analyser will help you out with that. It looks for YARA rules within __.yar__ files saved in the `./yara` folder or in the one designated by the environment varirable `SYSDIAGNOSE_YARA_RULES_PATH`.\n\n# UnifiedLogs\n\nThis unifiedlogs parser tool is natively provided on a MacOS system. Fortunately some entities developed a linux compatible parser.\n\nBy default sysdiagnose will use the Apple unifiedlogs `log` binary.\nOn linux it expects the Mandiant developed UnifiedLogs tool to be present in the path. Follow below instructions to compile and install it on your system.\n\n## Building macos-UnifiedLogs for linux\n\nFirst, ensure `cargo` is installed so you can build rust projects.\n\n```bash\nsudo apt install cargo\n```\n\nNow you can download and compile the code:\n\n```bash\ngit clone https://github.com/mandiant/macos-UnifiedLogs\ncd macos-UnifiedLogs/examples/unifiedlog_iterator/\ncargo build --release\nsudo cp ../target/release/unifiedlog_iterator /usr/local/bin/\n```\n\nSee `unifiedlog_iterator --help` for more instructions to use the tool, or use it directly through sysdiagnose.\n\n# Troubleshooting\n\nBy default, whenever you [parse](#parsing-data-and-converting-it-to-a-usable-format) or [analyse](#analysers-to-process-parsed-data) data, the framework generates a log file in JSONL format in the subfolder `logs` located in the case folder.\n\n__Note:__ You can get that very same information in the output by adding the flag `-l DEBUG` to the parse/analyse command. But this is not very manageable when using all parsers/analysers.\n\nYou will be able to identify the execution of a parser/analyser by, at least, two entries generated by the main module (__module: main__) with an extra field named parser/analyser that contains the name of the parser/analyser.\n\nIn between those two entries, you may see any other produced by the parser/analyser, where the module, this time, will match the parser/analyser name.\n\n__Note:__ It is of utmost important that the parser/analyser provides logging to help troubleshooting potential issues. Please take a look to the [demo_parser](src/sysdiagnose/parsers/demo_parser.py) and the [demo_analyser](src/sysdiagnose/analysers/demo_analyser.py) files for inspiration.\n\nBelow you can find an example of traces within the log file.\n\n```json\n{\"levelname\": \"INFO\", \"module\": \"main\", \"message\": \"Parser 'demo_parser' started\", \"taskName\": null, \"parser\": \"demo_parser\", \"datetime\": \"2025-01-28T12:16:59.153931+01:00\", \"timestamp\": 1738063019.153931}\n{\"levelname\": \"INFO\", \"module\": \"demo_parser\", \"message\": \"Processing file ./cases/1/data/sysdiagnose_2024.09.18_11-33-57+0200_iPhone-OS_iPhone_21G93/demo_input_file.txt, new entry added\", \"taskName\": null, \"log_file\": \"./cases/1/data/sysdiagnose_2024.09.18_11-33-57+0200_iPhone-OS_iPhone_21G93/demo_input_file.txt\", \"entry\": \"{}\", \"datetime\": \"2025-01-28T12:16:59.201914+01:00\", \"timestamp\": 1738063019.201914}\n{\"levelname\": \"WARNING\", \"module\": \"demo_parser\", \"message\": \"Empty entry.\", \"taskName\": null, \"datetime\": \"2025-01-28T12:16:59.202071+01:00\", \"timestamp\": 1738063019.202071}\n{\"levelname\": \"INFO\", \"module\": \"main\", \"message\": \"Parser 'demo_parser' finished successfully\", \"taskName\": null, \"parser\": \"demo_parser\", \"result\": 0, \"datetime\": \"2025-01-28T12:16:59.202495+01:00\", \"timestamp\": 1738063019.2024949}\n\n```\n\n# Supported iOS versions\n\nTested on:\n\n- python 3.11\n- iOS13 (to be confirmed)\n- iOS14 (to be confirmed)\n- iOS15\n- iOS16\n- iOS17\n- iOS18\n\n# Contributors\n\n- Dario BORREGUERO RINCON (European Commission - EC DIGIT Cybersecurity Operation Centre)\n- David DURVAUX (European Commission - EC DIGIT Cybersecurity Operation Centre)\n- Aaron KAPLAN (European  Commission - EC DIGIT Cybersecurity Operation Centre)\n- Christophe VANDEPLAS (European Commission - EC DIGIT Cybersecurity Operation Centre)\n- Emilien  LE JAMTEL (CERT-EU)\n- Benoît ROUSSILLE (European Parliament)\n- For the Apollo library: https://github.com/mac4n6/APOLLO\n\n# Licence\n\nThis project is released under the European Public Licence\nhttps://commission.europa.eu/content/european-union-public-licence_en\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEC-DIGIT-CSIRC%2Fsysdiagnose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FEC-DIGIT-CSIRC%2Fsysdiagnose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEC-DIGIT-CSIRC%2Fsysdiagnose/lists"}