{"id":31055531,"url":"https://github.com/nextronsystems/thor2timesketch","last_synced_at":"2026-03-11T13:18:40.943Z","repository":{"id":304530086,"uuid":"960681768","full_name":"NextronSystems/thor2timesketch","owner":"NextronSystems","description":"thor2ts – A utility to convert THOR logs to Timesketch’s required format.","archived":false,"fork":false,"pushed_at":"2025-06-12T08:12:18.000Z","size":87,"stargazers_count":1,"open_issues_count":2,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-08-29T22:15:55.993Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NextronSystems.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-04T21:33:04.000Z","updated_at":"2025-06-29T21:22:06.000Z","dependencies_parsed_at":"2025-07-13T18:36:08.436Z","dependency_job_id":"70353a08-4136-4298-906f-b5f0ff30f5e6","html_url":"https://github.com/NextronSystems/thor2timesketch","commit_stats":null,"previous_names":["nextronsystems/thor2timesketch"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/NextronSystems/thor2timesketch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NextronSystems%2Fthor2timesketch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NextronSystems%2Fthor2timesketch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NextronSystems%2Fthor2timesketch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NextronSystems%2Fthor2timesketch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NextronSystems","download_url":"https://codeload.github.com/NextronSystems/thor2timesketch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NextronSystems%2Fthor2timesketch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275207874,"owners_count":25423896,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-15T02:00:09.272Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-15T04:53:12.505Z","updated_at":"2026-03-11T13:18:40.897Z","avatar_url":"https://github.com/NextronSystems.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# **THOR APT SCANNER** to Timesketch\nThis log conversion utility makes it easy to import [THOR](https://www.nextron-systems.com/thor/) logs into [Timesketch](https://timesketch.org/). Combining **THOR** findings on a shared timeline it enables cybersecurity analysts to enhance detection and analysis of malicious activity.\n\n---\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Quick Start](#quick-start)\n3. [Installation](#installation)\n   - [Prerequisites](#prerequisites)\n   - [Installation Steps](#steps)\n4. [Usage](#usage)\n   - [Command-Line Arguments](#command-line-arguments)\n   - [Examples](#examples)\n5. [Configuration for Timesketch Ingestion](#configuration-for-timesketch-ingestion)\n6. [Filter Configuration](#filter-configuration)\n   - Standard THOR logs (JSON v1 / v2)\n   - Audit-trail logs\n7. [Input and Output Files](#input-and-output-files)\n   - Input Files\n   - Output File\n8. [Ingesting into Timesketch](#ingesting-into-timesketch)\n   - Manual Upload `jsonl`\n   - Automatic Ingestion (`-s, --sketch`)\n9. [Technical Details](#technical-details)\n     - THOR JSON v1/v2\n     - Audit-trail Logs\n10. [Troubleshooting](#troubleshooting)\n   - Issues and solutions\n11. [Contributing](#contributing)\n    - How to contribute\n12. [License](./LICENSE)\n13. [Support](#support)\n\n---\n## Overview\n\n\n[![thor2ts](https://img.shields.io/badge/dynamic/json?label=thor2ts\u0026query=%24.info.version\u0026url=https://pypi.org/pypi/thor2timesketch/json)](https://pypi.org/project/thor2timesketch/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)\n\n\n**thor2ts** is a lightweight CLI utility that converts THOR security scanner logs into Timesketch-compatible JSONL format by:\n\n* Extracting relevant fields from **THOR** logs\n\n* Generating entries with the required Timesketch fields: message, datetime, and timestamp_desc\n\n* Handling **THOR** events with multiple timestamps by creating separate entries for each timestamp\n\n---\n## Quickstart with `thor2ts`\n\n#### Install the tool\n```bash\npip install thor2timesketch\n```\n#### Convert a THOR log file → Timesketch-ready JSONL\n```bash\nthor2ts thor_scan.json -o thor_events.jsonl\n```\n#### Convert and ingest directly into a (new or existing) sketch\n```bash\nthor2ts thor_scan.json -s \"THOR APT SCANNER\"\n```\n\u003e For filtering, batching, and other options, see [Usage](#usage).\n\n---\n## Installation\n### Prerequisites\nMake sure you have the following installed on your system:\n- [Git](https://git-scm.com/downloads)\n- [Python 3.9](https://www.python.org/downloads/) or higher\n- [Python venv package](https://docs.python.org/3/tutorial/venv.html)\n *(e.g., on Ubuntu/Debian: `sudo apt install python3-venv`)*\n- [THOR JSON logs](https://thor-manual.nextron-systems.com/en/latest/usage/output-options.html#json-output-json) \n\u003e Note: Scan with [--utc](https://thor-manual.nextron-systems.com/en/latest/usage/output-options.html#utc) parameter to ensure timestamps are in UTC format.\n\n### Steps\n1. Create a virtual environment:\n\n   * Linux / macOS (bash / zsh)\n    ```bash\n    python3 -m venv thor2ts-venv\n    ```\n   * Windows (cmd / powershell)\n    ```cmd\n    py -3 -m venv thor2ts-venv\n    ```\n2. Activate the virtual environment `thor2ts-venv`:\n\n   * Linux / macOS (bash / zsh)\n    ```bash\n    source thor2ts-venv/bin/activate\n    ```\n   * Windows (cmd / powershell)\n    ```cmd\n    thor2ts-venv\\Scripts\\activate\n    ```\n3. Install thor2timesketch package:\n    ```bash\n    pip install thor2timesketch\n    ```\n\n4. Future Use\n\n    To use `thor2ts` in the new terminal, activate the virtual environment, (see `step 2 - Activate the virtual environment` above).\n\n---\n## Usage\nOnce the virtual environment is active, you can run the tool from the command line:\n\n```bash\nthor2ts \u003cINPUT_FILE\u003e [OPTIONS]\n```\n### Command-Line Arguments\n\n| Argument                         | Description                                                                                                             |\n|----------------------------------|-------------------------------------------------------------------------------------------------------------------------|\n| `\u003cINPUT_FILE\u003e`                   | Path to the **THOR** JSON log file. **Required**.                                                                       |\n| `-o, --output-file \u003cJSONL_FILE\u003e` | Save the converted **THOR** logs to the specified JSONL output file. **Optional**.                                      |\n| `-s, --sketch \u003cID\\|NAME\u003e`        | Ingest directly into the specified Timesketch sketch (by ID or name). Auto-creates the sketch if missing. **Optional**. |\n| `-b, --buffer-size \u003cN\u003e`          | Set the Timesketch importer buffer size (batch size) for ingestion. **Optional**.                                       |\n| `-F, --filter \u003cYAML_FILE\u003e`       | Specify a YAML filter to select which **THOR** events are ingested. **Optional**.                                       |\n| `--generate-filter`              | Generate `thor_filter.yaml` by extracting filters from **THOR** v1/v2 logs or using a default template. **Optional**.   |\n| `-v, --verbose`                  | Enable verbose debugging output. **Optional**.                                                                          |\n| `--version`                      | Display the current `thor2ts` version. **Optional**.                                                                    |\n\n### Examples\n| Scenario                           | Command                                                            |\n|------------------------------------|--------------------------------------------------------------------|\n| Convert to JSONL Output File       | `thor2ts thor_scan.json -o mapped_events.jsonl`                    |\n| Convert \u0026 Ingest to Sketch         | `thor2ts thor_scan.json -s \"THOR APT SCANNER\"`                     |\n| Set Custom Buffer Size             | `thor2ts thor_scan.json -s \"THOR APT SCANNER\" -b 100000`           |\n| Convert, Filter \u0026 Ingest to Sketch | `thor2ts thor_scan.json -F thor_filter.yaml -s \"THOR APT SCANNER\"` |\n| Extract Filter Template (file)     | `thor2ts input_v1.json --generate-filter`                          |\n| Generate Default Filter Template   | `thor2ts --generate-filter`                                        |\n| Enable Debug Mode                  | `thor2ts thor_scan.json -s \"THOR APT SCANNER\" --verbose`           |\n\n---\n## Configuration for Timesketch Ingestion\n\nWhen you ingest for the first time (`-s, --sketch`), you will be prompted to enter your Timesketch connection settings:\n\n1. **host_uri**\n   URL of your Timesketch server (e.g. `https://timesketch.example.com`)\n\n2. **auth_mode**\n   Authentication mode:\n   - `userpass` (username/password)\n   - `oauth` (OAuth2)\n\n3. **username**\n   Timesketch USERNAME\n\n4. **password**\n   Timesketch password (**Note:** It will be tokenized and stored securely)\n\nThis creates two configuration files in the user's home directory $HOME/:\n\n| File                   | Purpose                                             |\n|------------------------|-----------------------------------------------------|\n| `~/.timesketch.token`  | Encrypted authentication tokens                     |\n| `~/.timesketchrc`      | Connection settings (host, auth mode, credentials)  |\n\n### `~/.timesketchrc`\n```ini\n[timesketch]\nhost_uri = https://timesketch.example.com\nusername = USERNAME\nverify = True\nclient_id = \nclient_secret = \nauth_mode = userpass\ncred_key = \u003cgenerated_key\u003e\n```\nFor more detailed information about the Timesketch API client configuration and usage, please check out the [Timesketch API client documentation](https://timesketch.org/developers/api-client/).\n\n___\n## Filter Configuration\n\nThor2timesketch supports two filter scopes. The usage of filters is optional and sane defaults are used (that can be explicitly written to a config using the `--generate-filter` flag):\n\n### 1. [Standard THOR logs (JSON v1 / v2)](https://thor-manual.nextron-systems.com/en/latest/usage/output-options.html#json-output-json)\n\n- **filters.levels**\n  A list of THOR log `level` values to include (e.g., `Alert`, `Warning`).\n\n- **filters.modules**\n  - `include`: THOR log `module` names to include\n  - `exclude`: THOR log `module` names to exclude\n\n### 2. [Audit-trail logs](https://thor-manual.nextron-systems.com/en/latest/usage/output-options.html#output-format)\n\n- **filters.audit**\n  - `info` — ingest all audit information entries\n  - `findings` — ingest audit findings entries (then apply `filters.levels` + `filters.modules`)\n\n### Example `thor_filter.yaml`\n\n```yaml\nfilters:\n  levels:\n    - Alert\n    - Warning\n  modules:\n    include:\n      - Antivirus\n      - Firewall\n    exclude:\n      - Debug\n  audit:\n    - info\n    - findings\n```\n\n---\n## Input and Output Files\n\nLogs generated by [THOR APT SCANNER v10.7](https://www.nextron-systems.com/thor/).\n### Input Files\n\n- [**THOR JSON v1.0.0**](https://thor-manual.nextron-systems.com/en/latest/usage/output-options.html#json-output-json) (generated by THOR 10.7 and earlier by default with `--jsonfile`)\n- [**THOR JSON v2.0.0**](https://thor-manual.nextron-systems.com/en/latest/usage/output-options.html#json-output-json) (can be generated by THOR 10.7 on request `--jsonv2`)\n- [**Audit-trail** logs](https://thor-manual.nextron-systems.com/en/latest/usage/output-options.html#output-format) (very verbose log; new in v10.7; generate with `--audit-trail`)\n\n### Output File\n\n- **Timesketch-formatted JSONL**\n  - If the target filename does not end with `.jsonl`, the extension will be automatically changed to `.jsonl`.\n  - If the file already exists, new events are **appended** rather than overwritten.\n### Warning\n\u003e Timesketch accepts only **JSON** files with a `.jsonl` extension. [Timesketch documentation](https://timesketch.org/guides/user/import-from-json-csv/)\n\n---\n## Ingesting into Timesketch\n\n### 1. Manual Upload `jsonl`\n- **Timesketch Web UI**\n- [**Timesketch command line client (CLI)**](https://timesketch.org/guides/user/cli-client/)\n### 2. Automatic Ingestion (`-s, --sketch`)\nAdd `-s, --sketch \u003cID|NAME\u003e` to your `thor2ts` command and it will:\n\n1. **Create or find** the specified sketch\n2. **Push** the mapped events directly\n3. **Wait** up to 60 seconds for indexing to finish\n   - If indexing completes in time, you can go to the sketch immediately\n   - Otherwise, ingestion continues in the background\n4. **Buffer size** is set to 50,000 events by default, but you can adjust it with `-b, --buffer-size \u003cN\u003e`.\n___\n## Technical Details\n### Field Mapping Logic\n\nWhen a THOR record contains multiple timestamp fields, each timestamp is extracted into its own Timesketch event.\n\u003eAll events derived from the same THOR log share a common `event_group_id` (a UUID) so you can correlate primary and secondary events.\n\n### 1. THOR JSON v1/v2\n\n#### **THOR event**\n  - `message` ← original `\"message\"` (e.g. `\"Malicious user details found\"`)\n  - `datetime` ← `\"time\"` (e.g. `2025-05-07T11:45:01+00:00`)\n  - `timestamp_desc` ← `THOR scan timestamp`\n  - `event_group_id` ← UUID generated per THOR event\n  - `tag` ← `[\"thor\", \u003cLevel\u003e]` (e.g. `[\"thor\",\"Alert\"]`)\n  - `**fields` ← all other fields from THOR log\n\n- **Secondary events** (for each timestamp in THOR event)\n  - `message` ← original `\"message\"` (e.g. `\"Malicious user details found\"`)\n  - `datetime` ← time (e.g. `\"last_logon\"`)\n  - `timestamp_desc` ← `\"\u003cmodule\u003e - \u003cfield\u003e\"` (e.g. `Users - last_logon`)\n  - `event_group_id` ← same UUID as THOR event\n  - `tag` ← `[\"ts_extra\", \u003cLevel\u003e]`\n\n### 2. Audit-trail Logs\n\n#### **Info entries**\n\nFor each timestamp in `info`:\n- `message` ← `Name` (e.g. `\"File\"`)\n- `datetime` ← timestamp value (e.g. `2025-05-07T11:45:01+00:00`)\n- `timestamp_desc` ← timestamp key (e.g. `accessed`)\n- `tag` ← `[\"audit_info\"]`\n- `event_group_id` ← UUID for this audit record\n- **First event** includes all other info fields\n\n#### **Findings entries**\n\nFor each finding under `findings`:\n- `message` ← finding’s `Message` (e.g. `\"Malware file found\"`)\n- `datetime` ← timestamp value\n- `timestamp_desc` ← `\"\u003cModule\u003e - \u003cfield\u003e\"` (e.g. `Filescan - created`)\n- `tag` ← `[\"findings\", \u003cLevel\u003e]` (e.g. `[\"findings\",\"Alert\"]`)\n- `event_group_id` ← UUID for this audit record\n- **First event** includes all other finding fields\n\u003eNOTE: Use `event_group_id` to correlate first and secondary events from the same THOR log or audit-trail log.\n\n## Troubleshooting\n_**Issues recorded on 20.05.2025**_\n\n### 1. Multiple data sources created\n- **Symptom:** Every ~50 000 events shows up as a separate data source in the sketch.\n- **Cause:** The importer’s default batch size is [50 000 events](https://github.com/google/timesketch/blob/master/importer_client/python/timesketch_import_client/importer.py) per upload.\n- **Solution:**\n  1. Use the `-b, --buffer-size` argument to increase the batch size.\n     ```bash\n     thor2ts input.json -s \"THOR APT SCANNER\" -b 100000\n     ```\n  2. Convert to JSONL:\n     ```bash\n     thor2ts input.json -o mapped_events.jsonl\n     ```\n     Ingest the JSONL file into Timesketch using the CLI importer:\n     ```bash\n     timesketch_importer --sketch_id \u003cID\u003e --threshold_entry 100000 mapped_events.jsonl\n     ```\n     \u003e Warning: Consider RAM size when increasing batch size.\n\n### 2. Timesketch host becomes unresponsive while ingesting THOR logs\n- **Symptom:** High memory/CPU use, shell freezes, or Timesketch web UI becomes unresponsive during ingestion.\n- **Cause:** OpenSearch (Timesketch backend) [allocates ~50 % of RAM for its JVM heap](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/auto-tune.html?utm_source=chatgpt.com), on \u003c 16 GB systems this leaves too little memory for the OS and the *N* event-buffer used by the importer.\n- **Solution:**\n  - Run on a host with ≥ 16 GB RAM (recommended) for the default 50_000 event buffer size. Timesketch explicitly states 8GB is a minimum and [\"the more the better\"](https://timesketch.org/guides/admin/install/#:~:text=,setup%20SSL%20for%20the%20webserver)\n  - Import in smaller batches using the `-b, --buffer-size` argument.\n\n### 3. JSON Line format required\n- **Symptom:** JSON parse errors.\n- **Cause:** Input is pretty-printed JSON or a JSON array, not newline-delimited.\n- **Solution:**\n  - Use the unmodified THOR JSON output format or dump the THOR logs into JSON Lines format.\n\n### 4. Web UI upload errors\n- **Symptom:** Web UI rejects large `JSONL entries` `“Unterminated string in JSON at position ...”`.\n- **Cause:** Browser-based uploader [can’t handle very large files](https://github.com/google/timesketch/issues/3243) or large JSONL entries\n- **Solution**:\n  - Ingest directly using `thor2ts` with the `-s, --sketch` argument.\n  - Import via [CLI importer](https://timesketch.org/guides/user/cli-client/) for already mapped THOR events from a `JSONL` file:\n  ```bash\n  timesketch_importer --sketch_id \u003cID\u003e mapped_events.jsonl\n  ```\n---\n## Contributing\nContributions to `thor2ts` are welcome! To contribute:\n1. Fork the repository.\n2. Create a feature branch.\n3. Submit a pull request with your improvements or bug fixes.\n\n---\n## Support\nIf you encounter any issues or have questions, please open an issue in the [GitHub repository](https://github.com/NextronSystems/thor2timesketch/issues).\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnextronsystems%2Fthor2timesketch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnextronsystems%2Fthor2timesketch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnextronsystems%2Fthor2timesketch/lists"}