{"id":19403684,"url":"https://github.com/dongju93/sysmon-to-rocksdb","last_synced_at":"2026-04-07T20:32:45.614Z","repository":{"id":186639854,"uuid":"675452282","full_name":"dongju93/sysmon-to-rocksdb","owner":"dongju93","description":"Query Elasticsearch to retrieve data, save it to CSV files, store it in RocksDB, and then use GraphQL to fetch the data.","archived":false,"fork":false,"pushed_at":"2024-03-09T13:07:02.000Z","size":35538,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-25T00:33:40.004Z","etag":null,"topics":["csv","elasticsearch","graphql","javascript","nextjs","postgresql","rocksdb","rust","typescript"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dongju93.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-08-07T00:37:48.000Z","updated_at":"2023-11-26T16:55:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"fe01b0ed-e238-4213-b0fd-34313ee4640a","html_url":"https://github.com/dongju93/sysmon-to-rocksdb","commit_stats":null,"previous_names":["dongju93/elarocks","dongju93/sysmon-to-rocksdb"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dongju93/sysmon-to-rocksdb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dongju93%2Fsysmon-to-rocksdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dongju93%2Fsysmon-to-rocksdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dongju93%2Fsysmon-to-rocksdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dongju93%2Fsysmon-to-rocksdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dongju93","download_url":"https://codeload.github.com/dongju93/sysmon-to-rocksdb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dongju93%2Fsysmon-to-rocksdb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31528472,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T16:28:08.000Z","status":"ssl_error","status_checked_at":"2026-04-07T16:28:06.951Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","elasticsearch","graphql","javascript","nextjs","postgresql","rocksdb","rust","typescript"],"created_at":"2024-11-10T11:29:59.542Z","updated_at":"2026-04-07T20:32:45.594Z","avatar_url":"https://github.com/dongju93.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Ultimate goal diagram. Ver.1\n```mermaid\nflowchart TB\n\tsubgraph Windows11\n\t  Winlogbeat--Read--\u003eSysmon\n\tend\n\tsubgraph LogStorageServer\n\t\tWinlogbeat--Push--\u003eElasticsearch\n\tend\n\tsubgraph PreprocessingServer\n\t\tElasticsearch\u003c--Request/Response--\u003eDataFetchBatch:::foo\n\t\tDataFetchBatch:::foo--Write--\u003eCSV\n\t\tDataStoreBatch:::foo--Read--\u003eCSV\n\tend\n\tsubgraph DatabaseServer\n\t\tDataStoreBatch:::foo--Store--\u003eRocksDB\n\tend\n\tsubgraph MiddlewareServer\n\t\tDataViewBinary:::foo\u003c--Iter./Fetch--\u003eRocksDB\n\t\tGraphQL:::bar\u003c--Execute/Return--\u003eDataViewBinary:::foo\n\tend\n\tsubgraph ApplicaionServer\n\t\tWebApplication:::foobar\u003c--Query/Mutate--\u003eGraphQL:::bar\n\tend\n\tBrowser--Access--\u003eWebApplication:::foobar\n\nclassDef foo stroke:#f00\nclassDef bar stroke:#0f0\nclassDef foobar stroke:#00f\n```\n\n# 1. Elasticsearch data to .csv file\n\nFirst, you need to collect [SYSMON](https://learn.microsoft.com/ko-kr/sysinternals/downloads/sysmon) data with [WINLOGBEAT](https://www.elastic.co/kr/beats/winlogbeat) and stored with [ELASTICSEARCH](https://www.elastic.co/kr/elasticsearch)   \nSecond, this code will extract data to CSV files with delimiter \"\\t\"\n\nit's parsing \"message\" field with \"agent.name\", \"agent.id\" field\n\nmay require to modify maximum size of search query, default is 10000\n```\n// replace with your Index name\nPUT /.ds-winlogbeat-8.8.2-2023.08.06-000001/_settings\n{\n    \"max_result_window\": 1000000\n}\n```\n\nPlease refer to the comments in the code for detailed explanation\n\n## Quickstart\n1. You need to create \"elastic.rs\" files, located \"/src/envs\"\n- /src/envs/elasric.rs\n```\npub const ES_URL_SECRET: \u0026str = \"YOUR ELASTICSEARCH URL\";\npub const ID_SECRET: \u0026str = \"YOUR ELASTICSEARCH USERNAME (default is elaseic)\";\npub const PW_SECRET: \u0026str = \"YOUR ELASTICSEARCH PASSWORD\";\n```\n2. You need set your index name, the name may start with \".ds-winlogbeat\" if you setup winlogbeat to elasticsearch automatically\nand if index is multiple, set numbers and write index names within array\n- /src/envs/env.rs\n```\npub const INDICES: [\u0026str; 1] = [\"YOUR INDEX NAME\"];\n\n// if you have three indexes\n// When the CSV is saved, if the file does not exist, a title line is added as the file is created, and if the file exists, the parsed data rows are added without the title line.\n// To explain further, if you specify multiple indexes, the file will be created from the first index and the data will be added to the file created from the second index.\npub const INDICES: [\u0026str; 3] = [\"YOUR INDEX NAME 1\", \"YOUR INDEX NAME 2\", \"YOUR INDEX NAME 3\"];\n```\n3. Set timestamp, query size, save location\n- /src/envs/env.rs\n```\npub const TIMESTAMP_START: \u0026str = \"START TIMESTAMP\";\npub const TIMESTAMP_END: \u0026str = \"END TIMESTAMP\";\n\npub const SIZE: usize = QUERY SIZE;\n\n// between SAVELOCATION, CSVNAME event code will automatically generated\npub const SAVELOCATION: \u0026str = \"SAVE LOCATION\";\npub const CSVNAME: \u0026str = \"FILENAME WITH FILE EXTENSTION (extenstion is .csv)\";\n```\n4. Execute code\n```\ncargo build\ncargo run --bin main\n```\n\n* Tip : Checking field types when selecting a wildcard type\n```\n// replace with your Index name\n// When checking the message field type\nGET /.ds-winlogbeat-8.8.2-2023.08.06-000001/_mapping/field/message\n```\n\u003c/br\u003e\u003c/br\u003e\n\n# 2. Data(.csv files) to RocksDB\n1. Place csv files location\n2. configure RocksDB location and execute code\n```\ncargo run --bin rocks\n```\n\u003c/br\u003e\u003c/br\u003e\n\n# 3. Data view on GraphQL(raw query)\n1. change directory\n```\ncd graphql\n```\n2. Run graphQL server\n```\nnpm run dev\n```\n3. Access apollo graphql server on 4000 port\n```\nhttp://localhost:4000\n```\n\u003c/br\u003e\u003c/br\u003e\n\n# 4. Data view on web(GUI)\n1. change directory\n```\ncd webapp\n```\n2. Run node server\n```\nnpm run dev\n```\n3. Access Next.js on 3000 port\n```\nhttp://localhost:3000\n```\n\u003c/br\u003e\u003c/br\u003e\n\n# 99. Todo\n1. auto fetch elasticsearch data every one minute\n2. if elasticsearch data exceed max than fetch more\n3. auto import data to RocksDB right after csv parsing\n4. data fetch from web application implements with react-query\n5. cursor based pagination\n6. web application api optimize\n7. add union on graphql for multiple data types\n8. fetch data from RocksDB using iteration (detach PostgreSQL) - speed test required - ✅\n9. apply lib.rs to main.rs for crate maintenance\n\u003c/br\u003e\u003c/br\u003e\n\n## Ultimate goal diagram. Ver.2\n```mermaid\nflowchart TB\n\tsubgraph Linux\n\t  Filebeat--Read--\u003eSysmon\n\t\tFilebeat--Read--\u003eSuricata\n\t\tFilebeat--Read--\u003eZeek\n\t\tFilebeat--Read--\u003eNetflow\n\tend\n\tsubgraph LogSendingServer\n\t\tFilebeat--Push--\u003eLogstash--Push--\u003eRedis\n\tend\n\tsubgraph PreprocessingServer\n\t\tRedis--Stream--\u003eDataStorePipe:::foo\n\tend\n\tsubgraph DatabaseServer\n\t\tPostgreSQL\u003c--Replication--\u003eReplica\n\t\tDataStorePipe:::foo--Store--\u003eRocksDB\n\tend\n\tsubgraph MiddlewareServer\n\t\tDataViewBinary:::foo\u003c--Iter./Fetch--\u003eRocksDB\n\t\tGraphQL:::bar\u003c--Execute/Return--\u003eDataViewBinary:::foo\n\t\tLargeDataViewBinary:::foo\u003c--Iter./Fetch--\u003eRocksDB\n\t\tGraphQL:::bar\u003c--Execute/Return--\u003eLargeDataViewBinary:::foo\n\t\tGraphQL:::bar\u003c--SQL/Return--\u003ePostgreSQL\n\tend\n\tsubgraph ApplicaionServer\n\t\tWebApplication1:::foobar\u003c--Query/Mutate--\u003eGraphQL:::bar\n\t\tWebApplication2:::foobar\u003c--Query/Mutate--\u003eGraphQL:::bar\n\t\tNginx--Proxy--\u003eWebApplication1:::foobar\n\t\tNginx--Proxy--\u003eWebApplication2:::foobar\n\tend\n\tBrowser--Access--\u003eNginx\n\nclassDef foo stroke:#f00\nclassDef bar stroke:#0f0\nclassDef foobar stroke:#00f\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdongju93%2Fsysmon-to-rocksdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdongju93%2Fsysmon-to-rocksdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdongju93%2Fsysmon-to-rocksdb/lists"}