{"id":19035330,"url":"https://github.com/rahmanda/mercury-py","last_synced_at":"2026-04-24T12:32:30.301Z","repository":{"id":73394516,"uuid":"91339848","full_name":"rahmanda/mercury-py","owner":"rahmanda","description":"Kibana complementary script to help data cleaning against json formatted log","archived":false,"fork":false,"pushed_at":"2017-05-16T13:04:33.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-11T09:11:50.498Z","etag":null,"topics":["csv","json","kibana"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rahmanda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-05-15T13:16:22.000Z","updated_at":"2019-06-08T11:34:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"bfa5b691-16bf-4384-89e7-227ead08b729","html_url":"https://github.com/rahmanda/mercury-py","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/rahmanda/mercury-py","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahmanda%2Fmercury-py","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahmanda%2Fmercury-py/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahmanda%2Fmercury-py/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahmanda%2Fmercury-py/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rahmanda","download_url":"https://codeload.github.com/rahmanda/mercury-py/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rahmanda%2Fmercury-py/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32223933,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T10:26:35.452Z","status":"ssl_error","status_checked_at":"2026-04-24T10:25:27.643Z","response_time":64,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","json","kibana"],"created_at":"2024-11-08T21:50:07.517Z","updated_at":"2026-04-24T12:32:30.296Z","avatar_url":"https://github.com/rahmanda.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mercury-py\n\nMercury-py is complementary script for Kibana to help you data cleaning against json formatted log. It reads exported csv from [ElasticSearch CSV Exporter](https://chrome.google.com/webstore/detail/elasticsearch-csv-exporte/kjkjddcjojneaeeppobfolgojhohbpjn?hl=en) chrome extension, then produce clean csv file out of json formatted field.\n\n## How to use\n1. Install mercury-py first via `pip install git+https://github.com/rahmanda/mercury-py.git`.\n2. Run this command `mercury` on your console.\n3. Enter csv file and json rule file in your directory.\n4. Your exported csv file will be created on your current directory under name 'exported_csv.csv'.\n\n## Json rule\nBelow is valid example of json rule. You can also find it under `src/mercury_py/example_rule.json` directory on this repository.\n```json\n{\n  \"column_names\": [\"booking message\", \"timestamp\", \"first name\", \"title\", \"pax type\", \"last name\", \"birth date\"],\n  \"rules\": {\n    \"0\": {\n      \"filter\": [\n        [\"Status\", \"Message\"]\n      ]\n    },\n    \"1\": {\n      \"no_json\": true\n    },\n    \"2\": {\n      \"entry_point\": [\"Passengers\", {\"name\": \"0\", \"hash_type\": \"int\"}],\n      \"filter\": [\n        \"FirstName\",\n        \"Title\",\n        \"PaxType\",\n        \"LastName\",\n        \"BirthDate\"\n      ]\n    }\n  }\n}\n```\nKey `rules` contains rule of every column you want to extract. For json formatted column, we should provide `entry_point` and `filter` list. Both of lists can contain plain string, dictionary, or list value.\n\nIf we provide list, the program will trace all keys until the last key to get the value we want. Let say one of your field looks like this.\n```\n\"{\"\"Status\"\":{\"\"Message\"\":\"\"Booking failed.\"\"}}\"\n```\nAccording to this filter `[\"Status\", \"Message\"]`, it will produce value `Booking failed.`\n\nIf we provide dictionary, it will reformat the key or the value before adding to final result. Let say one of your field looks like this.\n```\n\"{\"\"Passengers\"\": [{\"\"FirstName\"\": \"\"Rahmanda\"\", \"\"Title\"\": \"\"MR\"\", \"\"PaxType\"\": \"\"ADT\"\", \"\"LastName\"\": \"\"Wibowo\"\", \"\"BirtDate\"\": \"\"2012-01-01\"\"}]}\"\n```\nAccording to this entry point rule `[\"Passengers\", {\"name\": \"0\", \"hash_type\": \"int\"}]`, we start fetching data from `['Passengers'][0]` key, then get the data based on filter rule.\n\nIf we don't want the program to parse certain field into json, we have to provide `\"no_json\": true`.\n\n## Options\n- `--csv`: Directory path of consummable csv (required).\n- `--rule`: Directory path of json rule (required).\n- `--csv-delimiter` : Specify csv delimiter. Default value ',' (optional).\n- `--export` : Directory path of exported csv. Default value 'exported_csv.csv' (optional).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frahmanda%2Fmercury-py","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frahmanda%2Fmercury-py","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frahmanda%2Fmercury-py/lists"}