{"id":22680094,"url":"https://github.com/smdmts/embulk-filter-key_in_redis","last_synced_at":"2025-03-29T13:43:11.663Z","repository":{"id":62557894,"uuid":"99699504","full_name":"smdmts/embulk-filter-key_in_redis","owner":"smdmts","description":null,"archived":false,"fork":false,"pushed_at":"2017-08-28T05:19:01.000Z","size":93,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-06T21:46:21.679Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/smdmts.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-08T14:04:09.000Z","updated_at":"2017-08-14T07:53:50.000Z","dependencies_parsed_at":"2022-11-03T06:30:46.879Z","dependency_job_id":null,"html_url":"https://github.com/smdmts/embulk-filter-key_in_redis","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smdmts%2Fembulk-filter-key_in_redis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smdmts%2Fembulk-filter-key_in_redis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smdmts%2Fembulk-filter-key_in_redis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smdmts%2Fembulk-filter-key_in_redis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/smdmts","download_url":"https://codeload.github.com/smdmts/embulk-filter-key_in_redis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246193156,"owners_count":20738450,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-09T19:12:11.774Z","updated_at":"2025-03-29T13:43:11.645Z","avatar_url":"https://github.com/smdmts.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Key In Redis filter plugin for Embulk\n\nFiltering by aggregated the keys in included Redis's SET.\n\nThis plugin is designed to extract data set diff files used with the combination in below use cases.\n\n1. Use this plugin and output specified key's to redis.  \n    - https://github.com/smdmts/embulk-output-key_to_redis\n2. Input another data source and use filter key_in_redis plugin with specified key's then filtered the key's (or that hash).  \n    - this plugin\n\n \n## Overview\n\n* **Plugin type**: output\n* **Load all or nothing**: no\n* **Resume supported**: no\n* **Cleanup supported**: no\n\n## Configuration\n\n\n| name                                 | type        | required?  | default                  | description            |  \n|:-------------------------------------|:------------|:-----------|:-------------------------|:-----------------------|\n|  host                                | string      | optional   | \"127.0.0.1\"              | redis servers host     |\n|  port                                | integer     | optional   | \"6379\"                   | redis servers port     |\n|  db                                  | integer     | optional   | \"null\"                   | redis servers db       |\n|  redis_set_key                       | string      | required   |                          | redis of key of set name |\n|  load_on_memory                      | boolean     | optional   | \"false\"                  | load all data from redis *1 |\n|  appender                            | string      | optional   | \"-\"                      | multi key of appender  |\n|  match_as_md5                        | boolean     | optional   | \"false\"                  | smembers the value to converted md5 |\n|  replica_hosts                       | hash: Map\u003cString,Int\u003e    | optional   |             | list of replica redis servers host: port |\n|  key_with_index                      | hash: Map\u003cInt,String\u003e | required with key_with_index or json_key_with_index or only one || index with key name |\n|  json_key_with_index                 | hash: Map\u003cInt,String\u003e | required with key_with_index or json_key_with_index or only one || json columns's expanded key name |\n|  default_timezone                    | string      | optional   | UTC                      | |\n|  default_timestamp_format            | string      | optional   | %Y-%m-%d %H:%M:%S.%6N    | |\n\n\n*1: load_on_memory mode requires JVM memory as all records stored on Redis.\n\n\n## Example\n\n- inside redis\n```\nsadd redis_key \"ABC_1502590079312009_user_id_12345\"\n```\n\n- input json\n```json\n{  \n   \"device_id\":\"ABC\",\n   \"timestamp_micros\":1502590079312009,\n   \"params\":{  \n      \"UserID\":\"user_id_12345\"\n   }\n}\n```\n\n- definition's yaml\n```yaml\nfilters:\n  - type: expand_json\n    json_column_name: record\n    root: \"$.\"\n    stop_on_invalid_record: false\n    expanded_columns:\n      - { name: \"device_id\", type: string }\n      - { name: \"timestamp_micros\", type: long }\n      - { name: \"params\", type: json }\n  - type: \"key_in_redis\"\n    redis_set_key: redis_key\n    match_as_md5: false\n    key_with_index: \n      1: \"device_id\"\n      2: \"timestamp_micros\"\n    json_key_with_index:\n      3: \"UserID\" \nout:\n  type: \"stdout\"\n```\n\n- filter as redis command\n```\nsmembers redis_key \"ABC_1502590079312009_user_id_12345\"\n```\n\n- output \nNone output causes aggregated key is inside in the redis.\n\n## Build\n\n```\n$ ./gradlew gem  # -t to watch change of files and rebuild continuously\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmdmts%2Fembulk-filter-key_in_redis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmdmts%2Fembulk-filter-key_in_redis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmdmts%2Fembulk-filter-key_in_redis/lists"}