{"id":30687603,"url":"https://github.com/cameronnewman/redis-dumper","last_synced_at":"2026-05-05T08:32:10.133Z","repository":{"id":308064661,"uuid":"1028080099","full_name":"cameronnewman/redis-dumper","owner":"cameronnewman","description":"Simple tool to dump data from Redis into a DuckDB queryable format","archived":false,"fork":false,"pushed_at":"2025-07-29T04:33:43.000Z","size":31,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-12T00:42:45.946Z","etag":null,"topics":["csv","duckdb","export","golang","parquet","redis"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cameronnewman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-29T02:23:52.000Z","updated_at":"2025-07-29T04:38:53.000Z","dependencies_parsed_at":"2025-08-04T00:11:11.758Z","dependency_job_id":"feeac753-cac9-4b36-a46f-44a9523aa835","html_url":"https://github.com/cameronnewman/redis-dumper","commit_stats":null,"previous_names":["cameronnewman/redis-dumper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cameronnewman/redis-dumper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cameronnewman%2Fredis-dumper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cameronnewman%2Fredis-dumper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cameronnewman%2Fredis-dumper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cameronnewman%2Fredis-dumper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cameronnewman","download_url":"https://codeload.github.com/cameronnewman/redis-dumper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cameronnewman%2Fredis-dumper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32642007,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-04T10:08:07.713Z","status":"online","status_checked_at":"2026-05-05T02:00:06.033Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","duckdb","export","golang","parquet","redis"],"created_at":"2025-09-02T00:03:27.969Z","updated_at":"2026-05-05T08:32:10.117Z","avatar_url":"https://github.com/cameronnewman.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Redis Dumper\n\nA high-performance tool for exporting Redis data to CSV or Parquet format with Hive-style partitioning for optimal DuckDB querying.\n\n## Features\n\n- Export Redis data to CSV or Parquet format\n- Memory-efficient streaming for large datasets\n- Hive-style partitioning for efficient querying\n- Support for all Redis data types (strings, hashes, sets, sorted sets, lists)\n- Configurable batch sizes and file rotation\n- TLS/SSL support\n- DuckDB-optimized output format\n\n## Installation\n\n```bash\ngo install github.com/cameronnewman/redis-dumper/cmd/dumper@latest\n```\n\nOr build from source:\n\n```bash\ngit clone https://github.com/cameronnewman/redis-dumper.git\ncd redis-dumper\ngo build -o dumper ./cmd/dumper\n```\n\n## Usage\n\nThe tool uses subcommands and environment variables for configuration.\n\n### Commands\n\n- `keys-only` - Export only key metadata (recommended for large datasets)\n- `pattern` - Export full data for keys matching a pattern\n- `full` - Export all data (use with caution on large datasets)\n\n### Basic Usage\n\nExport only key metadata:\n```bash\ndumper keys-only\n```\n\nExport keys matching a pattern:\n```bash\ndumper pattern \"user:*\"\n```\n\nExport all data:\n```bash\ndumper full\n```\n\n### Using Environment Variables\n\nConfigure via environment variables:\n```bash\nexport REDIS_URL=redis://localhost:6379/0\nexport OUTPUT_DIR=./export\nexport OUTPUT_FORMAT=csv\nexport BATCH_SIZE=5000\n\ndumper keys-only\n```\n\nOr inline:\n```bash\nREDIS_URL=redis://localhost:6379 OUTPUT_DIR=./export dumper pattern \"session:*\"\n```\n\n### TLS/SSL Support\n\nFor Redis with TLS:\n```bash\nREDIS_URL=rediss://user:pass@redis.example.com:6380/0 dumper keys-only\n```\n\nOr manually enable TLS:\n```bash\nexport REDIS_URL=redis://redis.example.com:6380\nexport ENABLE_TLS=true\nexport SKIP_TLS_VERIFY=true\ndumper keys-only\n```\n\n## Configuration\n\nAll configuration is done through environment variables:\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `REDIS_URL` | Redis connection URL | `redis://localhost:6379/0` |\n| `OUTPUT_DIR` | Output directory path | `/tmp/dumper` |\n| `OUTPUT_FORMAT` | Output format: csv or parquet | `parquet` |\n| `BATCH_SIZE` | Number of keys to process in each batch | `1000` |\n| `MAX_RECORDS_PER_FILE` | Maximum records per file before rotation | `100000` |\n| `ENABLE_TLS` | Enable TLS connection | `false` |\n| `SKIP_TLS_VERIFY` | Skip TLS certificate verification | `true` |\n\n### Redis URL Schemes\n\n- `redis://` - Plain connection\n- `rediss://` - TLS connection (automatically enables TLS)\n\n## Output Format\n\nData is exported with Hive-style partitioning:\n```\noutput/\n├── year=2024/\n│   └── month=01/\n│       └── day=15/\n│           └── hour=14/\n│               ├── redis_data_part_0001.csv\n│               └── redis_data_part_0002.csv\n└── export_metadata.json\n```\n\n### Schema\n\nAll Redis data is exported with a unified schema:\n\n| Column | Type | Description |\n|--------|------|-------------|\n| key | string | Redis key |\n| type | string | Redis data type |\n| value | string | Serialized value |\n| ttl_seconds | int64 | TTL in seconds (-1 if no TTL) |\n| exported_at | string | Export timestamp |\n| partition_id | int | Partition identifier |\n\n### Parquet Schema Details\n\nThe Parquet files use the following schema definition:\n\n```\nmessage redis_data {\n  optional binary key (STRING);\n  optional binary type (STRING);\n  optional binary value (STRING);\n  optional int64 ttl_seconds;\n  optional binary exported_at (STRING);\n  optional int32 partition_id;\n}\n```\n\n### Data Type Representations\n\nDifferent Redis data types are stored in the unified schema as follows:\n\n#### Strings\n- **key**: Original Redis key (e.g., `\"user:123\"`)\n- **type**: `\"string\"`\n- **value**: The actual string value\n\n#### Hashes\n- **key**: `\"{original_key}:field:{field_name}\"` (e.g., `\"user:123:field:email\"`)\n- **type**: `\"hash_field\"`\n- **value**: The field's value\n\n#### Sets\n- **key**: `\"{original_key}:member:{member_value}\"` (e.g., `\"tags:member:golang\"`)\n- **type**: `\"set_member\"`\n- **value**: The member value\n\n#### Sorted Sets (ZSets)\n- **key**: `\"{original_key}:member:{member_value}\"` (e.g., `\"leaderboard:member:player1\"`)\n- **type**: `\"zset_member\"`\n- **value**: `\"score={score},rank={rank}\"` (e.g., `\"score=95.5,rank=0\"`)\n\n#### Lists\n- **key**: `\"{original_key}:index:{index}\"` (e.g., `\"queue:index:0\"`)\n- **type**: `\"list_item\"`\n- **value**: The item value\n\n## Querying with DuckDB\n\n### Basic Queries\n\nQuery all exported data:\n```sql\n-- For Parquet files\nSELECT * FROM read_parquet('output/**/*.parquet');\n\n-- For CSV files\nSELECT * FROM read_csv('output/**/*.csv');\n```\n\nCount by data type:\n```sql\nSELECT type, COUNT(*) as count \nFROM read_parquet('output/**/*.parquet')\nGROUP BY type\nORDER BY count DESC;\n```\n\n### Querying String Keys\n\nFind all string values:\n```sql\nSELECT key, value, ttl_seconds \nFROM read_parquet('output/**/*.parquet')\nWHERE type = 'string'\nLIMIT 10;\n```\n\n### Querying Hash Fields\n\nGet all fields for a specific hash:\n```sql\n-- Extract the original key and field name\nSELECT \n    SPLIT_PART(key, ':field:', 1) as hash_key,\n    SPLIT_PART(key, ':field:', 2) as field_name,\n    value\nFROM read_parquet('output/**/*.parquet')\nWHERE type = 'hash_field'\n  AND key LIKE 'user:123:field:%'\nORDER BY field_name;\n```\n\nReconstruct hash objects:\n```sql\n-- Group hash fields into JSON objects\nSELECT \n    SPLIT_PART(key, ':field:', 1) as hash_key,\n    MAP_FROM_ENTRIES(\n        ARRAY_AGG(\n            ROW(\n                SPLIT_PART(key, ':field:', 2),\n                value\n            )\n        )\n    ) as fields\nFROM read_parquet('output/**/*.parquet')\nWHERE type = 'hash_field'\nGROUP BY SPLIT_PART(key, ':field:', 1)\nLIMIT 5;\n```\n\n### Querying Sets\n\nGet all members of a specific set:\n```sql\nSELECT \n    SPLIT_PART(key, ':member:', 1) as set_key,\n    value as member\nFROM read_parquet('output/**/*.parquet')\nWHERE type = 'set_member'\n  AND key LIKE 'tags:member:%'\nORDER BY member;\n```\n\nCount members per set:\n```sql\nSELECT \n    SPLIT_PART(key, ':member:', 1) as set_key,\n    COUNT(*) as member_count\nFROM read_parquet('output/**/*.parquet')\nWHERE type = 'set_member'\nGROUP BY SPLIT_PART(key, ':member:', 1)\nORDER BY member_count DESC;\n```\n\nFind sets containing a specific member:\n```sql\nSELECT DISTINCT SPLIT_PART(key, ':member:', 1) as set_key\nFROM read_parquet('output/**/*.parquet')\nWHERE type = 'set_member'\n  AND value = 'golang';\n```\n\n### Querying Sorted Sets\n\nGet leaderboard with scores:\n```sql\nSELECT \n    SPLIT_PART(key, ':member:', 1) as zset_key,\n    SPLIT_PART(key, ':member:', 2) as member,\n    CAST(SPLIT_PART(SPLIT_PART(value, 'score=', 2), ',', 1) AS DOUBLE) as score,\n    CAST(SPLIT_PART(value, 'rank=', 2) AS INTEGER) as rank\nFROM read_parquet('output/**/*.parquet')\nWHERE type = 'zset_member'\n  AND key LIKE 'leaderboard:%'\nORDER BY score DESC;\n```\n\n### Querying Lists\n\nGet list items in order:\n```sql\nSELECT \n    SPLIT_PART(key, ':index:', 1) as list_key,\n    CAST(SPLIT_PART(key, ':index:', 2) AS INTEGER) as index,\n    value\nFROM read_parquet('output/**/*.parquet')\nWHERE type = 'list_item'\n  AND key LIKE 'queue:%'\nORDER BY list_key, index;\n```\n\n### Advanced Queries\n\nFind keys expiring soon:\n```sql\nSELECT key, type, ttl_seconds, \n    ttl_seconds / 3600.0 as hours_remaining\nFROM read_parquet('output/**/*.parquet')\nWHERE ttl_seconds \u003e 0 \n  AND ttl_seconds \u003c 3600  -- Expiring within 1 hour\nORDER BY ttl_seconds;\n```\n\nAnalyze data distribution by partition:\n```sql\nSELECT \n    partition_id,\n    COUNT(*) as record_count,\n    COUNT(DISTINCT SPLIT_PART(key, ':', 1)) as unique_key_prefixes\nFROM read_parquet('output/**/*.parquet')\nGROUP BY partition_id\nORDER BY partition_id;\n```\n\nExport query results:\n```sql\n-- Export filtered data to a new Parquet file\nCOPY (\n    SELECT * FROM read_parquet('output/**/*.parquet')\n    WHERE type = 'hash_field' AND key LIKE 'user:%'\n) TO 'user_hashes.parquet' (FORMAT 'parquet');\n```\n\n## Setting Up DuckDB\n\n### Installation\n\nInstall DuckDB CLI:\n\n**macOS:**\n```bash\nbrew install duckdb\n```\n\n**Linux:**\n```bash\nwget https://github.com/duckdb/duckdb/releases/download/v0.10.0/duckdb_cli-linux-amd64.zip\nunzip duckdb_cli-linux-amd64.zip\nchmod +x duckdb\nsudo mv duckdb /usr/local/bin/\n```\n\n**Windows:**\nDownload from [DuckDB releases](https://github.com/duckdb/duckdb/releases)\n\n### Creating Views for Easy Querying\n\nStart DuckDB and create persistent views:\n\n```bash\n# Start DuckDB with a persistent database\nduckdb redis_export.db\n```\n\nIn the DuckDB shell:\n\n```sql\n-- Create a view for all Redis data\nCREATE VIEW redis_data AS \nSELECT * FROM read_parquet('./output/**/*.parquet');\n\n-- Create views for each data type\nCREATE VIEW redis_strings AS \nSELECT * FROM redis_data WHERE type = 'string';\n\nCREATE VIEW redis_hashes AS \nSELECT \n    SPLIT_PART(key, ':field:', 1) as hash_key,\n    SPLIT_PART(key, ':field:', 2) as field_name,\n    value,\n    ttl_seconds,\n    exported_at\nFROM redis_data \nWHERE type = 'hash_field';\n\nCREATE VIEW redis_sets AS \nSELECT \n    SPLIT_PART(key, ':member:', 1) as set_key,\n    value as member,\n    ttl_seconds,\n    exported_at\nFROM redis_data \nWHERE type = 'set_member';\n\nCREATE VIEW redis_zsets AS \nSELECT \n    SPLIT_PART(key, ':member:', 1) as zset_key,\n    SPLIT_PART(key, ':member:', 2) as member,\n    CAST(SPLIT_PART(SPLIT_PART(value, 'score=', 2), ',', 1) AS DOUBLE) as score,\n    CAST(SPLIT_PART(value, 'rank=', 2) AS INTEGER) as rank,\n    ttl_seconds,\n    exported_at\nFROM redis_data \nWHERE type = 'zset_member';\n\nCREATE VIEW redis_lists AS \nSELECT \n    SPLIT_PART(key, ':index:', 1) as list_key,\n    CAST(SPLIT_PART(key, ':index:', 2) AS INTEGER) as index,\n    value,\n    ttl_seconds,\n    exported_at\nFROM redis_data \nWHERE type = 'list_item';\n\n-- Show available views\nSHOW TABLES;\n```\n\n### Using the Views\n\nNow you can query Redis data more easily:\n\n```sql\n-- Count records by type\nSELECT COUNT(*) FROM redis_strings;\nSELECT COUNT(*) FROM redis_hashes;\nSELECT COUNT(*) FROM redis_sets;\n\n-- Query specific hash\nSELECT * FROM redis_hashes WHERE hash_key = 'user:123';\n\n-- Find all sets containing a member\nSELECT set_key FROM redis_sets WHERE member = 'golang';\n\n-- Get top 10 from leaderboard\nSELECT * FROM redis_zsets \nWHERE zset_key = 'leaderboard' \nORDER BY score DESC \nLIMIT 10;\n```\n\n### Performance Tips\n\n1. **Use Parquet format** - It's columnar and compressed, making queries much faster than CSV\n2. **Partition your queries** - Use the partition_id or date filters when possible\n3. **Create indexes** for frequently queried columns:\n   ```sql\n   CREATE INDEX idx_key_prefix ON redis_data (SPLIT_PART(key, ':', 1));\n   ```\n4. **Use EXPLAIN** to understand query plans:\n   ```sql\n   EXPLAIN SELECT * FROM redis_sets WHERE member = 'test';\n   ```\n\n## Running Locally\n\n### Quick Start with Docker Compose\n\nThe easiest way to test Redis Dumper is using the included docker-compose setup with test data:\n\n```bash\n# Start Redis with test data\ndocker-compose up -d\n\n# Wait for data to load\nsleep 5\n\n# Run the dumper on test data\ngo run ./cmd/dumper keys-only\n\n# Or export full data for specific patterns\ngo run ./cmd/dumper pattern \"user:*\"\ngo run ./cmd/dumper pattern \"product:*\"\n\n# Check the output\nls -la ./output/\n```\n\nThe test data includes:\n- String values (configs, sessions, cached pages)\n- Hashes (user profiles, products, orders)\n- Sets (tags, categories, user skills)\n- Sorted sets (leaderboards, trending items, view counts)\n- Lists (queues, logs, user history)\n- Keys with TTLs\n- Complex JSON structures\n\n### Manual Docker Setup\n\nAlternatively, you can manually start Redis:\n\n```bash\n# Start a Redis instance\ndocker run -d --name redis-local -p 6379:6379 redis:latest\n\n# Add some test data\ndocker exec -it redis-local redis-cli SET mykey \"Hello World\"\ndocker exec -it redis-local redis-cli HSET user:123 name \"Alice\" email \"alice@example.com\"\ndocker exec -it redis-local redis-cli SADD fruits \"apple\" \"banana\" \"orange\"\n\n# Run the dumper\ngo run ./cmd/dumper keys-only\n```\n\n### Full Export Example\n\n```bash\nexport REDIS_URL=redis://localhost:6379\nexport OUTPUT_DIR=./local-export\nexport OUTPUT_FORMAT=parquet\nexport BATCH_SIZE=5000\n\n# Export full data for all keys\ngo run ./cmd/dumper full\n\n# Or export data matching a pattern\ngo run ./cmd/dumper pattern \"user:*\"\n```\n\n### Testing with DuckDB\n\nAfter exporting the test data, you can explore it with DuckDB:\n\n```bash\n# View all data types in the export\nduckdb -c \"SELECT type, COUNT(*) as count FROM read_parquet('./output/**/*.parquet') GROUP BY type ORDER BY count DESC;\"\n\n# Check the leaderboard\nduckdb -c \"\n  SELECT \n    SPLIT_PART(key, ':member:', 2) as player,\n    CAST(SPLIT_PART(SPLIT_PART(value, 'score=', 2), ',', 1) AS DOUBLE) as score\n  FROM read_parquet('./output/**/*.parquet')\n  WHERE key LIKE 'leaderboard:global:%'\n  ORDER BY score DESC;\n\"\n\n# View user hashes\nduckdb -c \"\n  SELECT \n    SPLIT_PART(key, ':field:', 1) as user,\n    SPLIT_PART(key, ':field:', 2) as field,\n    value\n  FROM read_parquet('./output/**/*.parquet')\n  WHERE type = 'hash_field' AND key LIKE 'user:%'\n  ORDER BY user, field;\n\"\n\n# Find all programming-related tags\nduckdb -c \"\n  SELECT DISTINCT \n    SPLIT_PART(key, ':member:', 1) as set_name,\n    value as tag\n  FROM read_parquet('./output/**/*.parquet')\n  WHERE type = 'set_member' \n    AND value IN ('python', 'golang', 'javascript', 'rust', 'java', 'typescript')\n  ORDER BY set_name, tag;\n\"\n```\n\n## Development\n\n### Requirements\n\n- Go 1.21 or higher\n- Docker (for running tests with make commands)\n- Redis (for local testing)\n- Make\n\n### Building\n\n```bash\nmake build\n```\n\n### Testing\n\n```bash\nmake go-test\n```\n\n### Linting\n\n```bash\nmake go-lint\n```\n\n### Formatting\n\n```bash\nmake go-fmt\n```\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details on:\n\n- Setting up your development environment\n- Running tests and linting\n- Submitting pull requests\n- Reporting issues\n\n## License\n\nMIT License - see LICENSE file for details.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcameronnewman%2Fredis-dumper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcameronnewman%2Fredis-dumper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcameronnewman%2Fredis-dumper/lists"}