{"id":21661008,"url":"https://github.com/karanpratapsingh/scale-etl","last_synced_at":"2025-07-10T01:42:26.013Z","repository":{"id":214264716,"uuid":"695393859","full_name":"karanpratapsingh/scale-etl","owner":"karanpratapsingh","description":"Partition, Transform, Load, and Search large CSV files","archived":false,"fork":false,"pushed_at":"2023-12-27T18:00:37.000Z","size":1787,"stargazers_count":10,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-25T19:01:35.259Z","etag":null,"topics":["concurrency","data","etl","golang"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/karanpratapsingh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-23T03:43:00.000Z","updated_at":"2025-01-18T22:08:11.000Z","dependencies_parsed_at":"2023-12-27T02:28:26.021Z","dependency_job_id":"c603d982-6261-46af-a876-996a7125fe65","html_url":"https://github.com/karanpratapsingh/scale-etl","commit_stats":null,"previous_names":["karanpratapsingh/scale-etl"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karanpratapsingh%2Fscale-etl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karanpratapsingh%2Fscale-etl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karanpratapsingh%2Fscale-etl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karanpratapsingh%2Fscale-etl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/karanpratapsingh","download_url":"https://codeload.github.com/karanpratapsingh/scale-etl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248492868,"owners_count":21113163,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","data","etl","golang"],"created_at":"2024-11-25T09:43:12.783Z","updated_at":"2025-04-11T23:10:27.077Z","avatar_url":"https://github.com/karanpratapsingh.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ScaleETL\n\nPartition, Transform, Load, and Search large CSV files.\n\n![system](docs/diagrams/system.png)\n\nThe core idea behind this architecture is to be able to adjust parameters to scale the system as per the amount of computing available in the environment. For instance, the system can process more batches on a CPU with more threads.\n\n## Development\n\nThe system is implemented as a standard Go CLI application, checkout the `Makefile` for available commands.\n\nGenerate sample test files using the `make generate_sample_data` command for development, benchmarks etc.\n\n```sh\n$ make generate_sample_data\nGenerated sample data\n4.0K    samples/sample_1k.csv\n240K    samples/sample_10k.csv\n2.3M    samples/sample_100k.csv\n 23M    samples/sample_1m.csv\n230M    samples/sample_10m.csv\n2.3G    samples/sample_100m.csv\n 23G    samples/sample_1b.csv\n```\n\n## Build\n\n`amd64` and `arm64` binaries are available for `linux`, `windows`, and `darwin` via the [build github action](https://github.com/karanpratapsingh/scale-etl/actions/workflows/build.yml).\n\n## Features\n\nBenchmarks were done on Apple M2 CPU with 16 GB Memory. `scripts/pandas_benchmark.sh` has benchmark implementation for [pandas](https://pandas.pydata.org).\n\n### Partition\n\nPartition input CSV file into multiple smaller files.\n\n![partitioner](docs/diagrams/partitioner.png)\n\n**Partition Manifest**\n\nThe manifest includes information about how the CSV file is partitioned into smaller chunks, specifying the start and end buffer indices of each partition.\n\n```js\n{\n  \"total_rows\": \u003cint\u003e,\n  \"partition_size\": \u003cint\u003e,\n  \"partitions\": [\n    { \"start\": \u003cint\u003e, \"end\": \u003cint\u003e },\n    { \"start\": \u003cint\u003e, \"end\": \u003cint\u003e }\n  ]\n}\n```\n\n**Usage**\n\n```sh\n$ scale-etl partition [command options] [arguments...]\n\nOPTIONS:\n   --file-path value       Input CSV file path\n   --partition-dir value   Output directory for partition manifest files (default: \"partitions\")\n   --partition-size value  Partition size (default: 0)\n```\n\n**Example**\n\n```sh\n$ scale-etl partition --file-path samples/sample_10m.csv --partition-size 100000\n```\n\n**Benchmark**\n\n| Sample Size | Partition Size | Pandas    | ScaleETL   | Improvement |\n| ----------- | -------------- | --------- | ---------- | ----------- |\n| 100k        | 10,000         | 34.7ms    | 5.0497ms   | 6.94x       |\n| 1m          | 10,000         | 316.1ms   | 50.6128ms  | 6.26x       |\n| 10m         | 100,000        | 3.020s    | 398.8537ms | 7.5x        |\n| 100m        | 1,000,000      | 41.2236s  | 3.8394s    | 10.72x      |\n| 1b          | 1,000,000      | 462.6593s | 40.1622s   | 11.52x      |\n\n### Transform\n\nTransform partitions into a particular format (`dynamodb`, `parquet`, `json`, `csv`).\n\n![transformer](docs/diagrams/transformer.png)\n\n**Schema**\n\nYAML structure represents a schema definition used by the transformer for a given CSV file. Optional fields such as `table_name`, `key` can be used in certain scenarios. For example, the `table_name` field is can for transform type `dynamodb`.\n\n```yaml\ntable_name: \u003cstr\u003e [optional]\nkey: \u003cstr\u003e [optional]\ncolumns:\n  - \u003cname\u003e: \u003ctype\u003e\n```\n\n**Usage**\n\n```sh\n$ scale-etl transform [command options] [arguments...]\n\nOPTIONS:\n   --file-path value       Input CSV file path\n   --partition-dir value   Output directory for partition manifest files (default: \"partitions\")\n   --batch-size value      Number of partitions to be processed concurrently (default: 5)\n   --segment-size value    Size of segment each partition will be divided into (default: 0)\n   --schema-path value     Schema file path (default: \"schema.yaml\")\n   --delimiter value       Delimiter character (default: \",\")\n   --no-header             CSV file does not have a header row (default: false)\n   --transform-type value  Output format of the transform (default: \"csv\")\n   --output-dir value      Output directory for transformed files (default: \"output\")\n```\n\n**Example**\n\n```sh\n$ scale-etl transform --file-path samples/sample_10m.csv --segment-size 10000\n```\n\n**Benchmark**\n\n| Sample Size | Batch Size | Segment Size | Pandas   | ScaleETL   | Improvement |\n| ----------- | ---------- | ------------ | -------- | ---------- | ----------- |\n| 100k        | 10         | 10,000       | 181.2ms  | 14.0836ms  | 12.87x      |\n| 1m          | 10         | 10,000       | 789.4ms  | 86.9535ms  | 9.09x       |\n| 10m         | 20         | 10,000       | 7.9503s  | 821.9298ms | 9.67x       |\n| 100m        | 20         | 100,000      | 91.2204s | 6.7518s    | 13.50x      |\n| 1b          | 20         | 100,000      | 988.845s | 70.2302s   | 14.06x      |\n\n### Search\n\nSearches partitions for a specific pattern.\n\n![search-interface](docs/diagrams/search-interface.png)\n\n**Usage**\n\n```sh\n   --pattern value        Search pattern\n   --output value         Output CSV file path (default: \"matches.csv\")\n   --file-path value      Input CSV file path\n   --partition-dir value  Output directory for partition manifest files (default: \"partitions\")\n   --batch-size value     Number of partitions to be processed concurrently (default: 5)\n   --segment-size value   Size of segment each partition will be divided into (default: 0)\n   --schema-path value    Schema file path (default: \"schema.yaml\")\n   --delimiter value      Delimiter character (default: \",\")\n   --no-header            CSV file does not have a header row (default: false)\n```\n\n**Example**\n\n```sh\n$ scale-etl search --file-path samples/sample_10m.csv --segment-size 10000 --pattern abc\n```\n\n**Benchmark**\n\n| Sample Size | Segment Size | Pandas     | ScaleETL   | Improvement |\n| ----------- | ------------ | ---------- | ---------- | ----------- |\n| 100k        | 10,000       | 212.7ms    | 12.1287ms  | 18.73x      |\n| 1m          | 10,000       | 1.1510s    | 96.2360ms  | 11.98x      |\n| 10m         | 100,000      | 11.5798s   | 541.4265ms | 21.41x      |\n| 100m        | 1,000,000    | 146.9870s  | 4.7879s    | 30.72x      |\n| 1b          | 1,000,000    | 1492.5549s | 48.6179s   | 30.68x      |\n\n### Load\n\nLoad transformed segments concurrently.\n\n![loader](docs/diagrams/loader.png)\n\n**Usage**\n\n```sh\n$ scale-etl load [command options] [arguments...]\n\nOPTIONS:\n   --file-path value    Input CSV file path\n   --output-dir value   Output directory for transformed files (default: \"output\")\n   --pool-size value    Number of concurrent calls of the specified script (default: 0)\n   --script-path value  Path of script to be executed for each segment\n```\n\n**Example**\n\n```sh\n$ scale-etl load --file-path samples/sample_10m.csv --pool-size 50 --script-path ./scripts/sample_load_script.sh\n```\n\n**Benchmark**\n\nLoader benchmark can be quite subjective as there are a lot of external factors to consider like data store latency and network bandwidth. Below is a sample benchmark for PostgreSQL 16.1 running on Docker 4.25.2 (`scripts/sample_pg_load_script.sh`).\n\n| Sample Size | Segment Size | Time       |\n| ----------- | ------------ | ---------- |\n| 100k        | 10,000       | 12.1287ms  |\n| 1m          | 10,000       | 96.2360ms  |\n| 10m         | 100,000      | 541.4265ms |\n| 100m        | 1,000,000    | 4.7879s    |\n| 1b          | 1,000,000    | 48.6179s   |\n\n## Future Scope\n\nPotential areas of future development and improvement:\n\n- In partitions streaming without reading the entire file and creating a partition manifest.\n- Remove, Add or Derive a column from an existing columns.\n- View/stream any partition within a specified range.\n- RESTful interface for all core features.\n- Multiple input formats (xls, JSON, etc) support.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaranpratapsingh%2Fscale-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkaranpratapsingh%2Fscale-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaranpratapsingh%2Fscale-etl/lists"}