{"id":31785549,"url":"https://github.com/ydb-platform/ydb-parallel-processor","last_synced_at":"2026-07-28T03:31:09.061Z","repository":{"id":295980881,"uuid":"987056251","full_name":"ydb-platform/ydb-parallel-processor","owner":"ydb-platform","description":"YDB Parallel Record Processor","archived":false,"fork":false,"pushed_at":"2025-09-27T11:57:26.000Z","size":124,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-27T13:28:20.193Z","etag":null,"topics":["batch-processing","ydb"],"latest_commit_sha":null,"homepage":"https://ydb.tech","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ydb-platform.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-20T14:05:59.000Z","updated_at":"2025-09-27T11:57:30.000Z","dependencies_parsed_at":"2025-06-19T10:30:49.191Z","dependency_job_id":"dc8b22ce-177f-454b-901b-3ccacc2eb624","html_url":"https://github.com/ydb-platform/ydb-parallel-processor","commit_stats":null,"previous_names":["zinal/ydb-parallel-exporter","ydb-platform/ydb-parallel-processor"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/ydb-platform/ydb-parallel-processor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-parallel-processor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-parallel-processor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-parallel-processor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-parallel-processor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ydb-platform","download_url":"https://codeload.github.com/ydb-platform/ydb-parallel-processor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-parallel-processor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279003724,"owners_count":26083610,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch-processing","ydb"],"created_at":"2025-10-10T11:58:10.254Z","updated_at":"2026-07-28T03:31:09.047Z","avatar_url":"https://github.com/ydb-platform.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# YDB parallel records batch processor\n\n[See the Releases page for downloads](https://github.com/zinal/ydb-parallel-processor/releases).\n\nThis tool provides basic capabilities for parallel job processing of records stored in [YDB](https://ydb.tech) tables.\nIt can be used as a standalone program, or embedded into the user application (which needs to be based on Java-compatible stack) as a library.\n\nThe basic workflow implemented by the tool consists of two phases:\n\n1. The tool grabs the record keys using the initial query, applying the basic filtering. Output paging can optionally be used, as [described in the YDB documentation](https://ydb.tech/docs/en/dev/paging). For paging, the additional \"paging\" query needs to be configured.\n2. The tool uses the collected record keys to apply the additional processing, with the following options:\n  * Table joins and additional computation may be applied to grab and enrich the records, and output the full records in CSV or JSON format.\n  * Data records can be updated, such as setting some fields in the existing records or inserting new records.\n\nPhase 2 is executed using the configurable executor pool, providing parallel execution for enhanced performance. This means that the output records cannot generally be sorted, as the output from different parallel jobs will be provided in the undefined order.\n\n## Requirements\n\n- Java SDK 17 or later (for running and building)\n- Apache Maven (for building)\n\n## Running the tool as a standalone program\n\n```bash\n./Run.sh connection.xml jobdef.xml\n... or ...\n./Run.sh connection.xml jobdef.xml vars.xml\n```\n\n* The first parameter should point to the file with the connection parameters.\n* The second parameter should point to the file with the job definition.\n* The optional third parameter can be missing, or should point to the file with the substitution variables.\n\nFor information about the substitution variables, see the section at the end of this file (`README.md`), or in the corresponding section of `README-ru.md`.\n\n## Embedding the tool into the user program\n\nEmbedding the tool into the user program can be performed using the following Maven dependency:\n\n```xml\n        \u003cdependency\u003e\n            \u003cgroupId\u003etech.ydb.app\u003c/groupId\u003e\n            \u003cartifactId\u003eydb-parallel-processor\u003c/artifactId\u003e\n            \u003cversion\u003e1.3\u003c/version\u003e\n        \u003c/dependency\u003e\n```\n\n\u003e [!WARNING]\n\u003e Currently the tool artifacts are not published to Maven Central.\n\u003e In order to use the dependency shown above, local build is necessary.\n\nUsing the class `tech.ydb.app.parproc.Tool`, the following can be implemented:\n\n```java\nJobDef job = new JobDef();\njob.setMainQuery(\"SELECT ...\");\njob.setDetailsQuery(\"SELECT ...\");\njob.getDetailsInput().add(\"id\");\n...\nProperties propsConn = new Properties();\npropsConn.setProperty(\"ydb.url\", \"grpcs://ydb01.localdomain:2135/cluster1/testdb\");\n...\ntry (YdbConnector yc = new YdbConnector(propsConn)) {\n    try (Tool app = new Tool(yc, job)) {\n        app.run();\n    }\n}\n```\n\n## Connection parameters\n\nConnection parameters are provided either programmatically (as `java.util.Properties` object), or via XML properties file.\n\n| **Parameter** | **Description** |\n| --- | --- |\n| `ydb.url` | YDB connection URL |\n| `ydb.cafile` | Path to the custom CA certificate file |\n| `ydb.auth.mode` | [Authentication mode](https://ydb.tech/docs/en/reference/ydb-sdk/auth#auth-provider) (NONE, ENV, STATIC, METADATA, SAKEY) |\n| `ydb.auth.username` | Username for STATIC authentication |\n| `ydb.auth.password` | Password for STATIC authentication |\n| `ydb.auth.sakey` | Service account key file name for SAKEY authentication |\n| `ydb.preferLocalDc` | Prefer local or nearest datacenter for connections (true or false, default false) |\n\nExample properies file:\n\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003c!DOCTYPE properties SYSTEM \"http://java.sun.com/dtd/properties.dtd\"\u003e\n\u003cproperties\u003e\n\u003centry key=\"ydb.url\"\u003egrpcs://ydb01.localdomain:2135/cluster1/testdb\u003c/entry\u003e\n\u003centry key=\"ydb.cafile\"\u003e/home/ydbadmin/Work/cluster1/tls/ca.crt\u003c/entry\u003e\n\u003centry key=\"ydb.auth.mode\"\u003eSTATIC\u003c/entry\u003e\n\u003centry key=\"ydb.auth.username\"\u003eroot\u003c/entry\u003e\n\u003centry key=\"ydb.auth.password\"\u003e???\u003c/entry\u003e\n\u003centry key=\"ydb.poolSize\"\u003e1000\u003c/entry\u003e\n\u003c/properties\u003e\n```\n\n## Processing parameters\n\nProcessing parameters are provided either programmatically (as `tech.ydb.app.parproc.JobDef` object), or via XML configuration file.\n\n| **Parameter** | **Description** |\n| --- | --- |\n| `worker-count` | Number of parallel workers |\n| `queue-size` | Size of the queue between the main query and the detail query |\n| `batch-limit` | Maximum number of keys in a batch for details query |\n| `output-format` | Output format (CSV, TSV, JSON, CUSTOM1 - see below for details). Default is CSV if omitted  |\n| `output-file` | Output file name, with value '-' for STDOUT. Default is '-' if omitted |\n| `isolation` | Transaction isolation level (SERIALIZABLE_RW, SNAPSHOT_RO, STALE_RO, ONLINE_RO, ONLINE_INCONSISTENT_RO). Default is SERIALIZABLE_RW if omitted |\n| `timeout` | Query timeout, in milliseconds, for query-main, query-page or query-detail, default -1 (unlimited) |\n| `query-main` | Main query (executed first) |\n| `query-page` | Paging query (optional). Requires key sorting and row count limit both for itself and for the main query. |\n| `query-details` | Detail query. Takes the keys from main and page queries, and applies extra logic. |\n| `input-page` | List of input columns for the page query (subset of columns from main and page queries). |\n| `input-details` | List of input columns for the detail query (subset of columns from main and page queries), optional |\n\nIn the tags `query-main`, `query-page` and `query-details` an optional `timeout` attribute can also be specified, setting the maximum execution time of the query in milliseconds. If the specified time is exceeded, the query execution is aborted and retried. This timeout mechanism helps protect against performance degradation caused by rare slowdowns of query execution.\n\nSupported output formats:\n- `CSV` - regular CSV according to RFC4180, with comma delimited, double quotes, and CR-LF line separators\n- `TSV` - tab-delimoted format, double quotes, CR-LF\n- `JSON` - a separate JSON document per line, CR delimited\n- `CUSTOM1` - CSV-style format with 0x19 as field delimiter, LF (0x0A) as record delimiter and minimized (mostly no) quotes.\n\nExample configuration file:\n\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cydb-parallel-exporter\u003e\n    \u003cworker-count\u003e5\u003c/worker-count\u003e\n    \u003cqueue-size\u003e100\u003c/queue-size\u003e\n    \u003cbatch-limit\u003e1000\u003c/batch-limit\u003e\n    \u003coutput-format\u003eCSV\u003c/output-format\u003e\n    \u003coutput-file\u003eexample1.csv\u003c/output-file\u003e\n    \u003cisolation\u003eSNAPSHOT_RO\u003c/isolation\u003e\n    \u003cquery-main timeout=\"3000\"\u003e\u003c![CDATA[\nSELECT sys_update_tv, id\nFROM my_documents VIEW ix_sys_update_tv\nWHERE sys_update_tv \u003e= Timestamp('2021-01-01T00:00:00Z')\n  AND sys_update_tv  \u003c Timestamp('2021-01-02T00:00:00Z')\nORDER BY sys_update_tv, id   -- Mandatory sorting on primary key or secondary index\nLIMIT 1000;   -- Mandatory limit on the number of output records\n]]\u003e \u003c/query-main\u003e\n\n    \u003c!-- Колонки для параметров страничного запроса --\u003e\n    \u003cinput-page\u003e\n        \u003ccolumn-name\u003esys_update_tv\u003c/column-name\u003e\n        \u003ccolumn-name\u003eid\u003c/column-name\u003e\n    \u003c/input-page\u003e\n    \u003cquery-page timeout=\"3000\"\u003e\u003c![CDATA[\nDECLARE $input AS Struct\u003csys_update_tv:Timestamp?, id:Text\u003e;\nSELECT sys_update_tv, id\nFROM my_documents VIEW ix_sys_update_tv\nWHERE (sys_update_tv, id) \u003e ($input.sys_update_tv, $input.id) -- Paging condition\n  AND sys_update_tv \u003c Timestamp('2021-01-02T00:00:00Z') -- Repeat the filter from the main query\nORDER BY sys_update_tv, id   -- Mandatory sorting on primary key or secondary index\nLIMIT 1000;   -- Mandatory limit on the number of output records\n]]\u003e \u003c/query-page\u003e\n    \u003cinput-details\u003e\n        \u003ccolumn-name\u003eid\u003c/column-name\u003e\n    \u003c/input-details\u003e\n    \u003cquery-details timeout=\"10000\"\u003e\u003c![CDATA[\nDECLARE $input AS List\u003cStruct\u003cid:Text\u003e\u003e;\nSELECT\n    documents.*,\n    d1.attr1 AS d1_attr1,\n    d2.attr1 AS d2_attr1\nFROM AS_TABLE($input) AS input\nINNER JOIN my_documents VIEW PRIMARY KEY AS documents\n  ON input.id=documents.id\nLEFT JOIN my_dict1 AS d1\n  ON d1.key=documents.dict1\nLEFT JOIN my_dict2 AS d2\n  ON d2.key=documents.dict2\nWHERE documents.some_state IN ('ONE'u, 'TWO'u, 'THREE'u, 'FOUR'u)\n  AND documents.input_tv IS NOT NULL;\n]]\u003e \u003c/query-details\u003e\n\u003c/ydb-parallel-exporter\u003e\n```\n\n## Substitution variables\n\nThe tool can optionally apply variable substitutions to the XML file with job definition.\n\nSubstitutions are applied after parsing the XML content (so the file needs to be a valid XML before substititions), but before converting it to the job definition.\n\nThe variables can be specified as `${varname}` in attribute values, text values and CDATA section values, but not in the tag names or attribute names.\n\nExample properies file containing some substitution variables:\n\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003c!DOCTYPE properties SYSTEM \"http://java.sun.com/dtd/properties.dtd\"\u003e\n\u003cproperties\u003e\n\u003centry key=\"file_name\"\u003eoutput_with_some_suffix.csv\u003c/entry\u003e\n\u003centry key=\"worker_count\"\u003e10\u003c/entry\u003e\n\u003centry key=\"start_time\"\u003e2020-01-01T00:00:00Z\u003c/entry\u003e\n\u003centry key=\"finish_time\"\u003e2026-01-01T00:00:00Z\u003c/entry\u003e\n\u003c/properties\u003e\n```\n\n## Usage examples\n\nThe YDB parallel processor tool can be used for various data processing tasks. Below are examples for common use cases.\n\n### 1. Filling a new column with computed data\n\nThis example shows how to fill a new column in a table with data computed from other columns using SQL queries.\n\nSuppose that the table `some_table` has got a new column called `new_field` using the following statement:\n\n```SQL\nALTER TABLE some_table ADD COLUMN new_field Text;\n```\n\nThe `new_field` column contains just nulls, and there is a requirement that it should be filled with some data computed from the other fields (and possibly by requesting the data from other tables as well).\n\n**Job definition file (`fill-column.xml`):**\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cydb-parallel-exporter\u003e\n    \u003cworker-count\u003e10\u003c/worker-count\u003e\n    \u003cqueue-size\u003e100\u003c/queue-size\u003e\n    \u003cbatch-limit\u003e1000\u003c/batch-limit\u003e\n    \u003coutput-format\u003eTSV\u003c/output-format\u003e\n    \u003coutput-file\u003e-\u003c/output-file\u003e\n    \u003cisolation\u003eSERIALIZABLE_RW\u003c/isolation\u003e\n\n    \u003c!-- Select records that need the new column filled --\u003e\n    \u003cquery-main timeout=\"120000\"\u003e\u003c![CDATA[\nSELECT id FROM some_table\nWHERE new_field IS NULL\nORDER BY id;\n]]\u003e \u003c/query-main\u003e\n\n    \u003cinput-details\u003e\n        \u003ccolumn-name\u003eid\u003c/column-name\u003e\n    \u003c/input-details\u003e\n\n    \u003c!-- Update the new column with computed data --\u003e\n    \u003cquery-details timeout=\"10000\"\u003e\u003c![CDATA[\nDECLARE $input AS List\u003cStruct\u003cid:Text\u003e\u003e;\nUPSERT INTO some_table\n  SELECT i.id AS id,\n      t.old_field || ' 'u || a.some_name AS new_field\n  FROM AS_TABLE($input) AS i\n  JOIN some_table AS t\n    ON t.id = i.id\n  LEFT JOIN another_table AS a\n    ON a.id = t.ref_a;\n]]\u003e \u003c/query-details\u003e\n\u003c/ydb-parallel-exporter\u003e\n```\n\n**Execution:**\n```bash\n./Run.sh connection.xml fill-column.xml\n```\n\n### 2. Archiving old records to another table\n\nThis example demonstrates archiving older records to an archive table and then deleting them from the original table.\n\nSuppose that the original table `documents` has to be cleared every month from the documents older that 3 months.\nThe old documents should be put into the `documents_archive` table, and deleted from the original `documents` table.\n\n**Job definition file (`archive-records.xml`):**\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cydb-parallel-exporter\u003e\n    \u003cworker-count\u003e5\u003c/worker-count\u003e\n    \u003cqueue-size\u003e100\u003c/queue-size\u003e\n    \u003cbatch-limit\u003e1000\u003c/batch-limit\u003e\n    \u003coutput-format\u003eTSV\u003c/output-format\u003e\n    \u003coutput-file\u003e-\u003c/output-file\u003e\n    \u003cisolation\u003eSERIALIZABLE_RW\u003c/isolation\u003e\n\n    \u003c!-- Select records to archive (older than 92 days, ~3 months) --\u003e\n    \u003cquery-main timeout=\"120000\"\u003e\u003c![CDATA[\nSELECT id FROM documents\nWHERE created_date \u003c CurrentUtcTimestamp() - Interval('P92D');\n]]\u003e \u003c/query-main\u003e\n\n    \u003cinput-details\u003e\n        \u003ccolumn-name\u003eid\u003c/column-name\u003e\n    \u003c/input-details\u003e\n\n    \u003c!-- Archive records and then delete from original table --\u003e\n    \u003cquery-details timeout=\"10000\"\u003e\u003c![CDATA[\nDECLARE $input AS List\u003cStruct\u003cid:Text\u003e\u003e;\n\n-- First, insert into archive table\nUPSERT INTO documents_archive\nSELECT d.*\nFROM AS_TABLE($input) AS i\nJOIN documents AS d ON d.id = i.id;\n\n-- Then, delete from original table\nDELETE FROM documents\nON SELECT * FROM AS_TABLE($input);\n]]\u003e \u003c/query-details\u003e\n\u003c/ydb-parallel-exporter\u003e\n```\n\n**Execution:**\n```bash\n./Run.sh connection.xml archive-records.xml\n```\n\n### 3. Extracting large amounts of data to CSV files\n\nThis example shows how to extract large datasets to CSV files with parallel processing and paging for better performance.\n\nSuppose that there is a huge amount of data stored in the normalized table structure, and the BI system needs the de-normalized data to fill the data mart. Every day a job is run to extract the last two days' data into the CSV file in the form of a wide table, containing all the required attritutes to fill the data mart. The file is then loaded into the BI system using its native tools.\n\nJob parameters are put into the separate substitution variables file, which is re-generated every time before job run.\n\n**Job substitution variable file (`export-to-csv_params.xml`):**\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003c!DOCTYPE properties SYSTEM \"http://java.sun.com/dtd/properties.dtd\"\u003e\n\u003cproperties\u003e\n\u003centry key=\"file_name\"\u003eexported_data.csv\u003c/entry\u003e\n\u003centry key=\"worker_count\"\u003e10\u003c/entry\u003e\n\u003centry key=\"start_time\"\u003e2020-01-01T00:00:00Z\u003c/entry\u003e\n\u003centry key=\"finish_time\"\u003e2026-01-01T00:00:00Z\u003c/entry\u003e\n\u003c/properties\u003e\n```\n\n**Job definition file (`export-to-csv.xml`):**\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cydb-parallel-exporter\u003e\n    \u003cworker-count\u003e${worker_count}\u003c/worker-count\u003e\n    \u003cqueue-size\u003e100\u003c/queue-size\u003e\n    \u003cbatch-limit\u003e1000\u003c/batch-limit\u003e\n    \u003coutput-format\u003eCSV\u003c/output-format\u003e\n    \u003coutput-file\u003e${file_name}\u003c/output-file\u003e\n    \u003coutput-encoding\u003eUTF-8\u003c/output-encoding\u003e\n    \u003cisolation\u003eSNAPSHOT_RO\u003c/isolation\u003e\n\n    \u003c!-- Main query with paging for large datasets --\u003e\n    \u003cquery-main timeout=\"5000\"\u003e\u003c![CDATA[\nSELECT sys_update_tv, id\nFROM my_documents VIEW ix_sys_update_tv\nWHERE sys_update_tv \u003e= Timestamp('${start_time}')\n  AND sys_update_tv \u003c Timestamp('${finish_time}')\nORDER BY sys_update_tv, id\nLIMIT 1000;\n]]\u003e \u003c/query-main\u003e\n\n    \u003c!-- Paging query for handling large datasets --\u003e\n    \u003cinput-page\u003e\n        \u003ccolumn-name\u003esys_update_tv\u003c/column-name\u003e\n        \u003ccolumn-name\u003eid\u003c/column-name\u003e\n    \u003c/input-page\u003e\n    \u003cquery-page timeout=\"5000\"\u003e\u003c![CDATA[\nDECLARE $input AS Struct\u003csys_update_tv:Timestamp?, id:Text\u003e;\nSELECT sys_update_tv, id\nFROM my_documents VIEW ix_sys_update_tv\nWHERE (sys_update_tv, id) \u003e ($input.sys_update_tv, $input.id)\n  AND sys_update_tv \u003c Timestamp('${finish_time}')\nORDER BY sys_update_tv, id\nLIMIT 1000;\n]]\u003e \u003c/query-page\u003e\n\n    \u003c!-- Detail query with joins for enriched data and extra filters --\u003e\n    \u003cinput-details\u003e\n        \u003ccolumn-name\u003eid\u003c/column-name\u003e\n    \u003c/input-details\u003e\n    \u003cquery-details timeout=\"5000\"\u003e\u003c![CDATA[\nDECLARE $input AS List\u003cStruct\u003cid:Text\u003e\u003e;\nSELECT\n    documents.*,\n    d1.attr1 AS d1_attr1,\n    d2.attr1 AS d2_attr1\nFROM AS_TABLE($input) AS input\nINNER JOIN my_documents VIEW PRIMARY KEY AS documents\n  ON input.id=documents.id\nLEFT JOIN my_dict1 AS d1\n  ON d1.key=documents.dict1\nLEFT JOIN my_dict2 AS d2\n  ON d2.key=documents.dict2\nWHERE documents.some_state IN ('ONE'u, 'TWO'u, 'THREE'u, 'FOUR'u)\n  AND documents.input_tv IS NOT NULL;\n]]\u003e \u003c/query-details\u003e\n\u003c/ydb-parallel-exporter\u003e\n```\n\n**Execution:**\n```bash\n./Run.sh connection.xml export-to-csv.xml export-to-csv_params.xml\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fydb-platform%2Fydb-parallel-processor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fydb-platform%2Fydb-parallel-processor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fydb-platform%2Fydb-parallel-processor/lists"}