{"id":19704694,"url":"https://github.com/datastax/dsbulk","last_synced_at":"2025-04-09T13:03:16.173Z","repository":{"id":38019861,"uuid":"97133378","full_name":"datastax/dsbulk","owner":"datastax","description":"DataStax Bulk Loader (DSBulk) is an open-source, Apache-licensed, unified tool for loading into and unloading from Apache Cassandra(R), DataStax Astra and DataStax Enterprise (DSE)","archived":false,"fork":false,"pushed_at":"2024-07-10T22:15:04.000Z","size":11772,"stargazers_count":87,"open_issues_count":50,"forks_count":30,"subscribers_count":14,"default_branch":"1.x","last_synced_at":"2025-04-02T08:34:35.543Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datastax.png","metadata":{"files":{"readme":"README.md","changelog":"changelog/README.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-13T14:42:53.000Z","updated_at":"2025-03-02T13:52:48.000Z","dependencies_parsed_at":"2024-06-26T23:58:50.315Z","dependency_job_id":"10a62d34-be4f-46a1-b80c-ab22c43561f1","html_url":"https://github.com/datastax/dsbulk","commit_stats":null,"previous_names":[],"tags_count":31,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fdsbulk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fdsbulk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fdsbulk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fdsbulk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datastax","download_url":"https://codeload.github.com/datastax/dsbulk/tar.gz/refs/heads/1.x","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248045230,"owners_count":21038553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T21:24:06.821Z","updated_at":"2025-04-09T13:03:16.139Z","avatar_url":"https://github.com/datastax.png","language":"Java","funding_links":[],"categories":["Java"],"sub_categories":[],"readme":"# DataStax Bulk Loader Overview\n\nThe DataStax Bulk Loader tool (DSBulk) is a unified tool for loading into and unloading from \nCassandra-compatible storage engines, such as OSS Apache Cassandra\u0026reg;, DataStax Astra and DataStax \nEnterprise (DSE).\n\nOut of the box, DSBulk provides the ability to:\n \n1. Load (import) large amounts of data into the database efficiently and reliably; \n2. Unload (export) large amounts of data from the database efficiently and reliably; \n3. Count elements in a database table: how many rows in total, how many rows per replica and per \n   token range, and how many rows in the top N largest partitions.\n\nCurrently, CSV and Json formats are supported for both loading and unloading data.  \n\n## Installation\n\nDSBulk can be downloaded from several locations:\n\n* From [DataStax Downloads](https://downloads.datastax.com/#bulk-loader).\n  * Available formats: zip, tar.gz.\n* From [GitHub](https://github.com/datastax/dsbulk/releases).\n  * Available formats: zip, tar.gz and executable jar.\n* From Maven Central: download the artifact `dsbulk-distribution`, for example from\n  [here](https://repo.maven.apache.org/maven2/com/datastax/oss/dsbulk-distribution/).\n  * Available formats: zip, tar.gz and executable jar.\n\nPlease note: only the zip and tar.gz formats are considered production-ready. The executable jar is\nprovided as a convenience for users that want to try DSBulk, but it should not be deployed in\nproduction environments.\n\nTo install DSBulk, simply unpack the zip or tar.gz archives.\n\nThe executable jar can be executed with a command like `java -jar dsbulk-distribution.jar\n[subcommand] [options]`. See below for command line options.\n\n## Documentation\n\nThe most up-to-date documentation is available [online][onlineDocs]. \n\nWe also recommend reading the series of blog posts made by \n[Brian Hess](https://github.com/brianmhess); they target a somewhat older version of DSBulk, but\nmost of the contents are still valid and very useful:\n\n1. [DataStax Bulk Loader Pt. 1 — Introduction and Loading]\n2. [DataStax Bulk Loader Pt. 2 — More Loading]\n3. [DataStax Bulk Loader Pt. 3 — Common Settings]\n4. [DataStax Bulk Loader Pt. 4 — Unloading]\n5. [DataStax Bulk Loader Pt. 5 — Counting]\n5. [DataStax Bulk Loader: Examples for Loading From Other Locations]\n\nDevelopers and contributors: please read our [Contribution Guidelines](./CONTRIBUTING.md).\n\n## Basic Usage\n\nLaunch the tool with the appropriate script in the bin directory of your distribution. The help text \nof the tool provides summaries of all supported settings.\n\nThe `dsbulk` command takes a subcommand argument followed by options:\n\n```\n# Load data\ndsbulk load \u003coptions\u003e\n\n# Unload data\ndsbulk unload \u003coptions\u003e\n\n# Count rows\ndsbulk count \u003coptions\u003e\n``` \n\n### Long options\n\nAny DSBulk or Java Driver setting can be entered on the command line as a long-option argument of\nthe following general form:\n\n    --full.path.of.setting \"some-value\"\n    \nDSBulk settings always start with `dsbulk`; for convenience, this prefix can be omitted in a long\noption argument, so the following two options are equivalent and both map to DSBulk's \n`dsbulk.batch.mode` setting:\n\n    --dsbulk.batch.mode PARTITION_KEY\n    --batch.mode PARTITION_KEY\n    \nJava Driver settings always start with `datastax-java-driver`; for convenience, this prefix can be\nshortened to `driver` in a long option argument, so the following two options are equivalent and \nboth map to the driver's `datastax-java-driver.basic.cloud.secure-connect-bundle` setting:\n\n    --datastax-java-driver.basic.cloud.secure-connect-bundle /path/to/bundle\n    --driver.basic.cloud.secure-connect-bundle /path/to/bundle\n    \nMost settings have default values, or values that can be inferred from the input data. However, \nsometimes the default value is not suitable for you, in which case you will have to specify the\ndesired value either in the application configuration file (see below), or on the command line. \n\nFor example, the default value for `connector.csv.url` is to read from standard input or write to \nstandard output; if that does not work for you, you need to override this value and specify the \nsource path/url of the csv data to load (or path/url where to send unloaded data).\n\nSee the [Settings page](manual/settings.md) or DSBulk's \n[template configuration file](manual/application.template.conf) for details.\n\n### Short options (Shortcuts)\n\nFor convenience, many options (prefaced with `--`), have shortcut variants (prefaced with `-`).\nFor example, `--dsbulk.schema.keyspace` has an equivalent short option `-k`. \n\nConnector-specific options also have shortcut variants, but they are only available when\nthe appropriate connector is chosen. This allows multiple connectors to overlap shortcut\noptions. For example, the JSON connector has a `--connector.json.url`\nsetting with a `-url` shortcut. This overlaps with the `-url` shortcut option for the CSV \nconnector, that actually maps to `--connector.csv.url`. But in a given invocation of `dsbulk`, \nonly the appropriate shortcut will be active.  \n\nRun the tool with `--help` and specify the connector to see its short options:\n\n```\ndsbulk -c csv --help\n```\n\n## Configuration Files vs Command Line Options\n\nAll DSBulk options can be passed as command line arguments, or in a configuration file.\n\nUsing one or more configuration files is sometimes easier than passing all configuration \noptions via the command line. \n\nBy default, the configuration files are located under DSBulk's `conf` directory; the main\nconfiguration file is named `application.conf`. This location can be modified via the `-f` \nswitch. See examples below.\n\nDSBulk ships with a default, empty `application.conf` file that users can customize to their \nneeds; it also has a [template configuration file](manual/application.template.conf) that can \nserve as a starting point for further customization.\n\nConfiguration files are also required to be compliant with the [HOCON] syntax. This syntax\nis very flexible and allows sections to be grouped together in blocks, e.g.:\n\n```hocon\ndsbulk {\n  connector {\n    name = \"csv\"\n    csv {\n      url = \"C:\\\\Users\\\\My Folder\"\n      delimiter = \"\\t\"\n    }\n  }\n}\n```\n\nThe above is equivalent to the following snippet using dotted notation instead of blocks:\n\n```hocon\ndsbulk.connector.name = \"csv\"\ndsbulk.connector.csv.url = \"C:\\\\Users\\\\My Folder\"\ndsbulk.connector.csv.delimiter = \"\\t\"\n```\n\nYou can split your configuration in more than one file using file inclusions; see the HOCON\ndocumentation for details. The default configuration file includes another file called \n`driver.conf`, also located in the `conf` directory. This file should be used to configure \nthe Java Driver for DSBulk. This file is empty as well; users can customize it to their needs. \nA [driver template configuration file](manual/driver.template.conf) can serve as a starting \npoint for further customization.\n\n**Important caveats:**\n\n1. In configuration files, it is not possible to omit the prefix `dsbulk`. \n   For example, to select the connector to use in a configuration file,\n   use `dsbulk.connector.name = csv`, as in the example above; on the command line, \n   however, you can use `--dsbulk.connector.name csv` or `--connector.name csv` to achieve \n   the same effect, as stated above.\n2. In configuration files, it is not possible to abbreviate the prefix `datastax-java-driver`\n   to `driver`. For example, to select the consistency level to use in a configuration file,\n   use `datastax-java-driver.basic.request.consistency = QUORUM` in a configuration file; on \n   the command line, however, you can use both\n   `--datastax-java-driver.basic.request.consistency = QUORUM` or \n   `--driver.basic.request.consistency = QUORUM` to achieve the same effect.\n3. Options specified through the command line _override options specified in configuration \n   files_. See examples for details.\n\n## Escaping and Quoting Command Line Arguments\n\nRegardless of whether they are supplied via the command line or in a configuration file, all option \nvalues are expected to be in valid [HOCON] syntax: control characters, the backslash character, \nand the double-quote character all need to be properly escaped.\n\nFor example, `\\t` is the escape sequence that corresponds to the tab character:\n\n```bash\ndsbulk load -delim '\\t'\n```\n\nIn general, string values containing special characters (such as a colon or a whitespace) also \nneed to be properly quoted with double-quotes, as required by the HOCON syntax:\n\n```bash\ndsbulk load -h '\"host.com:9042\"'\n```\n\nFile paths on Windows systems usually contain backslashes; `\\\\` is the escape sequence for the \nbackslash character, and since Windows paths also contain special characters, the whole path \nneeds to be double-quoted: \n\n```bash\ndsbulk load -url '\"C:\\\\Users\\\\My Folder\"'\n```\n\nHowever, when the expected type of an option is a string, it is possible to omit the\nsurrounding double-quotes, for convenience:\n\n```bash\ndsbulk load -url 'C:\\\\Users\\\\My Folder'\n```\n\nSimilarly, when an argument is a list, it is possible to omit the surrounding square\nbrackets; making the following two lines equivalent:\n\n```bash\ndsbulk load --codec.nullStrings 'NIL, NULL'\ndsbulk load --codec.nullStrings '[NIL, NULL]'\n```\n\nThe same applies for arguments of type map: it is possible to omit the surrounding\ncurly braces, making the following two lines equivalent:\n\n```bash\ndsbulk load --connector.json.deserializationFeatures '{ USE_BIG_DECIMAL_FOR_FLOATS : true }'\ndsbulk load --connector.json.deserializationFeatures 'USE_BIG_DECIMAL_FOR_FLOATS : true'\n```\n\nThis syntactic sugar is only available for command line arguments of type string, list or map; \nall other option types, as well as all options specified in a configuration file _must_ be fully \ncompliant with HOCON syntax, and it is the user's responsibility to ensure that such options are \nproperly escaped _and_ quoted.\n\nAlso, note that this syntactic sugar is not capable of quoting single elements inside a list or a \nmap; all elements in a list or a map must be individually quoted if they contain special characters.\nFor example, to specify a list with 2 contact points containing port numbers, it is necessary to \nquote each contact point individually, however the surrounding brackets, as explained above, can \nbe omitted for brevity:\n\n```bash\ndsbulk load -h '\"host1.com:9042\",\"host2.com:9042\"'\n```\n\n## Load Examples\n\n* Load table `table1` in keyspace `ks1` from CSV data read from `stdin`.\n  Use a cluster with a `localhost` contact point. Field names in the data match column names in the\n  table. Field names are obtained from a *header row* in the data; by default the tool presumes \n  a header exists in each file being loaded:\n\n  `dsbulk load -k ks1 -t table1`\n\n* Load table `table1` in keyspace `ks1` from a gzipped CSV file by unzipping it to `stdout` and piping to\n  `stdin` of the tool:\n\n  `gzcat table1.csv.gz | dsbulk load -k ks1 -t table1`\n\n* Load the file `export.csv` to table `table1` in keyspace `ks1` using the short form option for `url`\n  and the tab character as a field delimiter:\n\n  `dsbulk load -k ks1 -t table1 -url export.csv -delim '\\t'`\n\n* Specify a few hosts (initial contact points) that belong to the desired cluster and\n  load from a local file, without headers. Map field indices of the input to table columns:\n  \n  `dsbulk load -url ~/export.csv -k ks1 -t table1 -h '10.200.1.3,10.200.1.4' -header false -m '0=col1,1=col3'`\n\n* Specify port 9876 for the cluster hosts and load from an external source url:\n\n  `dsbulk load -url https://192.168.1.100/data/export.csv -k ks1 -t table1 -h '10.200.1.3,10.200.1.4' -port 9876`\n\n* Load all csv files from a directory. The files do not have header rows. Map field indices\n  of the input to table columns:\n\n  `dsbulk load -url ~/export-dir -k ks1 -t table1 -m '0=col1,1=col3' -header false`\n\n* Load a file containing three fields per row. The file has no header row. Map all fields to\n  table columns in field order. Note that field indices need not be provided.\n\n  `dsbulk load -url ~/export-dir -k ks1 -t table1 -m 'col1, col2, col3' -header false`\n\n* With default port for cluster hosts, keyspace, table, and mapping set in\n  `conf/application.conf`:\n\n  `dsbulk load -url https://192.168.1.100/data/export.csv -h '10.200.1.3,10.200.1.4'`\n\n* With default port for cluster hosts, keyspace, table, and mapping set in `dsbulk_load.conf`:\n\n  `dsbulk load -f dsbulk_load.conf -url https://192.168.1.100/data/export.csv -h '10.200.1.3,10.200.1.4'`\n\n* Load table `table1` in keyspace `ks1` from a CSV file, where double-quote characters in fields are\n  escaped with a double-quote; for example,  `\"f1\",\"value with \"\"quotes\"\" and more\"` is a line in\n  the CSV file:\n\n  `dsbulk load -url ~/export.csv -k ks1 -t table1 -escape '\\\"'`\n\n* Load table `table1` in keyspace `ks1` from an AWS S3 URL, passing the `region` and `profile` as\n  [query parameters in the URL](./url/README.md):\n\n  `dsbulk load -k ks1 -t table1 -url s3://bucket-name/key?region=us-west-1\u0026profile=bucket-profile`\n\n## Unload Examples\n\nUnloading is simply the inverse of loading and due to the symmetry, many settings are\nused in both load and unload.\n\n* Unload data to `stdout` from the `ks1.table1` table in a cluster with a `localhost` contact \n  point. Column names in the table map to field names in the data. Field names must be emitted \n  in a *header row* in the output:\n\n  `dsbulk unload -k ks1 -t table1`\n\n* Unload data to `stdout` from the `ks1.table1` and gzip the result:\n\n  `dsbulk unload -k ks1 -t table1 | gzip \u003e table1.gz`\n\n* Unload data to a local directory (which may not yet exist):\n                                          \n  `dsbulk unload -url ~/data-export -k ks1 -t table1`\n\n\n## Count Examples\n\nWhen counting rows in a table, no connector is required, and `schema.mapping` should not be present.\n\n* Count the total rows in the `ks1.table1` table in a cluster with a `localhost` contact point. \n\n  `dsbulk count -k ks1 -t table1`\n\n* Count the total number of rows per token range in the `ks1.table1` table in a cluster with a `localhost` contact point. \n\n  `dsbulk count -k ks1 -t table1 -stats ranges`\n\n* Count the total number of rows per host in the `ks1.table1` table in a cluster with a `localhost` contact point. \n\n  `dsbulk count -k ks1 -t table1 -stats hosts`\n\n* Count the total number of rows for the biggest 100 partitions in the `ks1.table1` table in a cluster with a `localhost` contact point (by default, DSBulk returns the number of rows for the 10 biggest partitions in the table). \n\n  `dsbulk count -k ks1 -t table1 -stats partitions -partitions 100`\n\n* Count the total number of rows, the total number of rows per token range, the total number of rows per hosts in the `ks1.table1`, and the total number of rows for the biggest 10 partitions table in a cluster with a `localhost` contact point. \n\n  `dsbulk count -k ks1 -t table1 -stats global,ranges,hosts,partitions`\n\n\n## Command-line Help\n\nAvailable settings along with defaults are documented [here](manual/settings.md), they are also\ndocumented in DSBulk's [template configuration file](manual/application.template.conf) and in\nthe driver [template configuration file](manual/driver.template.conf). This information is also\navailable on the command-line via the `help` subcommand.\n\n* Get help for common options and a list of sections from which more help is available:\n\n  `dsbulk help`\n  \n* Get help for all `connector.csv` options:\n\n  `dsbulk help connector.csv`\n  \n[onlineDocs]:https://docs.datastax.com/en/dsbulk/doc/\n[HOCON]:https://github.com/lightbend/config/blob/master/HOCON.md\n[DataStax Bulk Loader Pt. 1 — Introduction and Loading]: https://www.datastax.com/blog/2019/03/datastax-bulk-loader-introduction-and-loading\n[DataStax Bulk Loader Pt. 2 — More Loading]: https://www.datastax.com/blog/2019/04/datastax-bulk-loader-more-loading\n[DataStax Bulk Loader Pt. 3 — Common Settings]: https://www.datastax.com/blog/2019/04/datastax-bulk-loader-common-settings\n[DataStax Bulk Loader Pt. 4 — Unloading]: https://www.datastax.com/blog/2019/06/datastax-bulk-loader-unloading\n[DataStax Bulk Loader Pt. 5 — Counting]: https://www.datastax.com/blog/2019/07/datastax-bulk-loader-counting\n[DataStax Bulk Loader: Examples for Loading From Other Locations]: https://www.datastax.com/blog/2019/12/datastax-bulk-loader-examples-loading-other-locations\n \n## License\nCopyright DataStax, Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatastax%2Fdsbulk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatastax%2Fdsbulk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatastax%2Fdsbulk/lists"}