{"id":14988013,"url":"https://github.com/apache/doris-streamloader","last_synced_at":"2025-10-19T12:30:33.346Z","repository":{"id":218882298,"uuid":"744436069","full_name":"apache/doris-streamloader","owner":"apache","description":"Stream Loader for Apache Doris","archived":false,"fork":false,"pushed_at":"2024-10-09T09:34:21.000Z","size":68,"stargazers_count":18,"open_issues_count":6,"forks_count":16,"subscribers_count":41,"default_branch":"master","last_synced_at":"2025-01-30T10:23:51.522Z","etag":null,"topics":["bigquery","database","dbt","delta-lake","elt","etl","hadoop","hive","hudi","iceberg","lakehouse","olap","query-engine","real-time","redshift","snowflake","spark","sql"],"latest_commit_sha":null,"homepage":"https://doris.apache.org","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-17T09:47:53.000Z","updated_at":"2025-01-04T06:16:09.000Z","dependencies_parsed_at":"2024-05-09T03:44:15.554Z","dependency_job_id":"75f66b64-95ae-4a48-919c-7d50012a4b6e","html_url":"https://github.com/apache/doris-streamloader","commit_stats":{"total_commits":18,"total_committers":5,"mean_commits":3.6,"dds":0.5,"last_synced_commit":"5cd581b1409303c6a35989dd6a5a44a9b11412d8"},"previous_names":["apache/doris-streamloader"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fdoris-streamloader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fdoris-streamloader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fdoris-streamloader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fdoris-streamloader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/doris-streamloader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237125530,"owners_count":19259300,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","database","dbt","delta-lake","elt","etl","hadoop","hive","hudi","iceberg","lakehouse","olap","query-engine","real-time","redshift","snowflake","spark","sql"],"created_at":"2024-09-24T14:15:57.334Z","updated_at":"2025-10-19T12:30:28.034Z","avatar_url":"https://github.com/apache.png","language":"Go","readme":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n--\u003e\n\n# Apache Doris Streamloader\n\nA robust, high-performance and user-friendly alternative to the traditional curl-based Stream Load.\n\n\n\n## Key Features\n\n- **Parallel Loading**: Split data files automatically and perform parallel loading\n- **Support for Multiple Files and Directories**: Support multiple files and directories load with one shot\n- **Path Traversal Support**: Support path traversal when the source files are in directories\n- **Resilience and Continuity**: Resume loading from previous failures and cancellations\n- **Automatic Retry Mechanism**: Retry automatically when failure\n- **Comprehensive and Concise Input Parameters**\n\n\n\n## Usage\n\n```shell\ndoris-streamloader --source_file={FILE_LIST} --url={FE_OR_BE_SERVER_URL}:{PORT} --header={STREAMLOAD_HEADER} --db={TARGET_DATABASE} --table={TARGET_TABLE}\n```\n\n- `FILE_LIST`: directory or file list, support \\* wildcard\n- `FE_OR_BE_SERVER_URL` \u0026 `PORT`: Doris FE or BE hostname or IP and HTTP port\n- `STREAMLOAD_HEADER`: supports all headers as `curl` Stream Load does，multiple headers are separated by '?'\n- `TARGET_DATABASE` \u0026 `TARGET_TABLE`: indicate the target database and table where the data will be loaded\n\ne.g.:\n\n```shell\ndoris-streamloader --source_file=\"data.csv\" --url=\"http://localhost:8330\" --header=\"column_separator:|?columns:col1,col2\" --db=\"testdb\" --table=\"testtbl\"\n```\n\nFor additional details and options, refer to our comprehensive docs below.\n\n\n\n## Docs\n\n[User Guide](https://doris.apache.org/docs/ecosystem/doris-streamloader)\n\n[中文使用文档](https://doris.apache.org/zh-CN/docs/ecosystem/doris-streamloader)\n\n\n\n## Build\n\nTo build Streamloader, ensure you have golang installed (version \u003e= 1.19.9). For example, on CentOS:\n\n```\nyum install golang\n```\n\nThen, navigate to the doris-streamloader directory and execute:\n\n```\ncd doris-streamloader \u0026\u0026 sh build.sh\n```\n\n\n\n## License\n\n[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fdoris-streamloader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fdoris-streamloader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fdoris-streamloader/lists"}