{"id":48101196,"url":"https://github.com/distributed-system-analysis/py-es-bulk","last_synced_at":"2026-04-04T15:43:29.021Z","repository":{"id":56226100,"uuid":"135532750","full_name":"distributed-system-analysis/py-es-bulk","owner":"distributed-system-analysis","description":"A simple wrapper around elasticsearch-py client streaming_bulk() API with robust error handling","archived":false,"fork":false,"pushed_at":"2021-03-16T19:34:23.000Z","size":48,"stargazers_count":3,"open_issues_count":3,"forks_count":4,"subscribers_count":21,"default_branch":"master","last_synced_at":"2024-09-21T09:11:04.159Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/distributed-system-analysis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-31T04:46:20.000Z","updated_at":"2020-11-19T14:35:16.000Z","dependencies_parsed_at":"2022-08-15T15:00:50.334Z","dependency_job_id":null,"html_url":"https://github.com/distributed-system-analysis/py-es-bulk","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/distributed-system-analysis/py-es-bulk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/distributed-system-analysis%2Fpy-es-bulk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/distributed-system-analysis%2Fpy-es-bulk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/distributed-system-analysis%2Fpy-es-bulk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/distributed-system-analysis%2Fpy-es-bulk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/distributed-system-analysis","download_url":"https://codeload.github.com/distributed-system-analysis/py-es-bulk/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/distributed-system-analysis%2Fpy-es-bulk/sbom","scorecard":{"id":344927,"data":{"date":"2025-08-11","repo":{"name":"github.com/distributed-system-analysis/py-es-bulk","commit":"d3ab7ff475238aad9a1e5da69c1ad255ff8e29d4"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.9,"checks":[{"name":"Code-Review","score":5,"reason":"Found 8/15 approved changesets -- score normalized to 5","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/git-actions.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/git-actions.yml:13: update your workflow using https://app.stepsecurity.io/secureworkflow/distributed-system-analysis/py-es-bulk/git-actions.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/git-actions.yml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/distributed-system-analysis/py-es-bulk/git-actions.yml/master?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/git-actions.yml:20","Warn: pipCommand not pinned by hash: .github/workflows/git-actions.yml:21","Warn: pipCommand not pinned by hash: .github/workflows/git-actions.yml:22","Info:   0 out of   2 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   3 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: GNU General Public License v3.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 23 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-18T06:49:38.582Z","repository_id":56226100,"created_at":"2025-08-18T06:49:38.582Z","updated_at":"2025-08-18T06:49:38.582Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31403960,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T10:20:44.708Z","status":"ssl_error","status_checked_at":"2026-04-04T10:20:06.846Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-04T15:43:28.367Z","updated_at":"2026-04-04T15:43:29.010Z","avatar_url":"https://github.com/distributed-system-analysis.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# py-es-bulk\nA simple wrapper around the Python `elasticsearch` client `put_template()`, `streaming_bulk()`, and `parallel_bulk()` helper APIs with robust error handling.\n\nThis library is designed to work across various versions of the\nelasticsearch Python module and of the Elasticsearch server, by\ndynamically identifying the module used to create the `Elasticsearch` object.\n\nThese names are available for import:\n\n* `put_template`\n\n    Push a document template to the server using a specified\n    Elasticsearch object. This module will determine whether\n    a template document of the same name and version already\n    exists, and PUT the new template if not.\n\n    Args:\n    - `es`: An instance of the `Elasticsearch` class.\n    - `name`: The name of the template.\n    - `mapping_name`: The name of the mapping used in the template.\n    - `body`: The payload body of the template.\n\n    Returns: A tuple (start_time, end_time, retry_count, error_keys)\n\n* `streaming_bulk`\n\n    Push multiple source documents to Elasticsearch indices,\n    using proper error handling and retry logic.\n\n    Args:\n    - 'es': An instance of the `Elasticsearch` class.\n    - `actions`: An iterable of Elasticsearch action records (passed directly to Elasticsearch).\n    - `errorsfp`: A file pointer where HTTP 400 errors are logged.\n    - `logger`: A `Logger` object where messages can be logged.\n\n    Returns: A tuple (start_time, end_time, successfully_indexed, duplicate, failed, retry_count).\n\n\n* `parallel_bulk`\n\n    Push multiple source documents to Elasticsearch indices\n    in parallel across multiple threads, using proper error\n    handling and retry logic.\n\n    Args:\n    - `es`: An instance of the `Elasticsearch` class.\n    - `actions`: An iterable of Elasticsearch action records\n    (passed directly to Elasticsearch)\n    - `errorsfp`: A file pointer where HTTP 400 errors are logged.\n    - `logger`: A `Logger` object where messages can be logged.\n    - `chunk_size=10000000`: Number of docs sent in one chunk to Elasticsearch.\n    - `max_chunk_bytes=104857600`: The maximum size of a request.\n    - `thread_count=8`: The size of the thread pool to use.\n    - `queue_size=4`: The size of the task queue between the controller and processing threads.\n\n    Returns: A tuple (start_time, end_time, successfully_indexed, duplicate, failed, retry_count)\n\n* `TemplateException`\n\n    This exception is raised by put_template when a\n    template document does not contain the required version\n    metadata (`{\"_meta\": {\"version\": \u003cinteger\u003e}}`); or, when\n    multiple template documents are included in a single call\n    to put_template, if the versions of those documents are\n    not all identical.\n\n__Unit testing support__\n\nThe `pyesbulk` package attempts to dynamically determine the\nPython module used to produce the `Elasticsearch` object that's passed in to `pyesbulk` methods. This is necessary in order to properly resolve exception classes for the error handling and retry logic.\n\nHowever, unit tests often work with mocked objects which\nwon't have \"real\" Python package structure, and the dynamic\nmodule recognition algorithm may fail. When this happens,\n`pyesbulk` will attempt to import `elasticsearch`. If that's not correct (e.g., if you're using `elasticsearch1`\nor `elasticsearch5`), you can override the automatic search\nby including a `force_elastic_search_module` property on\nyour mocked `Elasticsearch` object.\n\nFor example,\n\n```\nclass MockElasticsearch:\n    def __init__(self):\n        self.force_elastic_search_module = \"elasticsearch5\"\n```\n\nor\n\n```\n    es = MockElasticSearch()\n    es.force_elastic_search_module = \"elasticsearch1\"\n```\n\n\nSee also https://pypi.org/project/pyesbulk/.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdistributed-system-analysis%2Fpy-es-bulk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdistributed-system-analysis%2Fpy-es-bulk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdistributed-system-analysis%2Fpy-es-bulk/lists"}