{"id":28722323,"url":"https://github.com/docling-project/docling-jobkit","last_synced_at":"2025-07-29T21:42:30.998Z","repository":{"id":282370603,"uuid":"934776823","full_name":"docling-project/docling-jobkit","owner":"docling-project","description":null,"archived":false,"fork":false,"pushed_at":"2025-07-24T08:41:39.000Z","size":1637,"stargazers_count":6,"open_issues_count":6,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-24T12:10:45.015Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/docling-project.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-18T11:42:12.000Z","updated_at":"2025-07-24T08:41:42.000Z","dependencies_parsed_at":"2025-03-14T08:38:12.203Z","dependency_job_id":"1acb101f-a18e-4fda-9abc-cb429889549c","html_url":"https://github.com/docling-project/docling-jobkit","commit_stats":null,"previous_names":["docling-project/docling-jobkit"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/docling-project/docling-jobkit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/docling-project%2Fdocling-jobkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/docling-project%2Fdocling-jobkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/docling-project%2Fdocling-jobkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/docling-project%2Fdocling-jobkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/docling-project","download_url":"https://codeload.github.com/docling-project/docling-jobkit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/docling-project%2Fdocling-jobkit/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267764424,"owners_count":24140850,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-29T02:00:12.549Z","response_time":2574,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-15T08:08:57.861Z","updated_at":"2025-07-29T21:42:30.984Z","avatar_url":"https://github.com/docling-project.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Docling Jobkit\n\nRunning a distributed job processing documents with Docling.\n\n\n## How to use it\n\n## Kubeflow pipeline with Docling Jobkit\n\n### Using Kubeflow pipeline web dashboard UI\n\n1. From the main page, open \"Pipelines\" section on the left\n2. Press on \"Upload pipeline\" button at top-right\n3. Give pipeline a name and in \"Upload a file\" menu point to location of `docling-jobkit/docling_jobkit/kfp_pipeline/docling-s3in-s3out.yaml` file\n4. Now you can press \"Create run\" button at the top-right to create an instance of the pipeline\n5. Customize required inputs according to provided examples and press \"Start\" to start pipeline run\n\n### Using OpenshiftAI web dashboard UI\n1. From the main page of Red Hat Openshift AI open \"Data Science Pipelines -\u003e Pipelines\" section on the left side\n2. Switch \"Project\" to namespace where you plan to run pipelines\n3. Press on \"Import Pipeline\", provide a name and upload the `docling-jobkit/docling_jobkit/kfp_pipeline/docling-s3in-s3out.yaml` file\n4. From the selected/created pipeline interface, you can start new run by pressing \"Actions -\u003e Create Run\"\n5. Customize required inputs according to provided examples and press \"Start\" to start pipeline run\n \n### Customizing pipeline to specifics of your infrastructure\n\nSome customizations, such as paralelism level, node selector or tollerations, require changing source script and compiling new yaml manifest.\nSource script is located at `docling-jobkit/docling_jobkit/kfp_pipeline/docling-s3in-s3out.py`.\n\nIf you use web UI to run pipelines, then python script need to be compiled into yaml and new version of yaml uploaded to pipeline.\nFor example, you can use poetry to handle python environment and run following command:\n``` sh\nuv run python semantic-ingest-batches.py\n```\nThe yaml file will be generated in the local folder from where you execute command.\nNow in the web UI, you can open existing pipeline and upload new version of the script using \"Upload version\" at top-right.\n\nBy defaul, paralelism is set to 20 instances, this can be change in the source `docling-jobkit/docling_jobkit/kfp_pipeline/docling-s3in-s3out.py` script, look for this line `with dsl.ParallelFor(batches.outputs[\"batch_indices\"], parallelism=20) as subbatch:`.\n\nBy default, the resources requests/limits for the document convertion component are set to following:\n``` py\nconverter.set_memory_request(\"1G\")\nconverter.set_memory_limit(\"7G\")\nconverter.set_cpu_request(\"200m\")\nconverter.set_cpu_limit(\"1\")\n```\n\nBy default, the resource request/limit are not set for the nodes with GPU, you can uncomment following lines in the `inputs_s3in_s3out` pipeline function to enable it:\n``` py\nconverter.set_accelerator_type(\"nvidia.com/gpu\")\nconverter.set_accelerator_limit(\"1\")\n```\n\nThe node selector and tollerations can be enabled with following commands, customize actual values to your infrastructure:\n``` py\nfrom kfp import kubernetes\n\nkubernetes.add_node_selector(\n  task=converter,\n  label_key=\"nvidia.com/gpu.product\",\n  label_value=\"NVIDIA-A10\",\n)\n\nkubernetes.add_toleration(\n  task=converter,\n  key=\"gpu_compute\",\n  operator=\"Equal\",\n  value=\"true\",\n  effect=\"NoSchedule\",\n)\n```\n\n### Running pipeline programatically\n\nAt the end of the script file you can find an example code for submitting pipeline run programatically.\nYou can provide your custom values as environment variables in an `.env` file and bind it during execution:\n``` sh\nuv run --env-file .env python docling-s3in-s3out.py\n```\n\n\n## Ray runtime with Docling Jobkit\n\n\nMake sure your Ray cluster has `docling-jobkit` installed, then submit the job.\n\n```sh\nray job submit --no-wait --working-dir . --runtime-env runtime_env.yml -- docling-ray-job\n```\n\n### Custom runtime environment\n\n\n1. Create a file `runtime_env.yml`:\n\n    ```yaml\n    # Expected environment if clean ray image is used. Take into account that ray worker can timeout before it finishes installing modules.\n    pip:\n    - docling-jobkit\n    ```\n\n\n2. Submit the job using the custom runtime env: \n\n    ```sh\n    ray job submit --no-wait --runtime-env runtime_env.yml -- docling-ray-job\n    ```\n\nMore examples and customization are provided in [docs/ray-job/](docs/ray-job/README.md).\n\n\n### Custom image with all dependencies\n\nComing soon. Initial instruction from [OpenShift AI docs](https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/2-latest/html/working_with_distributed_workloads/managing-custom-training-images_distributed-workloads#creating-a-custom-training-image_distributed-workloads).\n\n\n## Get help and support\n\nPlease feel free to connect with us using the [discussion section](https://github.com/docling-project/docling/discussions) of the main [Docling repository](https://github.com/docling-project/docling).\n\n## Contributing\n\nPlease read [Contributing to Docling Serve](https://github.com/docling-project/docling-jobkit/blob/main/CONTRIBUTING.md) for details.\n\n## References\n\nIf you use Docling in your projects, please consider citing the following:\n\n```bib\n@techreport{Docling,\n  author = {Deep Search Team},\n  month = {1},\n  title = {Docling: An Efficient Open-Source Toolkit for AI-driven Document Conversion},\n  url = {https://arxiv.org/abs/2501.17887},\n  eprint = {2501.17887},\n  doi = {10.48550/arXiv.2501.17887},\n  version = {2.0.0},\n  year = {2025}\n}\n```\n\n## License\n\nThe Docling Serve codebase is under MIT license.\n\n## LF AI \u0026 Data\n\nDocling is hosted as a project in the [LF AI \u0026 Data Foundation](https://lfaidata.foundation/projects/).\n\n### IBM ❤️ Open Source AI\n\nThe project was started by the AI for Knowledge team at IBM Research Zurich.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdocling-project%2Fdocling-jobkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdocling-project%2Fdocling-jobkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdocling-project%2Fdocling-jobkit/lists"}