{"id":14065823,"url":"https://github.com/Azure/batch-shipyard","last_synced_at":"2025-07-29T21:33:44.347Z","repository":{"id":10679095,"uuid":"66606388","full_name":"Azure/batch-shipyard","owner":"Azure","description":"Simplify HPC and Batch workloads on Azure","archived":true,"fork":false,"pushed_at":"2023-03-20T21:31:25.000Z","size":6444,"stargazers_count":277,"open_issues_count":43,"forks_count":121,"subscribers_count":48,"default_branch":"master","last_synced_at":"2024-10-01T16:11:50.419Z","etag":null,"topics":["azure","azure-batch","azure-functions","batch-processing","containers","docker","glusterfs","gpu","hpc","infiniband","mpi","nfs","rdma","serverless","singularity","slurm","windows-containers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Azure.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2016-08-26T01:33:23.000Z","updated_at":"2024-09-04T23:05:02.000Z","dependencies_parsed_at":"2024-02-19T18:20:29.859Z","dependency_job_id":"f651a38c-1f51-422c-b77c-a28a63e2ea52","html_url":"https://github.com/Azure/batch-shipyard","commit_stats":null,"previous_names":[],"tags_count":71,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Fbatch-shipyard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Fbatch-shipyard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Fbatch-shipyard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Fbatch-shipyard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Azure","download_url":"https://codeload.github.com/Azure/batch-shipyard/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228052622,"owners_count":17862103,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","azure-batch","azure-functions","batch-processing","containers","docker","glusterfs","gpu","hpc","infiniband","mpi","nfs","rdma","serverless","singularity","slurm","windows-containers"],"created_at":"2024-08-13T07:04:45.071Z","updated_at":"2024-12-04T05:31:01.278Z","avatar_url":"https://github.com/Azure.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"[![Build Status](https://azurebatch.visualstudio.com/batch-shipyard/_apis/build/status/batch-shipyard-CI)](https://azurebatch.visualstudio.com/batch-shipyard/_build/latest?definitionId=11)\n[![Build Status](https://travis-ci.org/Azure/batch-shipyard.svg?branch=master)](https://travis-ci.org/Azure/batch-shipyard)\n[![Build status](https://ci.appveyor.com/api/projects/status/3a0j0gww57o6nkpw/branch/master?svg=true)](https://ci.appveyor.com/project/alfpark/batch-shipyard)\n\n# Batch Shipyard\n\n## PROJECT STATUS\n**This toolkit is no longer actively maintained.** The `develop` branch has\nproposed fixes for outstanding issues, but they will not be merged back to\n`master`. Please see the main [Azure Batch](https://github.com/Azure/Batch)\nGitHub repository for more information about Azure Batch.\n\n\u003cimg src=\"https://azurebatchshipyard.blob.core.windows.net/github/README-dash.gif\" alt=\"dashboard\" width=\"1024\" /\u003e\n\n[Batch Shipyard](https://github.com/Azure/batch-shipyard) is a tool to help\nprovision, execute, and monitor container-based batch processing and HPC\nworkloads on\n[Azure Batch](https://azure.microsoft.com/services/batch/). Batch Shipyard\nsupports both [Docker](https://www.docker.com) and\n[Singularity](https://www.sylabs.io) containers. No experience with the\n[Azure Batch SDK](https://github.com/Azure/azure-batch-samples) is needed; run\nyour containers with easy-to-understand configuration files. All Azure\nregions are supported, including non-public Azure regions.\n\nAdditionally, Batch Shipyard provides the ability to provision and manage\nentire [standalone remote file systems (storage clusters)](https://batch-shipyard.readthedocs.io/en/latest/65-batch-shipyard-remote-fs/)\nin Azure, independent of any integrated Azure Batch functionality.\n\n## Major Features\n### Container Runtime and Image Management\n* Support for multiple container runtimes including\n[Docker](https://docker.com), [Singularity](https://www.sylabs.io), and\n[Kata Containers](https://katacontainers.io/) tuned for Azure Batch\ncompute nodes\n* Automated deployment of container images required for tasks to compute nodes\n* Support for container registries including\n[Azure Container Registry](https://azure.microsoft.com/services/container-registry/)\nfor both Docker and Singularity images (ORAS), other Internet-accessible\npublic and private registries, and support for\nthe [Sylabs Singularity Library](https://cloud.sylabs.io/library) and\n[Singularity Hub](https://singularity-hub.org/)\n* Transparent support for GPU-accelerated container applications on both\n[Docker](https://github.com/NVIDIA/nvidia-docker) and Singularity\non [Azure N-Series VM instances](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-gpu)\n* Transparent assist for running Docker and Singularity containers utilizing\nInfiniband/RDMA on HPC Azure VM instances including\n[A-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-hpc),\n[H-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-hpc),\n[Hb/Hc-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-hpc),\nand [N-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-gpu)\n* Integrated support for\n[Singularity Encrypted Containers](https://batch-shipyard.readthedocs.io/en/latest/50-batch-shipyard-encrypted-containers/)\n\n### Data Management and Shared File Systems\n* Comprehensive [data movement](https://batch-shipyard.readthedocs.io/en/latest/70-batch-shipyard-data-movement/)\nsupport: move data easily between locally accessible storage systems, remote\nfilesystems, Azure Blob or File Storage, and compute nodes\n* [Standalone Remote Filesystem Provisioning](https://batch-shipyard.readthedocs.io/en/latest/65-batch-shipyard-remote-fs/)\nwith integration to auto-link these filesystems to compute nodes with\nsupport for [NFS](https://en.wikipedia.org/wiki/Network_File_System) and\n[GlusterFS](https://www.gluster.org/) distributed network file system\n* Automatic shared data volume support for linking to\n[Remote Filesystems](https://batch-shipyard.readthedocs.io/en/latest/65-batch-shipyard-remote-fs/),\n[Azure File](https://azure.microsoft.com/services/storage/files/)\nvia SMB, [Azure Blob](https://azure.microsoft.com/services/storage/blobs/)\nvia [blobfuse](https://github.com/Azure/azure-storage-fuse),\n[GlusterFS](https://www.gluster.org/) provisioned directly on compute nodes,\nand custom Linux mount support (fstab)\n* Support for automated on-demand, per-job distributed scratch space\nprovisioning via [BeeGFS BeeOND](https://www.beegfs.io/wiki/BeeOND)\n\n### Monitoring\n* Automated, integrated\n[resource monitoring](https://batch-shipyard.readthedocs.io/en/latest/66-batch-shipyard-resource-monitoring/)\nwith [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/)\nfor Batch pools and RemoteFS storage clusters\n* Support for [Batch Insights](https://github.com/Azure/batch-insights)\n\n### Open Source Scheduler Integration\n* Support for [elastic cloud bursting](https://batch-shipyard.readthedocs.io/en/latest/69-batch-shipyard-slurm/)\non [Slurm](https://slurm.schedmd.com/) to Batch pools with automated\nRemoteFS shared file system linking\n\n### Azure Ecosystem Integration\n* Support for\n[serverless execution](https://batch-shipyard.readthedocs.io/en/latest/60-batch-shipyard-site-extension/)\nbinding with Azure Functions\n* Support for credential management through\n[Azure KeyVault](https://azure.microsoft.com/services/key-vault/)\n\n### Azure Batch Integration and Enhancements\n* [Federation](https://batch-shipyard.readthedocs.io/en/latest/68-batch-shipyard-federation/)\nsupport: enables unified, constraint-based scheduling to collections of\nheterogeneous pools, including across multiple Batch accounts and Azure\nregions\n* Support for simple, scenario-based [pool autoscale](https://batch-shipyard.readthedocs.io/en/latest/30-batch-shipyard-autoscale/)\nand autopool to dynamically scale and control computing resources on-demand\n* Support for [Task Factories](https://batch-shipyard.readthedocs.io/en/latest/35-batch-shipyard-task-factory-merge-task/)\nwith the ability to generate tasks based on parametric (parameter) sweeps,\nrandomized input, file enumeration, replication, and custom Python code-based\ngenerators\n* Support for\n[multi-instance tasks](https://batch-shipyard.readthedocs.io/en/latest/80-batch-shipyard-multi-instance-tasks/)\nto accommodate MPI and multi-node cluster applications packaged as Docker or\nSingularity containers on compute pools with automatic job completion and\ntask termination\n* Seamless, direct high-level configuration support for popular MPI runtimes\nincluding OpenMPI, MPICH, MVAPICH, and Intel MPI with automatic configuration\nfor Infiniband, including SR-IOV RDMA VM sizes\n* Seamless integration with Azure Batch job, task and file concepts along with\nfull pass-through of the\n[Azure Batch API](https://azure.microsoft.com/documentation/articles/batch-api-basics/)\nto containers executed on compute nodes\n* Support for [Azure Batch task dependencies](https://azure.microsoft.com/documentation/articles/batch-task-dependencies/)\nallowing complex processing pipelines and DAGs\n* Support for merge or final task specification that automatically depends\non all other tasks within the job\n* Support for job schedules and recurrences for automatic execution of\ntasks at set intervals\n* Support for live job and job schedule migration between pools\n* Support for [Low Priority Compute Nodes](https://docs.microsoft.com/azure/batch/batch-low-pri-vms)\n* Support for deploying Batch compute nodes into a specified\n[Virtual Network](https://batch-shipyard.readthedocs.io/en/latest/64-batch-shipyard-byovnet/)\nand pre-defined public IP addresses\n* Automatic setup of SSH or RDP users to all nodes in the compute pool and\noptional creation of SSH tunneling scripts to Docker Hosts on compute nodes\n* Support for [custom host images](https://batch-shipyard.readthedocs.io/en/latest/63-batch-shipyard-custom-images/)\nincluding Shared Image Gallery\n* Support for [Windows Containers](https://docs.microsoft.com/virtualization/windowscontainers/about/)\non compliant Windows compute node pools with the ability to activate\n[Azure Hybrid Use Benefit](https://azure.microsoft.com/pricing/hybrid-benefit/)\nif applicable\n\n## Installation\n### Local Installation\nPlease see [the installation guide](https://batch-shipyard.readthedocs.io/en/latest/01-batch-shipyard-installation/)\nfor more information regarding the various local installation options and\nrequirements.\n\n### Azure Cloud Shell\nBatch Shipyard is integrated directly into\n[Azure Cloud Shell](https://docs.microsoft.com/azure/cloud-shell/overview)\nand you can execute any Batch Shipyard workload using your web browser or\nthe Microsoft Azure [Android](https://play.google.com/store/apps/details?id=com.microsoft.azure\u0026hl=en)\nand [iOS](https://itunes.apple.com/us/app/microsoft-azure/id1219013620?mt=8)\napp.\n\nSimply request a Cloud Shell session and type `shipyard` to invoke the CLI;\nno installation is required. Try Batch Shipyard now\n[in your browser](https://shell.azure.com).\n\n## Documentation and Recipes\nPlease refer to the\n[Batch Shipyard Documentation on Read the Docs](https://batch-shipyard.readthedocs.io/).\n\nVisit the\n[Batch Shipyard Recipes](https://github.com/Azure/batch-shipyard/blob/master/recipes)\nsection for various sample container workloads using Azure Batch and Batch\nShipyard.\n\n## Batch Shipyard Compute Node Host OS Support\nBatch Shipyard is currently compatible with popular Azure Batch supported\n[Marketplace Linux VMs](https://docs.microsoft.com/azure/virtual-machines/linux/endorsed-distros),\n[compliant Linux custom images](https://batch-shipyard.readthedocs.io/en/latest/63-batch-shipyard-custom-images/),\nand native Azure Batch\n[Windows Server with Containers](https://azuremarketplace.microsoft.com/marketplace/apps/Microsoft.WindowsServer?tab=Overview)\nVMs. Please see the\n[platform image support](https://batch-shipyard.readthedocs.io/en/latest/25-batch-shipyard-platform-image-support/)\ndocumentation for more information specific to Batch Shipyard support of\ncompute node host operating systems.\n\n## Change Log\nPlease see the\n[Change Log](https://batch-shipyard.readthedocs.io/en/latest/CHANGELOG/)\nfor project history.\n\n* * *\nPlease see this project's [Code of Conduct](CODE_OF_CONDUCT.md) and\n[Contributing](CONTRIBUTING.md) guidelines.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAzure%2Fbatch-shipyard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAzure%2Fbatch-shipyard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAzure%2Fbatch-shipyard/lists"}