{"id":22318019,"url":"https://github.com/adampaternostro/azure-docker-shipyard","last_synced_at":"2025-08-21T08:16:11.492Z","repository":{"id":69583484,"uuid":"96435838","full_name":"AdamPaternostro/Azure-Docker-Shipyard","owner":"AdamPaternostro","description":"Demo of using Docker contatiners in Azure Batch (using Shipyard)","archived":false,"fork":false,"pushed_at":"2018-02-07T15:51:48.000Z","size":573,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-06-21T11:02:05.323Z","etag":null,"topics":["azure","azure-batch","containers","docker"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AdamPaternostro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-06T13:54:01.000Z","updated_at":"2023-08-29T11:02:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"b06cc3fb-4a06-47fa-8ef0-c1b974bed28b","html_url":"https://github.com/AdamPaternostro/Azure-Docker-Shipyard","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AdamPaternostro/Azure-Docker-Shipyard","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdamPaternostro%2FAzure-Docker-Shipyard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdamPaternostro%2FAzure-Docker-Shipyard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdamPaternostro%2FAzure-Docker-Shipyard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdamPaternostro%2FAzure-Docker-Shipyard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AdamPaternostro","download_url":"https://codeload.github.com/AdamPaternostro/Azure-Docker-Shipyard/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdamPaternostro%2FAzure-Docker-Shipyard/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271448405,"owners_count":24761441,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-21T02:00:08.990Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","azure-batch","containers","docker"],"created_at":"2024-12-03T23:12:03.634Z","updated_at":"2025-08-21T08:16:11.474Z","avatar_url":"https://github.com/AdamPaternostro.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Azure-Docker-Shipyard\nDemo of using Docker contatiners in Azure Batch (using Shipyard).  This also shows how to mount a Docker Volume using Shipyard.\n\n## Links\n- https://github.com/Azure/batch-shipyard\n- https://azure.github.io/batch-shipyard/\n\n## Create Azure Resource Group\n- Create a Resource Group (e.g. AdamShipyardDemo)\n\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/New-Resource-Group.png)\n\n## Create Azure Batch Service (create in the resource group above)\n- Create a \"Batch Service\" account (e.g. adamshipyardbatchservice)\n- When creating the \"Batch Service\" create a storage account (e.g. adamshipyardstorage)\n\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/New-Batch-Service-Account.png)\n\n## Create a Linux VM (create in the resource group above)\n- Create Ubuntu Server 16.04 LTS  (e.g. adamshipyardvm)\n```\nUsername: shipyarduser  Password: \u003c\u003cREMOVED\u003e\u003e\nHard disk: HDD\nSize: D1_V2 (does not need to be powerful)\nUse Managed Disk: Yes\nMonitoring: Disabled\nUse the defualts for Networking\n```\n\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/New-Linux-VM.png)\n\n## All Resources\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/Complete-Resource-Group.png)\n\n\n## Install Docker and Shipyard\n#### ssh to the Linux computer\n\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/ssh-To-Linux.png)\n\n#### Install Docker\n```\nsudo apt-get -y install apt-transport-https ca-certificates curl\ncurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\nsudo add-apt-repository \"deb [arch=amd64] https://download.docker.com/linux/ubuntu  $(lsb_release -cs) stable\"  \nsudo apt-get update\nsudo apt-get -y install docker-ce\n```\n\n#### Test docker\n```\nsudo docker run hello-world\n```\n\n#### For Shipyard image\n```\nsudo docker pull alfpark/batch-shipyard:cli-latest \n```\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/Install-Shipyard.png)\n\n## Create a Docker program\n- Create a directory mkdir docker\n- Create a file named Dockerfile and place this in the file\n```\nFROM alpine\nWORKDIR /app\nRUN apk add --update bash\nRUN apk add --update curl \u0026\u0026 rm -rf /var/cache/apk/*\nADD download.sh /app\nCMD [\"bash\",\"./download.sh\"]\n```\n\n- Create a file named download.sh and place this in the file\n```\n#!/bin/bash\nfor i in {1..10}\ndo\n   filename=$(date +\"%m-%d-%y-%T\").pdf\n   echo \"Starting Download: $filename\"\n   curl -o /share/$filename http://ipv4.download.thinkbroadband.com/20MB.zip\n   echo \"Downloaded: $filename\"\ndone\n```\n\n- Build the Docker image\n```\nsudo docker build -t adamshipyarddockerimage .\n```\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/Build-Dockerfile.png)\n\n\n- List the images\n```\nsudo docker images\n```\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/Docker-Images.png)\n\n\n- Run the image locally\n```\nsudo docker run -v ~/mnt/share:/share adamshipyarddockerimage \n```\n\n- Create a respository on Dockerhub (e.g. adamshipyardrepository)\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/Create-Repository.png)\n\n- Upload image to repository\n```\nsudo docker login\nsudo docker tag adamshipyarddockerimage adampaternostro/adamshipyarddockerimage:latest\nsudo docker push adampaternostro/adamshipyarddockerimage:latest\n```\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/Push-Image-to-Repo.png)\n\n\n## Create Shipyard files\n- Go back to your home directory (cd ..)\n- Make a directory (e.g. config)\n- cd config\n\n- Create a file named \"config.yaml\"\n  - Description: This contains a reference to a storage account used by Shipyard for its own internal purposes.  It also has a reference to our Docker image as well a Data Volume so we know where to download our files.\n  - For full schema: https://github.com/Azure/batch-shipyard/blob/master/docs/12-batch-shipyard-configuration-global.md\n```\nbatch_shipyard:\n  storage_account_settings: mystorageaccount\nglobal_resources:\n  docker_images:\n  - adampaternostro/adamshipyarddockerimage:latest\n  docker_volumes:\n    data_volumes:\n      ephemeraldisk:\n        host_path: \"/mnt/docker-tmp\"\n        container_path: \"/share\"\n```\n\n- Create a file named \"credentials.yaml\"\n  - For full schema: https://github.com/Azure/batch-shipyard/blob/master/docs/11-batch-shipyard-configuration-credentials.md\n  - Change the Batch: account_key and account_service_url\n  - Change the mystorageaccount: account and account key (NOTE: keep the name mystorageaccount since it is referenced in the \"config.json\")\n```\ncredentials:\n  batch:\n    account_key: \"\u003c\u003cREMOVED\u003e\u003e\"\n    account_service_url: https://adamshipyardbatchservice.eastus2.batch.azure.com\n  storage:\n    mystorageaccount:\n      account: adamshipyardstorage\n      account_key: \"\u003c\u003cREMOVED\u003e\u003e\"\n      endpoint: core.windows.net\n```\n\n- Create a file named \"jobs.ymal\"\n  - For full schema: https://github.com/Azure/batch-shipyard/blob/master/docs/14-batch-shipyard-configuration-jobs.md\n```\njob_specifications:\n- id: adamshipyardjob\n  data_volumes:\n  - ephemeraldisk\n  tasks:\n  - docker_image: adampaternostro/adamshipyarddockerimage:latest\n    remove_container_after_exit: true\n    command: bash /app/download.sh\n\n```\n\n- Create a file named \"pool.yaml\"\n  - For full schema: https://github.com/Azure/batch-shipyard/blob/master/docs/13-batch-shipyard-configuration-pool.md\n```\npool_specification:\n  id: adampool\n  vm_size: STANDARD_D1_V2\n  vm_count:\n    dedicated: 2\n    low_priority: 0\n  vm_configuration:\n    platform_image:\n      publisher: Canonical\n      offer: UbuntuServer\n      sku: 16.04-LTS\n  reboot_on_start_task_failed: false\n  block_until_all_global_resources_loaded: true\n\n```\n\n## Run Shipyard\n#### Create the batch pool to run our Docker container on\n```\nsudo docker run --rm -it -v /home/shipyarduser/config:/configs -e SHIPYARD_CONFIGDIR=/configs alfpark/batch-shipyard:latest-cli pool add\n```\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/Create-Pools-1.png)\n\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/Create-Pools-2.png)\n\n\n#### Run the Docker container (you can use stdout.txt or stderr.txt)\n```\nsudo docker run --rm -it -v /home/shipyarduser/config:/configs -e SHIPYARD_CONFIGDIR=/configs alfpark/batch-shipyard:latest-cli jobs add --tail stdout.txt\n```\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/Run-Job-1.png)\n\n\n#### Delete our pool \n(DO NOT DO THIS UNTIL YOU ARE DONE EXPLORING THE AZURE PORTAL) (you do not have to do this, you can set the auto scaling down to zero when the pool is not in use)\n```\nsudo docker run --rm -it -v /home/shipyarduser/config:/configs -e SHIPYARD_CONFIGDIR=/configs alfpark/batch-shipyard:latest-cli pool del\n```\n\n\n---\n\n## In the Azure Portal\n- You can see the pools, jobs, output files (stdout, stderr) and ssh into each node:\n\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/Jobs-UI-1.png)\n\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/Jobs-UI-2.png)\n\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/Jobs-UI-3.png)\n\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/ssh-To-Node-1.png)\n\n![alt tag](https://raw.githubusercontent.com/AdamPaternostro/Azure-Docker-Shipyard/master/images/ssh-To-Node-2.png)\n\n## What's Next\n- Change the Docker image to do some processing\n\n- In the jobs.json you can pass parameters to your container.  Typically you would stage your data to be processed in Blob or Azure Data Lake Storage (ADLS) and then download to the batch worker node, process and then upload the results.  The parameter is typically the name of the blob data or ADLS data to process.\n\n- To use this for a real workload you would need a small program to generate a jobs.json and then submit the job to Azure Batch\n\n- You can also use Azure Functions and go serverless to submit jobs through Azure Batch: https://github.com/Azure/batch-shipyard/blob/master/docs/60-batch-shipyard-site-extension.md\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadampaternostro%2Fazure-docker-shipyard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadampaternostro%2Fazure-docker-shipyard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadampaternostro%2Fazure-docker-shipyard/lists"}