{"id":24880131,"url":"https://github.com/s4heid/bosh-azure-stemcell-mirror","last_synced_at":"2026-04-27T21:32:06.716Z","repository":{"id":274814074,"uuid":"907448320","full_name":"s4heid/bosh-azure-stemcell-mirror","owner":"s4heid","description":"Mirror for stemcells published on bosh.io","archived":false,"fork":false,"pushed_at":"2025-10-31T23:04:20.000Z","size":101,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-01T00:14:22.461Z","etag":null,"topics":["azure","bosh"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/s4heid.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-23T15:51:21.000Z","updated_at":"2025-10-31T23:04:23.000Z","dependencies_parsed_at":"2025-01-29T15:27:28.232Z","dependency_job_id":"ccaac695-d002-420d-8b5a-b0303ca24310","html_url":"https://github.com/s4heid/bosh-azure-stemcell-mirror","commit_stats":null,"previous_names":["s4heid/bosh-azure-stemcell-mirror"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/s4heid/bosh-azure-stemcell-mirror","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s4heid%2Fbosh-azure-stemcell-mirror","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s4heid%2Fbosh-azure-stemcell-mirror/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s4heid%2Fbosh-azure-stemcell-mirror/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s4heid%2Fbosh-azure-stemcell-mirror/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/s4heid","download_url":"https://codeload.github.com/s4heid/bosh-azure-stemcell-mirror/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s4heid%2Fbosh-azure-stemcell-mirror/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32356598,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-27T20:07:02.737Z","status":"ssl_error","status_checked_at":"2026-04-27T20:07:00.910Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","bosh"],"created_at":"2025-02-01T10:19:01.375Z","updated_at":"2026-04-27T21:32:06.710Z","avatar_url":"https://github.com/s4heid.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bosh-azure-stemcell-mirror\n\n[![Tests](https://github.com/s4heid/bosh-azure-stemcell-mirror/actions/workflows/test.yaml/badge.svg?branch=main)](https://github.com/s4heid/bosh-azure-stemcell-mirror/actions/workflows/test.yaml)\n[![Open in Dev Containers](https://img.shields.io/static/v1?label=Dev%20Containers\u0026message=Open\u0026color=blue)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/s4heid/bosh-azure-stemcell-mirror)\n\nThis repository contains an Azure Container Apps Job that mirrors BOSH stemcells from [bosh.io](https://bosh.io/stemcells) to an [Azure Compute Gallery](https://learn.microsoft.com/en-us/azure/virtual-machines/azure-compute-gallery).\n\nThe job runs on a scheduled basis (daily by default) to check for new stemcell versions and automatically uploads them to your Azure infrastructure.\n\n## Architecture\n\n```mermaid\nsequenceDiagram\n    participant User\n    participant StemcellMirror as StemcellMirror.run()\n    participant BoshIO as bosh.io\n    participant Storage as Azure Storage Account\n    participant Gallery as Azure Compute Gallery\n\n    User-\u003e\u003eStemcellMirror: Invoke\n    StemcellMirror-\u003e\u003eGallery: gallery_image_version_exists?\n    alt VersionExists\n        StemcellMirror-\u003e\u003eUser: No new stemcell required\n    else NoVersionFound\n        StemcellMirror-\u003e\u003eBoshIO: Download latest stemcell (tgz)\n        StemcellMirror-\u003e\u003eStemcellMirror: Extract .vhd from tgz\n        StemcellMirror-\u003e\u003eStorage: Upload root.vhd\n        StemcellMirror-\u003e\u003eGallery: check_or_create_gallery_image\n        StemcellMirror-\u003e\u003eGallery: create_gallery_image_version\n    end\n    StemcellMirror-\u003e\u003eUser: Completed stemcell check\n```\n\n## Prerequisites\n\n- [Azure Developer CLI (azd)](https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/install-azd)\n- [Azure CLI (az)](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)\n- [Docker](https://docs.docker.com/get-docker/)\n- An Azure subscription\n\n## Deployment\n\n### Initial Setup\n\n1. Clone this repository and navigate to the project directory.\n\n2. Log in to your Azure subscription:\n\n    ```bash\n    azd auth login\n    ```\n\n3. Initialize the Azure Developer environment (if not already done):\n\n    ```bash\n    azd init\n    ```\n\n    You'll be prompted to provide:\n    - **Environment name**: A unique name for your deployment (e.g., `stemcell-mirror-prod`)\n    - **Azure location**: The Azure region where resources will be deployed (e.g., `eastus`)\n\n### Deploy to Azure\n\nDeploy the infrastructure and application in one command:\n\n```bash\nazd up\n```\n\nThis command will:\n\n- Provision all required Azure resources (see [Infrastructure](#infrastructure) section)\n- Build the Docker container image\n- Push the image to Azure Container Registry\n- Deploy the Container Apps Job\n\nAlternatively, you can run these steps separately:\n\n```bash\nazd provision  # Provision infrastructure\nazd deploy     # Build and deploy the application\n```\n\n### Infrastructure\n\nThe deployment creates the following Azure resources:\n\n| Resource Type | Purpose |\n|---------------|---------|\n| **Resource Group** | Container for all resources |\n| **Container Registry** | Stores Docker images |\n| **Storage Account** | Stores stemcell VHD files |\n| **Compute Gallery** | Stores stemcell image definitions and versions |\n| **Container Apps Environment** | Runtime environment for the job |\n| **Container Apps Job** | Scheduled job that runs the mirror process |\n| **Log Analytics Workspace** | Collects logs and telemetry |\n| **Application Insights** | Application monitoring and diagnostics |\n| **Key Vault** | Securely stores sensitive configuration (e.g., GitHub token) |\n| **Managed Identity** | Provides authentication for Azure resources |\n\n### Role Assignments\n\nThe managed identity is automatically assigned the following roles:\n\n- **Contributor** (Resource Group scope): For managing compute resources\n- **Storage Blob Data Contributor** (Storage Account scope): For uploading VHD files\n- **AcrPull** (Container Registry scope): For pulling container images\n- **Compute Gallery Sharing Admin** (Compute Gallery scope): For creating gallery images\n- **Key Vault Secrets User** (Key Vault scope): For reading secrets from Key Vault\n\n## Configuration\n\n### Environment Variables\n\nThe application is configured through environment variables. These are automatically set during deployment but can be customized in [`infra/main.parameters.json`](infra/main.parameters.json).\n\n#### (Required) Azure Deployment\n\nThese variables are automatically configured by the deployment:\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `AZURE_REGION` | Azure region | Deployment location |\n| `AZURE_RESOURCE_GROUP` | Resource group name | Set automatically |\n| `AZURE_SUBSCRIPTION_ID` | Azure subscription ID | Set automatically |\n| `AZURE_MANAGED_IDENTITY_ID` | User Managed identity client ID | Set automatically |\n| `AZURE_STORAGE_ACCOUNT_NAME` | Storage account name | Set automatically |\n| `AZURE_GALLERY_NAME` | Azure Compute Gallery name | Set automatically |\n\n#### (Optional) Azure Compute Gallery\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `BASM_GALLERY_PUBLISHER` | Gallery image publisher | `bosh` |\n| `BASM_GALLERY_OFFER` | Gallery image offer | Extracted from stemcell series |\n| `BASM_GALLERY_SKU` | Gallery image SKU | Extracted from stemcell series |\n| `BASM_GALLERY_IMAGE_NAME` | Gallery image definition name | Extracted from stemcell series |\n\n#### (Optional) Stemcell\n\n##### bosh.io\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `BASM_STEMCELL_SERIES` | BOSH stemcell series to mirror (from bosh.io) | `bosh-azure-hyperv-ubuntu-jammy-go_agent` |\n| `BASM_MOUNTED_DIRECTORY` | Directory for temporary extraction. | `/stemcellfiles` |\n\n\u003e [!IMPORTANT]\n\u003e The `BASM_STEMCELL_SERIES` environment variable must be set to a valid bosh.io stemcell series name (e.g. `bosh-azure-hyperv-ubuntu-noble-go_agent`, `bosh-azure-hyperv-ubuntu-jammy-go_agent`, `bosh-azure-hyperv-ubuntu-xenial-go_agent`).\n\u003e\n\u003e A list of available stemcell series can be found at [bosh.io/stemcells](https://bosh.io/stemcells/).\n\n##### Ephemeral Storage\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `BASM_MOUNTED_DIRECTORY` | Directory for temporary extraction. | `/stemcellfiles` |\n\n\u003e [!NOTE]\n\u003e The `BASM_MOUNTED_DIRECTORY` allows you to set a custom temporary extraction directory within the container. This is helpful if you want to use smaller container sizes with ephemeral storage for extraction, since downloaded stemcells are usually larger than 5GB.\n\n#### (Optional) Notification\n\nThe stemcell mirror can send out a notification about successful upload of new stemcells.\n\n##### GitHub Actions Workflow Dispatch\n\nConfigure these variables to dispatch an external GitHub Actions workflow whenever a new stemcell version is published. The `BASM_NOTIFY_GITHUB_TOKEN` must be stored in Azure Key Vault as a secret named `github-token`. If this secret is not set, notifications remain disabled.\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `BASM_NOTIFY_GITHUB_TOKEN` | GitHub token with `workflow` scope (stored in Key Vault as `github-token`) | _Required to enable notifications_ |\n| `BASM_NOTIFY_GITHUB_API_URL` | GitHub API base URL (`https://api.github.com` for GitHub.com, `https://\u003cghe-host\u003e/api/v3` for GHES) | `https://api.github.com` |\n| `BASM_NOTIFY_GITHUB_OWNER` | GitHub repository owner | - |\n| `BASM_NOTIFY_GITHUB_REPO` | GitHub repository name | - |\n| `BASM_NOTIFY_GITHUB_WORKFLOW` | Workflow filename or ID to dispatch | - |\n| `BASM_NOTIFY_GITHUB_REF` | Branch or tag reference used for the workflow dispatch | - |\n\n##### Setting the GitHub Token in Key Vault\n\nAfter the initial deployment, store your GitHub token in Key Vault:\n\n```bash\naz keyvault secret set \\\n  --vault-name \u003ckey-vault-name\u003e \\\n  --name github-token \\\n  --value \"\u003cyour-github-token\u003e\"\n```\n\nYou can find the Key Vault name in the `.azure/\u003cenvironment-name\u003e/.env` file:\n\n```bash\nAZURE_KEY_VAULT_NAME=\u003ckey-vault-name\u003e\n```\n\n\u003e [!NOTE]\n\u003e The GitHub token requires the `workflow` scope to trigger GitHub Actions workflow dispatches.\n\n### Resource Configuration\n\n#### Job Resources\n\nThe Container Apps Job is configured with:\n\n- **CPU**: 1.0 cores\n- **Memory**: 2.0 GiB\n- **Replica timeout**: 900 seconds (15 minutes)\n- **Retry limit**: 1 attempt\n\nThese can be adjusted in [`infra/app/app.bicep`](infra/app/app.bicep) if needed.\n\n#### Storage Volume\n\nThe job uses an EmptyDir volume mounted at `/stemcellfiles` for temporary extraction of stemcell archives. This provides better performance than using system temp directories.\n\n## Manual Execution\n\nTo manually trigger the job:\n\n```bash\naz containerapp job start --name \u003cjob-name\u003e --resource-group \u003cresource-group\u003e\n```\n\nYou can find the job name and resource group in the `.azure/\u003cenvironment-name\u003e/.env` file after deployment:\n\n```bash\nAZURE_CONTAINER_APPS_JOB_NAME=\u003cjob-name\u003e\nAZURE_RESOURCE_GROUP=\u003cresource-group\u003e\n```\n\n## Monitoring\n\n### Viewing Logs\n\nView logs in Azure Portal:\n\n1. Navigate to your Container Apps Job\n2. Select **Execution history**\n3. Click on a specific execution to view logs\n\nOr use Azure CLI:\n\n```bash\naz containerapp job execution list \\\n  --name \u003cjob-name\u003e \\\n  --resource-group \u003cresource-group\u003e\n```\n\n### Application Insights\n\nApplication telemetry is automatically collected in Application Insights. You can query logs and metrics through the Azure Portal or using KQL queries.\n\n## Development\n\n### Local Development Setup\n\n1. Configure a Python virtual environment:\n\n    ```bash\n    python -m venv .venv\n    source ./.venv/bin/activate\n    ```\n\n2. Install development dependencies:\n\n    ```bash\n    pip install -r requirements-dev.txt\n    ```\n\n### Running Tests\n\nRun the unit tests:\n\n```bash\npython -m unittest discover tests\n```\n\n### Local Execution\n\nTo run the application locally, you need to set the required environment variables:\n\n```bash\nexport AZURE_SUBSCRIPTION_ID=\"\u003cyour-subscription-id\u003e\"\nexport AZURE_RESOURCE_GROUP=\"\u003cyour-resource-group\u003e\"\nexport AZURE_STORAGE_ACCOUNT_NAME=\"\u003cyour-storage-account\u003e\"\n# ... other required variables (see .env.template)\n\npython src/main.py\n```\n\n\u003e [!IMPORTANT]\n\u003e Local execution requires Azure credentials. Use `az login` to authenticate.\n\n### Code Quality\n\nThe project uses Ruff for linting and Black for code formatting. Configuration is in [`pyproject.toml`](pyproject.toml):\n\n```bash\n# Format code\nblack src/ tests/\n\n# Lint code\nruff check src/ tests/\n```\n\n## Cleanup\n\nTo delete all Azure resources:\n\n```bash\nazd down\n```\n\nThis will remove the resource group and all associated resources.\n\n## Related Topics\n\n- [Deploying BOSH Stemcells from Azure Compute Gallery](docs/deploying-bosh-stemcells-from-azure-compute-gallery.md)\n- [Mirror BOSH stemcells using metalinks](https://github.com/dpb587/upstream-blob-mirror/blob/master/repository/bosh.io/stemcell/_metalink)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs4heid%2Fbosh-azure-stemcell-mirror","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fs4heid%2Fbosh-azure-stemcell-mirror","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs4heid%2Fbosh-azure-stemcell-mirror/lists"}