https://github.com/briandenicola/datafactory-batchjob-demo
This is to demo how to use Azure Data Factory with AKS/KEDA to run batch jobs in Azure
https://github.com/briandenicola/datafactory-batchjob-demo
aks datafactory keda
Last synced: 7 months ago
JSON representation
This is to demo how to use Azure Data Factory with AKS/KEDA to run batch jobs in Azure
- Host: GitHub
- URL: https://github.com/briandenicola/datafactory-batchjob-demo
- Owner: briandenicola
- License: apache-2.0
- Created: 2021-10-25T14:58:49.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2024-08-06T17:48:40.000Z (about 1 year ago)
- Last Synced: 2025-01-15T07:38:53.314Z (9 months ago)
- Topics: aks, datafactory, keda
- Language: HCL
- Homepage:
- Size: 124 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Overview
This is to demo how to use Azure Data Factory with AKS/KEDA to run batch jobs in Azure
# Architecture
# Deployment
### Prerequisite
* A Linux machine or Windows Subsytem for Linux or Docker for Windows
* Azure Cli and an Azure Subscription
* Terraform 0.12 or greater
* Kubectl
* Helm
* A virtual network with 2 subnets defined - one for private endpoint and one for kubernetes
* DNS zones for storage private endpoints - [refrenece](https://docs.microsoft.com/en-us/azure/storage/common/storage-private-endpoints)
* An Azure Container Repository
## Infrastructure
1. az extension add --name aks-preview
1. az extension update --name aks-preview
1. az login
1. az feature register --namespace "Microsoft.ContainerService" --name "AKS-AzureKeyVaultSecretsProvider"
1. az feature register --namespace "Microsoft.ContainerService" --name "EnablePodIdentityPreview"
1. az feature register --namespace "Microsoft.ContainerService" --name "AKS-OpenServiceMesh"
1. az feature register --namespace "Microsoft.ContainerService" --name "DisableLocalAccountsPreview"
1. az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService')].{Name:name,State:properties.state}"
1. Wait till the above features are enabled.
1. Update uat.tfvars with values for your environment
1. az provider register --namespace Microsoft.ContainerService
1. cd infrastructure
1. terraform init -backend=true -backend-config="access_key=${access_key}" -backend-config="key=uat.terraform.tfstate"
1. terraform plan -out="uat.plan" -var "resource_group_name=DevSub_K8S_RG" -var-file="uat.tfvars"
1. terraform apply -auto-approve "uat.plan"
1. ./aks-keda-install.sh $SUBSCRIPTION_ID $RG $CLUSTER_NAME $KEDA_IDENTITY $BATCH_IDENTITY
## Source
### Build
1. cd source
1. az login
1. az acr login -n ${ACR_NAME}
1. docker build -f DOCKERFILE -t ${ACR_NAME}.azurecr.io/queue-processor:{BUILD_ID} .
1. docker push ${ACR_NAME}.azurecr.io/queue-processor:{BUILD_ID}
### Deploy
1. cd chart
1. Update values.yaml
1. helm upgrade -i batchdemo .
## Datafactory Pipeline
_TBD_
# Validation
_TBD_
# Backlog
- [ ] Update Readme with additional details