Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dciborow/azureml-scheduler-on-adf
(DEPRICATED)Create Scheduler using Data Factory v2 to run (DEPRICATED) Azure Machine Learning Experimentation
https://github.com/dciborow/azureml-scheduler-on-adf
azure azure-data-factory azure-machine-learning
Last synced: 10 days ago
JSON representation
(DEPRICATED)Create Scheduler using Data Factory v2 to run (DEPRICATED) Azure Machine Learning Experimentation
- Host: GitHub
- URL: https://github.com/dciborow/azureml-scheduler-on-adf
- Owner: dciborow
- License: mit
- Created: 2017-11-15T02:13:46.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2018-03-20T13:01:57.000Z (almost 7 years ago)
- Last Synced: 2024-10-18T07:16:58.531Z (4 months ago)
- Topics: azure, azure-data-factory, azure-machine-learning
- Language: PowerShell
- Homepage: https://docs.microsoft.com/en-us/azure/machine-learning/preview/
- Size: 405 KB
- Stars: 3
- Watchers: 2
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Schedule ML Experimention on Data Factory v2
This sample shows how to leverage ADF to run an experiment.
## Prerequisites
Create Azure Machine Learning Preview accounts and install Azure Machine Learning Workbench using [these instructions](https://docs.microsoft.com/en-us/azure/machine-learning/preview/quickstart-installation).
## Create Compute Targets
First, create compute targets for running the parameter sweep: DSVM and optionally, HDInsight Spark Cluster. Select **File**, **Open Command Prompt** and enter following commands to create the compute targets.
Create a Ubuntu baesd DSVM using [these instructions](https://docs.microsoft.com/en-us/azure/machine-learning/data-science-virtual-machine/dsvm-ubuntu-intro#create-your-data-science-virtual-machine-for-linux). Attach it as compute target, and then prepare it by using:
```bash
$ az ml computetarget attach --name --address --username --password --type remotedocker
```
```bash
$ az ml experiment prepare -c
```
OR
Create HDInsight Spark Cluster using [these instructions](https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-spark-jupyter-spark-sql). Attach it as compute target, and then prepare it by using:
```bash
$ az ml computetarget attach --name --address --username --password --type cluster
```
```bash
$ az ml experiment prepare -c
```
## Run the Model from Local Machine
DSVM running a pyspark job.
```bash
$ az ml experiment submit -c
```
HDInsight Spark running a pyspark job.
```bash
$ az ml experiment submit -c
```
## Deploy ADFv2 and other required resources
The creation process creates a new service princple account when run locally. This new account must be used to login to the az ml client from within the workbench if the AMLW secret store is used to store passwords.
1. Download and place IaC folder within your AML Workspace project directory.
1. Download and place batch task within your AML Workspace project directory.
1. Select **File**, **Open Powershell** and enter the following command to create the deployment.
1. [Make VSTS Git Access Key](https://docs.microsoft.com/en-us/vsts/accounts/use-personal-access-tokens-to-authenticate)
1. DSVM Compute Target
1. Path in AML Workbench project to code
```
.\IaC\CreateDeployment.ps1 `
-gitPassword "" `
-dsvm "" `
-pythonPath ""
```
## On Batch Windows DSVM Image
RDP into Node using username rdpuser, and provided password.
### Update Azure PowerShell to >5.X
1. microsoft web platform installer is on the desktop of DSVM, use this to update Powershell
### Install AMLW - MSI must be run from admin cmd.
1. Download AML Workbench from here, https://aka.ms/azureml-wb-msi
1. Windows Key
1. Type cmd
1. Right click, select Run as Admin
1. Change to D drive with d:
1. cd User\{username}\Downloads
1. AmlWorkbenchSetup.msi
1. Log into AML Workbench, this is required to create starter folders, remote experiment submission will fail if this is not complete.
# Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.microsoft.com.
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [[email protected]](mailto:[email protected]) with any additional questions or comments.
## Disclaimer
©2017 Microsoft Corporation. All rights reserved. This information is provided "as-is" and may change without notice. Microsoft makes no warranties, express or implied, with respect to the information provided here.