{"id":20749299,"url":"https://github.com/agnostiqhq/covalent-awsbatch-plugin","last_synced_at":"2025-04-28T12:44:15.671Z","repository":{"id":56738494,"uuid":"467151797","full_name":"AgnostiqHQ/covalent-awsbatch-plugin","owner":"AgnostiqHQ","description":"Executor plugin interfacing Covalent with AWS Batch","archived":false,"fork":false,"pushed_at":"2025-03-03T16:59:15.000Z","size":490,"stargazers_count":11,"open_issues_count":9,"forks_count":2,"subscribers_count":9,"default_branch":"develop","last_synced_at":"2025-03-25T07:01:49.680Z","etag":null,"topics":["aws-batch","batch-processing","cloud-computing","covalent","docker","elastic-container-registry","etl","parallelization","pipelines","python","python3","workflow"],"latest_commit_sha":null,"homepage":"https://agnostiq.ai/covalent","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AgnostiqHQ.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-07T15:30:08.000Z","updated_at":"2024-02-26T08:47:54.000Z","dependencies_parsed_at":"2023-11-13T18:25:10.700Z","dependency_job_id":"d74fa465-09bf-49d2-860f-7a7a3a55c2fb","html_url":"https://github.com/AgnostiqHQ/covalent-awsbatch-plugin","commit_stats":{"total_commits":75,"total_committers":11,"mean_commits":6.818181818181818,"dds":0.6133333333333333,"last_synced_commit":"b3cee72fd7f58ffc7ee79a3448ff1a39f982d722"},"previous_names":[],"tags_count":31,"template":false,"template_full_name":"AgnostiqHQ/covalent-executor-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AgnostiqHQ%2Fcovalent-awsbatch-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AgnostiqHQ%2Fcovalent-awsbatch-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AgnostiqHQ%2Fcovalent-awsbatch-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AgnostiqHQ%2Fcovalent-awsbatch-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AgnostiqHQ","download_url":"https://codeload.github.com/AgnostiqHQ/covalent-awsbatch-plugin/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251315685,"owners_count":21569849,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-batch","batch-processing","cloud-computing","covalent","docker","elastic-container-registry","etl","parallelization","pipelines","python","python3","workflow"],"created_at":"2024-11-17T08:22:09.771Z","updated_at":"2025-04-28T12:44:15.612Z","avatar_url":"https://github.com/AgnostiqHQ.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u0026nbsp;\n\n\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/AgnostiqHQ/covalent-awsbatch-plugin/main/assets/aws_batch_readme_banner.jpg\" width=150%\u003e\n\n[![covalent](https://img.shields.io/badge/covalent-0.177.0-purple)](https://github.com/AgnostiqHQ/covalent)\n[![python](https://img.shields.io/pypi/pyversions/covalent-awsbatch-plugin)](https://github.com/AgnostiqHQ/covalent-awsbatch-plugin)\n[![tests](https://github.com/AgnostiqHQ/covalent-awsbatch-plugin/actions/workflows/tests.yml/badge.svg)](https://github.com/AgnostiqHQ/covalent-awsbatch-plugin/actions/workflows/tests.yml)\n[![codecov](https://codecov.io/gh/AgnostiqHQ/covalent-awsbatch-plugin/branch/main/graph/badge.svg?token=QNTR18SR5H)](https://codecov.io/gh/AgnostiqHQ/covalent-awsbatch-plugin)\n[![apache](https://img.shields.io/badge/License-Apache_License_2.0-blue)](https://www.apache.org/licenses/LICENSE-2.0)\n\n\u003c/div\u003e\n\n## Covalent AWS Batch Plugin\n\nCovalent is a Pythonic workflow tool used to execute tasks on advanced computing hardware.\n\nThis executor plugin interfaces Covalent with [AWS Batch](https://docs.aws.amazon.com/batch/) which allows tasks in a covalent workflow to be executed as AWS batch jobs.\n\n## 1. Installation\n\nTo use this plugin with Covalent, simply install it using `pip`:\n\n```\npip install covalent-awsbatch-plugin\n```\n\n## 2. Usage Example\n\nThis is an example of how a workflow can be adapted to utilize the AWS Batch Executor. Here we train a simple Support Vector Machine (SVM) model and use an existing AWS Batch Compute environment to run the `train_svm` electron as a batch job. We also note we require [DepsPip](https://covalent.readthedocs.io/en/latest/concepts/concepts.html#depspip) to install the dependencies when creating the batch job.\n\n```python\nfrom numpy.random import permutation\nfrom sklearn import svm, datasets\nimport covalent as ct\n\ndeps_pip = ct.DepsPip(\n\tpackages=[\"numpy==1.23.2\", \"scikit-learn==1.1.2\"]\n)\n\nexecutor = ct.executor.AWSBatchExecutor(\n    s3_bucket_name = \"covalent-batch-qa-job-resources\",\n    batch_queue = \"covalent-batch-qa-queue\",\n    batch_execution_role_name = \"ecsTaskExecutionRole\",\n    batch_job_role_name = \"covalent-batch-qa-job-role\",\n    batch_job_log_group_name = \"covalent-batch-qa-log-group\",\n    vcpu = 2, # Number of vCPUs to allocate\n    memory = 3.75, # Memory in GB to allocate\n    time_limit = 300, # Time limit of job in seconds\n)\n\n# Use executor plugin to train our SVM model.\n@ct.electron(\n    executor=executor,\n    deps_pip=deps_pip\n)\ndef train_svm(data, C, gamma):\n    X, y = data\n    clf = svm.SVC(C=C, gamma=gamma)\n    clf.fit(X[90:], y[90:])\n    return clf\n\n@ct.electron\ndef load_data():\n    iris = datasets.load_iris()\n    perm = permutation(iris.target.size)\n    iris.data = iris.data[perm]\n    iris.target = iris.target[perm]\n    return iris.data, iris.target\n\n@ct.electron\ndef score_svm(data, clf):\n    X_test, y_test = data\n    return clf.score(\n    \tX_test[:90],\n\t \ty_test[:90]\n    )\n\n@ct.lattice\ndef run_experiment(C=1.0, gamma=0.7):\n    data = load_data()\n    clf = train_svm(\n    \tdata=data,\n    \tC=C,\n    \tgamma=gamma\n    )\n    score = score_svm(\n    \tdata=data,\n\t \tclf=clf\n    )\n    return score\n\n# Dispatch the workflow\ndispatch_id = ct.dispatch(run_experiment)(\n\tC=1.0,\n\tgamma=0.7\n)\n\n# Wait for our result and get result value\nresult = ct.get_result(dispatch_id=dispatch_id, wait=True).result\n\nprint(result)\n```\n\nDuring the execution of the workflow one can navigate to the UI to see the status of the workflow, once completed however the above script should also output a value with the score of our model.\n\n```\n0.9777777777777777\n```\n\n\n## 3. Configuration\n\nThere are many configuration options that can be passed in to the class `ct.executor.AWSBatchExecutor` or by modifying the [covalent config file](https://covalent.readthedocs.io/en/latest/how_to/config/customization.html) under the section `[executors.awsbatch]`\n\nFor more information about all of the possible configuration values visit our [read the docs (RTD) guide](https://covalent.readthedocs.io/en/latest/api/executors/awsbatch.html) for this plugin.\n\n## 4. Required AWS Resources\n\nIn order to run your workflows with covalent there are a few notable AWS resources that need to be provisioned first.\n\nFor more information regarding which cloud resources need to be provisioned visit our [read the docs (RTD) guide](https://covalent.readthedocs.io/en/latest/api/executors/awsbatch.html) for this plugin.\n\nThe required AWS resources include a Batch Job Definition, Batch Job Role, Batch Queue, Batch Compute Environment, Log Group, Subnet, VPC, and an S3 Bucket.\n\n## Getting Started with Covalent\n\n\nFor more information on how to get started with Covalent, check out the project [homepage](https://github.com/AgnostiqHQ/covalent) and the official [documentation](https://covalent.readthedocs.io/en/latest/).\n\n## Release Notes\n\nRelease notes for this plugin are available in the [Changelog](https://github.com/AgnostiqHQ/covalent-awsbatch-plugin/blob/main/CHANGELOG.md).\n\n## Citation\n\nPlease use the following citation in any publications:\n\n\u003e W. J. Cunningham, S. K. Radha, F. Hasan, J. Kanem, S. W. Neagle, and S. Sanand.\n\u003e *Covalent.* Zenodo, 2022. https://doi.org/10.5281/zenodo.5903364\n\n## License\n\nCovalent is licensed under the Apache License 2.0. See the [LICENSE](https://github.com/AgnostiqHQ/covalent-executor-template/blob/main/LICENSE) file or contact the [support team](mailto:support@agnostiq.ai) for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagnostiqhq%2Fcovalent-awsbatch-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fagnostiqhq%2Fcovalent-awsbatch-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagnostiqhq%2Fcovalent-awsbatch-plugin/lists"}