{"id":24018624,"url":"https://github.com/nsakki55/aws-mlops-handson","last_synced_at":"2025-06-11T23:35:27.668Z","repository":{"id":168887715,"uuid":"640131114","full_name":"nsakki55/aws-mlops-handson","owner":"nsakki55","description":"This repository provides a comprehensive ML infrastructure for CTR prediction, focusing on AWS services and offering practical learning experience for MLOps.","archived":false,"fork":false,"pushed_at":"2023-07-27T13:14:53.000Z","size":13135,"stargazers_count":62,"open_issues_count":0,"forks_count":8,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-15T16:50:22.862Z","etag":null,"topics":["aws","ecs","mlops"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nsakki55.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-13T05:00:01.000Z","updated_at":"2025-02-23T05:59:00.000Z","dependencies_parsed_at":null,"dependency_job_id":"b50474b1-e5bb-47ce-b6fc-4baf081ecef2","html_url":"https://github.com/nsakki55/aws-mlops-handson","commit_stats":null,"previous_names":["nsakki55/aws-mlops-handson"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nsakki55/aws-mlops-handson","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nsakki55%2Faws-mlops-handson","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nsakki55%2Faws-mlops-handson/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nsakki55%2Faws-mlops-handson/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nsakki55%2Faws-mlops-handson/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nsakki55","download_url":"https://codeload.github.com/nsakki55/aws-mlops-handson/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nsakki55%2Faws-mlops-handson/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259363795,"owners_count":22846353,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","ecs","mlops"],"created_at":"2025-01-08T10:17:26.170Z","updated_at":"2025-06-11T23:35:27.662Z","avatar_url":"https://github.com/nsakki55.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS MLOps Handson\nThis repository is designed to provide a comprehensive ML infrastructure for CTR (Click-Through Rate) prediction.  \nWith a focus on AWS services, this repository offer practical learning experience for MLOps.  \nSlide[japanese]: https://speakerdeck.com/nsakki55/cyberagent-aishi-ye-ben-bu-mlopsyan-xiu-ying-yong-bian\n\n## Key Features\n### Python Development Environment  \nWe guide you through setting up a Python development environment that ensures code quality and maintainability.   \nThis environment is carefully configured to enable efficient development practices and facilitate collaboration.\n\n### Train Pipeline\nThis repository includes the implementation of a training pipeline.   \nThis pipeline covers the stages, including data preprocessing, model training, and evaluation.   \n\n### Prediction Server\nThis repository provides an implementation of a prediction server that serves predictions based on your trained CTR prediction model.   \n\n### AWS Deployment\nTo showcase industry-standard practices, this repository guide you in deploying the training pipeline and inference server on AWS. \n\n\n## AWS Infra Architecture\nAWS Infra Architecture made by this repository.\n\n### ML Pipeline\n![ml_pipeline](./imgs/ml_pipeline_architecture.png)\n\n### Predict Server\n![predict_server](./imgs/predict_server_architecture.png)\n\n## Requirements\n| Software                   | Install (Mac)              |\n|----------------------------|----------------------------|\n| [pyenv](https://github.com/pyenv/pyenv#installation)             | `brew install pyenv`       |\n| [Poetry](https://python-poetry.org/docs/#installation)           | curl -sSL https://install.python-poetry.org \u0026#x7C; python3 - |\n| [direnv](https://formulae.brew.sh/formula/direnv)           | `brew install direnv`      |\n| [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli#install-terraform)    | `brew install terraform`   |\n| [Docker](https://docs.docker.com/desktop/install/mac-install/) | install via dmg |\n| [awscli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-installjkkkkj.html) | `curl \"https://awscli.amazonaws.com/AWSCLIV2.pkg\" -o \"AWSCLIV2.pkg\"` |\n\n## Setup\n### Install Python Dependencies\nUse `pyenv` to install Python 3.9.0 environment\n```bash\n$ pyenv install 3.9.0\n$ pyenv local 3.9.0\n```\n\nUse `poetry` to install library dependencies \n```bash\n$ poetry install\n```\n\n### Configure environment variable\nUse `direnv` to configure environment variable\n```bash\n$ cp .env.example .env\n$ direnv allow .\n```\nSet your environment variable setting\n```bash\nAWS_REGION=\nAWS_ACCOUNT_ID=\nAWS_PROFILE=\nAWS_BUCKET=\nAWS_ALB_DNS=\nUSER_NAME=\nVERSION=2023-05-11\nMODEL=sgd_classifier_ctr_model\n```\n\n### Create AWS Resources\nmove current directory to `infra` \n```bash\n$ cd infra\n```\n\nUse terraform to create aws resources.   \nApply terraform\n```bash\n$ terraform init\n$ terraform apply\n```\n\n### Prepare train data\nunzip train data\n```bash\n$ unzip train_data.zip\n```\n\nupload train data to S3\n```bash\n$ aws s3 mv train_data s3://$AWS_BUCKET\n```\n\n## Code static analysis tool\n| Tool                   | Usage              |\n|----------------------------|----------------------------|\n| [isort](https://pycqa.github.io/isort/)             | library import statement check     |\n| [black](https://pypi.org/project/black/)           | format code style  |\n| [flake8](https://flake8.pycqa.org/en/latest/)           | code quality check      |\n| [mypy](https://mypy.readthedocs.io/en/stable/)    |  static type checking |\n| [pysen](https://github.com/pfnet/pysen)    | manage static analysis tool  |\n\n\n\n## Usage\nBuild ML Pipeline\n```bash\n$ make build-ml\n```\n\nRun ML Pipeline\n```bash\n$ make run-ml\n```\n\nBuild Predict API\n```bash\n$ make build-predictor\n```\n\nRun Predict API locally\n```bash\n$ make up\n```\n\nShutdown Predict API locally\n```bash\n$ make down\n```\n\nRun formatter\n```bash\n$ make format\n```\n\nRun linter \n```bash\n$ make lint \n```\n\nRun pytest \n```bash\n$ make test \n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnsakki55%2Faws-mlops-handson","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnsakki55%2Faws-mlops-handson","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnsakki55%2Faws-mlops-handson/lists"}