Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/jcm-ai/robust-mlops-elevating-machine-learning-to-production-scale
This project aims to address the challenge faced by the Office of Foreign Labor Certification in managing a high volume of job certification applications for employers seeking to hire foreign workers in the United States. By leveraging ML models, this project seeks to streamline the process of shortlisting visa applicants based on historical data.
https://github.com/jcm-ai/robust-mlops-elevating-machine-learning-to-production-scale
aws-ec2 aws-ecr aws-s3 cicd-pipeline data-science docker-image end-to-end exploratory-data-analysis git github-actions html machine-learning-models mlops mlops-project model-deployment mongodb numpy pandas python yaml
Last synced: 23 days ago
JSON representation
This project aims to address the challenge faced by the Office of Foreign Labor Certification in managing a high volume of job certification applications for employers seeking to hire foreign workers in the United States. By leveraging ML models, this project seeks to streamline the process of shortlisting visa applicants based on historical data.
- Host: GitHub
- URL: https://github.com/jcm-ai/robust-mlops-elevating-machine-learning-to-production-scale
- Owner: jcm-ai
- License: mit
- Created: 2024-12-03T09:18:41.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2024-12-18T12:15:33.000Z (about 1 month ago)
- Last Synced: 2024-12-28T17:27:46.415Z (27 days ago)
- Topics: aws-ec2, aws-ecr, aws-s3, cicd-pipeline, data-science, docker-image, end-to-end, exploratory-data-analysis, git, github-actions, html, machine-learning-models, mlops, mlops-project, model-deployment, mongodb, numpy, pandas, python, yaml
- Language: Jupyter Notebook
- Homepage:
- Size: 7.33 MB
- Stars: 0
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Robust MLOps: Elevating Machine Learning to Production Scale
**This is the final outcome of this project:**
- [Anaconda](https://www.anaconda.com/)
- [Vs Code](https://code.visualstudio.com/download)
- [Git and Git Bash](https://git-scm.com/)
- [Flowchart](https://whimsical.com/)
- [MLOps Tool](https://www.evidentlyai.com/) - Evidently AI
- [MongoDB](https://account.mongodb.com/account/login)
- [Dataset link](https://www.kaggle.com/datasets/moro23/easyvisa-dataset)
- [Cloud Computing](https://aws.amazon.com/free)
## Git commands
```Bash
git add .
```
```Bash
git commit -m "Updated"
```
```Bash
git push origin main
```
## How to run?
```Bash
conda create -n usvisa python=3.8 -y
```
```Bash
source activate base
```
```Bash
conda activate usvisa
```
```Bash
pip install -r requirements.txt
```
## Project WorkFlow:
1. constants
2. entity
3. components
4. pipeline
5. Main file
## Export the environment variable:
```Bash
export MONGODB_URL="mongodb+srv://:...."
```
```Bash
export AWS_ACCESS_KEY_ID=
```
```Bash
export AWS_SECRET_ACCESS_KEY=
```
## AWS CICD Deployment with Github Actions
### 1. Login to AWS console.
### 2. Create IAM user for deployment
#### with specific access:
1. EC2 access : It is virtual machine
2. ECR: Elastic Container registry to save your docker image in aws
#### Description: About the deployment
1. Build docker image of the source code
2. Push your docker image to ECR
3. Launch Your EC2
4. Pull Your image from ECR in EC2
5. Lauch your docker image in EC2
#### Policy:
```Bash
1. AmazonEC2ContainerRegistryFullAccess
```
```Bash
2. AmazonEC2FullAccess
```
```Bash
3. AmazonS3FullAccess
```
### 3. Create ECR repo to store/save docker image
```Bash
Save the URI: 954976285001.dkr.ecr.us-east-1.amazonaws.com/usvisa
```
### 4. Create EC2 machine (Ubuntu)
### 5. Open EC2 and Install docker in EC2 Machine:
**optinal**
```Bash
sudo apt-get update -y
```
```Bash
sudo apt-get upgrade
```
**required**
```Bash
curl -fsSL https://get.docker.com -o get-docker.sh
```
```Bash
sudo sh get-docker.sh
```
```Bash
sudo usermod -aG docker ubuntu
```
```Bash
newgrp docker
```
### 6. Configure EC2 as self-hosted runner:
```Bash
setting>actions>runner>new self hosted runner> choose os> then run command one by one
```
### 7. Setup github secrets:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_DEFAULT_REGION
- ECR_REPO
- MONGODB_URL