{"id":19771984,"url":"https://github.com/danieldacosta/ml-serverless-deploy","last_synced_at":"2025-04-30T17:33:05.417Z","repository":{"id":40811403,"uuid":"278925109","full_name":"DanielDaCosta/ml-serverless-deploy","owner":"DanielDaCosta","description":"Deploying Machine Learning models on AWS using Serverless Framework","archived":false,"fork":false,"pushed_at":"2024-06-24T21:07:27.000Z","size":115,"stargazers_count":4,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-06-24T22:44:52.682Z","etag":null,"topics":["aws","aws-lambda","s3-bucket","serverless","serverless-framework"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DanielDaCosta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-11T19:05:19.000Z","updated_at":"2024-06-24T21:07:30.000Z","dependencies_parsed_at":"2023-01-22T22:01:28.883Z","dependency_job_id":null,"html_url":"https://github.com/DanielDaCosta/ml-serverless-deploy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DanielDaCosta%2Fml-serverless-deploy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DanielDaCosta%2Fml-serverless-deploy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DanielDaCosta%2Fml-serverless-deploy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DanielDaCosta%2Fml-serverless-deploy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DanielDaCosta","download_url":"https://codeload.github.com/DanielDaCosta/ml-serverless-deploy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224219691,"owners_count":17275477,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-lambda","s3-bucket","serverless","serverless-framework"],"created_at":"2024-11-12T05:04:59.894Z","updated_at":"2024-11-12T05:05:00.563Z","avatar_url":"https://github.com/DanielDaCosta.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ML model deployment using Serverless Framework\nDeploying Machine Learning models on AWS using Serverless Framework.\n\nIn this repository, we'll be building the following architecture:\n\n![schema](Images/Schema_ML_Serverless.png \"Schema Architecture\")\n\n\nDon't forget to set up your AWS credentials using *aws cli*.\n\n# Details\n\nThe input data will be stored in an S3 bucket, in our case, the csv file. \n\nThe model file will also be stored inside a S3 bucket. Since the model file can be large (\u003e90Mb), we must load it during AWS Lambda inference execution from Amazon S3. You can observe that the model downloading was placed outside of the handler function. This was done to take advantage of AWS Lambda container reuse. Any code executed outside the handler method will be invoked only once upon container creation and kept in memory across calls to the same Lambda container, making subsequent calls to Lambda faster.\n\n## Files\n\n- s3_utils.py: functions for read and write from s3\n- config.py: load environments variables\n- serverless.yml: Serverless Framework YML file\n- handler.py: file to be invoked for AWS lambda when the service executes the code.\n- requirement.txt: python dependencies\n- .env: environment variables\n- Images/: contains all images from repository\n# Usage\n\nInstalling Serverless Framework\n\n```\nnpm install -g serverless\n```\nIn this repository will be using Python3. Create a repository, go inside and run the following command:\n```\nserverless create --template aws-python3\n```\n\nInstalling *serverless-python-requirements plugin*. A Serverless v1.x plugin to automatically bundle dependencies from requirements.txt and make them available in your PYTHONPATH.:\n```\nsls plugin install -n serverless-python-requirements\n```\nInstalling *serverless-dotenv-plugin*. Preload environment variables into serverless. Use this plugin if you have variables stored in a .env file that you want loaded into your serverless yaml config.:\n```\nnpm i -D serverless-dotenv-plugin\n```\n\n\nDeployment\n```\nserverless deploy\n```\n## Code\nThe model used is an energy forecasting model using XGboost. The dataset consists of hourly energy consumption rates in kWh for an industrial utility. You can check more about it in [here](https://github.com/DanielDaCosta/energy-forecast).\n\nThe model has a *forecasting horizon* (The number of time periods to forecast into the future) of 1 hour.\n\nThe model inputs are stored in s3 Bucket. It consists of the model inputs parameter of the last hour. The parameters are:\n- hour\n- weekday \n- dayofyear\n- is_weekend\n- weekofyear\n- month\n- season\n- shift\n- lag_1\n- lag_8\n- lag_25\n\nHaving energy forecast as output.\n\n**Inputs**\n\nThe inputs are saved in an s3 bucket as a *.csv*\n\n**Model**\n\nThe XGboost model is saved as *.sav* in s3\n\n**Results**\n\nThe results are also stored in s3 as a *.csv*\n\n### Pipeline\n\nCloudWatch trigger (every hour) -\u003e Read Inputs -\u003e Read Model -\u003e Prediction -\u003e Save Model\n\n## Testing function locally\n```\nserverless invoke local -f function_name\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanieldacosta%2Fml-serverless-deploy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanieldacosta%2Fml-serverless-deploy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanieldacosta%2Fml-serverless-deploy/lists"}