{"id":20408745,"url":"https://github.com/feast-dev/feast-aws-credit-scoring-tutorial","last_synced_at":"2025-04-12T15:40:59.901Z","repository":{"id":47269975,"uuid":"400041819","full_name":"feast-dev/feast-aws-credit-scoring-tutorial","owner":"feast-dev","description":"Feast AWS guide using Redshift / Spectrum / DynamoDB to build a credit scoring model","archived":false,"fork":false,"pushed_at":"2021-10-27T06:15:24.000Z","size":23493,"stargazers_count":63,"open_issues_count":2,"forks_count":30,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-26T10:11:18.651Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/feast-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-08-26T04:40:24.000Z","updated_at":"2025-03-01T14:11:01.000Z","dependencies_parsed_at":"2022-09-16T09:02:34.371Z","dependency_job_id":null,"html_url":"https://github.com/feast-dev/feast-aws-credit-scoring-tutorial","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feast-dev%2Ffeast-aws-credit-scoring-tutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feast-dev%2Ffeast-aws-credit-scoring-tutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feast-dev%2Ffeast-aws-credit-scoring-tutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feast-dev%2Ffeast-aws-credit-scoring-tutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/feast-dev","download_url":"https://codeload.github.com/feast-dev/feast-aws-credit-scoring-tutorial/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248590801,"owners_count":21129891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T05:36:01.344Z","updated_at":"2025-04-12T15:40:59.872Z","avatar_url":"https://github.com/feast-dev.png","language":"Python","readme":"# Real-time Credit Scoring with Feast on AWS\n\n## Overview\n\n![credit-score-architecture@2x](https://user-images.githubusercontent.com/6728866/132927464-5c9e9e05-538c-48c5-bc16-94a6d9d7e57b.jpg)\n\nThis tutorial demonstrates the use of Feast as part of a real-time credit scoring application.\n* The primary training dataset is a loan table. This table contains historic loan data with accompanying features. The dataset also contains a target variable, namely whether a user has defaulted on their loan.\n* Feast is used during training to enrich the loan table with zipcode and credit history features from a S3 files. The S3 files are queried through Redshift.\n* Feast is also used to serve the latest zipcode and credit history features for online credit scoring using DynamoDB.\n\n## Requirements\n\n* Terraform (v1.0 or later)\n* AWS CLI (v2.2 or later)\n\n## Setup\n\n### Setting up Redshift and S3\n\nFirst we will set up your data infrastructure to simulate a production environment. We will deploy Redshift, an S3 \nbucket containing our zipcode and credit history parquet files, IAM roles and policies for Redshift to access S3, and create a \nRedshift table that can query the parquet files. \n\nInitialize Terraform\n```\ncd infra\nterraform init\n```\n\nMake sure the Terraform plan looks good\n```\nterraform plan -var=\"admin_password=thisISyourPassword1\"\n```\n\nDeploy your infrastructure\n```\nterraform apply -var=\"admin_password=thisISyourPassword1\"\n```\n\nOnce your infrastructure is deployed, you should see the following outputs from Terraform\n```\nredshift_cluster_identifier = \"my-feast-project-redshift-cluster\"\nredshift_spectrum_arn = \"arn:aws:iam::\u003cAccount\u003e:role/s3_spectrum_role\"\ncredit_history_table = \"credit_history\"\nzipcode_features_table = \"zipcode_features\"\n```\n\nNext we create a mapping from the Redshift cluster to the external catalog\n```\naws redshift-data execute-statement \\\n    --region us-west-2 \\\n    --cluster-identifier [SET YOUR redshift_cluster_identifier HERE] \\\n    --db-user admin \\\n    --database dev --sql \"create external schema spectrum from data catalog database 'dev' iam_role \\\n    '[SET YOUR redshift_spectrum_arn here]' create external database if not exists;\"\n```\n\nTo see whether the command was successful, please run the following command (substitute your statement id)\n```\naws redshift-data describe-statement --id [SET YOUR STATEMENT ID HERE]\n``` \n\nYou should now be able to query actual zipcode features by executing the following statement\n```\naws redshift-data execute-statement \\\n    --region us-west-2 \\\n    --cluster-identifier [SET YOUR redshift_cluster_identifier HERE] \\\n    --db-user admin \\\n    --database dev --sql \"SELECT * from spectrum.zipcode_features LIMIT 1;\"\n```\nwhich should print out results by running\n```\naws redshift-data get-statement-result --id [SET YOUR STATEMENT ID HERE]\n```\n\nReturn to the root of the credit scoring repository\n```\ncd ..\n```\n\n### Setting up Feast\n\nInstall Feast using pip\n\n```\npip install feast\n```\n\nWe have already set up a feature repository in [feature_repo/](feature_repo/). It isn't necessary to create a new\nfeature repository, but it can be done using the following command\n```\nfeast init -t aws feature_repo # Command only shown for reference.\n```\n\nSince we don't need to `init` a new repository, all we have to do is configure the \n[feature_store.yaml/](feature_repo/feature_store.yaml) in the feature repository. Please set the fields under\n`offline_store` to the configuration you have received when deploying your Redshift cluster and S3 bucket.\n\nDeploy the feature store by running `apply` from within the `feature_repo/` folder\n```\ncd feature_repo/\nfeast apply\n```\n```\nRegistered entity dob_ssn\nRegistered entity zipcode\nRegistered feature view credit_history\nRegistered feature view zipcode_features\nDeploying infrastructure for credit_history\nDeploying infrastructure for zipcode_features\n```\n\nNext we load features into the online store using the `materialize-incremental` command. This command will load the\nlatest feature values from a data source into the online store.\n\n```\nCURRENT_TIME=$(date -u +\"%Y-%m-%dT%H:%M:%S\")\nfeast materialize-incremental $CURRENT_TIME\n```\n\nReturn to the root of the repository\n```\ncd ..\n```\n\n## Train and test the model\n\nFinally, we train the model using a combination of loan data from S3 and our zipcode and credit history features from Redshift\n(which in turn queries S3), and then we test online inference by reading those same features from DynamoDB \n\n```\npython run.py\n```\nThe script should then output the result of a single loan application\n```\nloan rejected!\n```\n\n## Interactive demo (using Streamlit)\n\nOnce the credit scoring model has been trained it can be used for interactive loan applications using Streamlit:\n\nSimply start the Streamlit application\n```\nstreamlit run streamlit_app.py\n```\nThen navigate to the URL on which Streamlit is being served. You should see a user interface through which loan applications can be made:\n\n![Streamlit Loan Application](streamlit.png)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeast-dev%2Ffeast-aws-credit-scoring-tutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffeast-dev%2Ffeast-aws-credit-scoring-tutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeast-dev%2Ffeast-aws-credit-scoring-tutorial/lists"}