{"id":20222288,"url":"https://github.com/ronlek/alexa-prize-coding-challenge","last_synced_at":"2026-05-11T17:39:12.300Z","repository":{"id":64620927,"uuid":"576846688","full_name":"RonLek/Alexa-Prize-Coding-Challenge","owner":"RonLek","description":"Submission repository for the Alexa Prize Coding Challenge.","archived":false,"fork":false,"pushed_at":"2022-12-11T18:02:27.000Z","size":113,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-03T12:23:02.079Z","etag":null,"topics":["alexa","aws","intent-classification"],"latest_commit_sha":null,"homepage":"http://apcc-asg-1-1665300744.us-east-1.elb.amazonaws.com/predict","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RonLek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-12-11T07:01:22.000Z","updated_at":"2023-08-24T19:19:05.000Z","dependencies_parsed_at":"2022-12-19T04:00:26.459Z","dependency_job_id":null,"html_url":"https://github.com/RonLek/Alexa-Prize-Coding-Challenge","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/RonLek/Alexa-Prize-Coding-Challenge","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RonLek%2FAlexa-Prize-Coding-Challenge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RonLek%2FAlexa-Prize-Coding-Challenge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RonLek%2FAlexa-Prize-Coding-Challenge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RonLek%2FAlexa-Prize-Coding-Challenge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RonLek","download_url":"https://codeload.github.com/RonLek/Alexa-Prize-Coding-Challenge/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RonLek%2FAlexa-Prize-Coding-Challenge/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32906506,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-11T17:09:15.040Z","status":"ssl_error","status_checked_at":"2026-05-11T17:08:45.420Z","response_time":120,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alexa","aws","intent-classification"],"created_at":"2024-11-14T06:56:01.466Z","updated_at":"2026-05-11T17:39:12.283Z","avatar_url":"https://github.com/RonLek.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Alexa Prize Coding Challenge\n\nThe repository is a submission for the Alexa Prize Coding Challenge. Description of the challenge can be found [here](https://rentry.co/alexa-prize-coding-test).\n\n```\nSubmitter: Rohan Lekhwani\nChallenge Received: December 8, 2022\nChallenge Completed: December 11, 2022\n```\n\n## Table of Contents\n\n- [Alexa Prize Coding Challenge](#alexa-prize-coding-challenge)\n  - [Table of Contents](#table-of-contents)\n  - [How to Run](#how-to-run)\n  - [Dataset](#dataset)\n  - [Data Preprocessing](#data-preprocessing)\n  - [Model Training](#model-training)\n    - [Hyperparameters](#hyperparameters)\n    - [Metrics](#metrics)\n  - [Deployment](#deployment)\n    - [Creating an AMI](#creating-an-ami)\n    - [Load Balancing, Target Groups and Auto Scaling Groups](#load-balancing-target-groups-and-auto-scaling-groups)\n  - [Development](#development)\n  - [Possible Improvements](#possible-improvements)\n    - [Model Improvements](#model-improvements)\n    - [Deployment Improvements](#deployment-improvements)\n  - [References](#references)\n\n## How to Run\n\nThe URL for the deployed model is: `http://apcc-asg-1-1665300744.us-east-1.elb.amazonaws.com/predict`. Make a `POST` request with the text to be classified within the body.\n\n```bash\ncurl -X POST -H \"Content-Type: application/json\" -d '{\"text\": \"Will I be making my own pasta for this recipe or using the packaged store bought kind?\"}' http://apcc-asg-1-1665300744.us-east-1.elb.amazonaws.com/predict\n# {\"intent\":\"ask_question_ingredients_tools\",\"score\":\"0.62276465\"}\n```\n\n## Dataset\n\nThe [Wizards of Tasks](https://registry.opendata.aws/wizard-of-tasks/) dataset has been used to train an intent classification model. The dataset comprises of two domains: `cooking` and `diy`. During training we only make use of the `text` and `intent` attributes of the samples. The `cooking` and `diy` domains have been combined to create a join dataset.\n\nThere are a total of 11 labels combined across the two domains:\n\n```python\nlabels = ['answer_question_external_fact', 'answer_question_recipe_steps',\n       'ask_question_ingredients_tools', 'ask_question_recipe_steps',\n       'ask_student_question', 'chitchat', 'misc', 'request_next_step',\n       'return_list_ingredients_tools', 'return_next_step', 'stop']\n```\n\n## Data Preprocessing\n\nThe splits into train, test and validation have been done in accordance with the split `data_split` label within the dataset.\n\nA few samples have `null` values for `text` or `intent` these have been removed from the split sets. The JSON data is converted to a Pandas dataframe for easier processing.\n\n## Model Training\n\nA pre-trained [BERT](https://arxiv.org/abs/1810.04805) model has been used to train on the Wizards of Tasks dataset. [Tensorflow Hub](https://tfhub.dev/google/collections/bert/1) was used to obtain a pre-trained BERT model. More specifically, the deployment makes use of the Small BERT model `small_bert/bert_en_uncased_L-2_H-256_A-4`. This model has 2 hidden layers (transformer blocks), a hidden size of 256 and 4 attention heads. More information about this model can be found [here](https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-256_A-4/2). `intent_classifier.ipynb` contains the model training code.\n\nTo transform the text inputs numeric token ids and to arrange them in Tensors before being input to BERT we make use of pre-processing models. TensorFlow Hub provides a matching preprocessing model for each of the BERT models, which implements this transformation using TF ops from the TF.text library.\n\nA customized model has been built upon the pre-trained Small BERT model. It consists of the preprocessing layer, the BERT encoder, dropout layer and a dense layer to predict the classification output. The model summary can be seen below:\n\n```python\nModel: \"model_2\"\n__________________________________________________________________________________________________\n Layer (type)                   Output Shape         Param #     Connected to                     \n==================================================================================================\n text (InputLayer)              [(None,)]            0           []                               \n                                                                                                  \n preprocessing (KerasLayer)     {'input_type_ids':   0           ['text[0][0]']                   \n                                (None, 128),                                                      \n                                 'input_word_ids':                                                \n                                (None, 128),                                                      \n                                 'input_mask': (Non                                               \n                                e, 128)}                                                          \n                                                                                                  \n BERT_encoder (KerasLayer)      {'encoder_outputs':  9591041     ['preprocessing[0][0]',          \n                                 [(None, 128, 256),               'preprocessing[0][1]',          \n                                 (None, 128, 256)],               'preprocessing[0][2]']          \n                                 'sequence_output':                                               \n                                 (None, 128, 256),                                                \n                                 'pooled_output': (                                               \n                                None, 256),                                                       \n                                 'default': (None,                                                \n                                256)}                                                             \n                                                                                                  \n dropout_2 (Dropout)            (None, 256)          0           ['BERT_encoder[0][3]']           \n                                                                                                  \n classifier (Dense)             (None, 11)           2827        ['dropout_2[0][0]']              \n                                                                                                  \n==================================================================================================\nTotal params: 9,593,868\nTrainable params: 9,593,867\nNon-trainable params: 1\n```\n\n### Hyperparameters\n\nThe CategoricalCrossentropy loss is used since the goal is a non-binary classification into intents. Adam optimizer with a learning rate of `1e-5` has been used while training. The model is trained in batches of 32 over 5 epochs.\n\n### Metrics\n\nAfter the final epoch the following are the metrics obtained with the current model\n\n```python\ntraining_loss: 0.7695\ncategorical_accuracy: 0.7319\nval_loss: 0.6979\nval_categorical_accuracy: 0.7587\ntest_loss: 0.7573978900909424\ntest_accuracy: 0.7222222089767456\n```\n\n## Deployment\n\n### Creating an AMI\n\nThe model obtained is saved to the Drive in an `.hd5` format. `server.py` is responsible for loading this model and starting a flask server on `PORT=5000`.\n\nA `t2.micro` EC2 instance is started with 1 vCPU and 1GiB memory. The repository is cloned and all dependencies are installed through the `requirements.txt` file within the instance.\n\n```bash\npip install -r requirements.txt\n```\n\nA systemd service `apcc.service` is created to run the flask server within `/etc/systemd/system`.\n\n```bash\n[Unit]\nDescription=APCC Service\nAfter=network.target\n\n[Service]\nUser=root\nWorkingDirectory=/home/ec2-user/apcc\nExecStart=/bin/python3 server.py\nRestart=always\n\n[Install]\nWantedBy=multi-user.target\n```\n\nThe service is started to verify correct deployment.\n\n```bash\nsudo systemctl start apcc.service\n```\n\nAn Amazon Machine Image (AMI) of the instance is created. Doing so enables spinning up ready-to-serve VMs within an autoscaler.\n\n### Load Balancing, Target Groups and Auto Scaling Groups\n\nA launch tempalte using the AMI created above is created to templatize new instance creations. An Auto Scaling Group is provision which makes use of this launch template. The following are capacity limits of the ASG in the current deployment.\n\n```txt\nDesired Capacity: 2\nMinimum Capacity: 1\nMaximum Capacity: 3\n```\n\nWith the above limits, two instances are always guaranteed to be running the application in two different zones which ensures scalability.\n\nA target group is created for port `5000` for the instances. The Internet-facing Application Load Balancer created next forwards its HTTP requests on `PORT=80` to this target group.\n\nThe following is the architecture diagram of the deployment.\n\n![architecture](./architecture.png)\n\n## Development\n\nClone the repo and `cd` into the `apcc` directory.\n\n```bash\ngit clone git@github.com:RonLek/Alexa-Prize-Coding-Challenge.git\ncd apcc\n```\n\nDownload a model to deploy from the [Google Drive](https://drive.google.com/drive/folders/17oGnAMfBkuxeXM_vI4qb3a9DZr2yPuqN?usp=share_link) and place it in the current directory. Make sure to change the name of the model to load in `server.py` correspondingly.\n\nInstall the dependencies using `pip` and start the server. If you are low on memory make use of the `--no-cache-dir` flag.\n\n```bash\nsudo pip3 install -r requirements.txt --no-cache-dir\nsudo python3 server.py\n```\n\n\u003e Note: `sudo` is required when serving on `PORT=80`.\n\nThe server should be up and running on `PORT=80`. You can test the server by sending a POST request to the server with the following command.\n\n```bash\ncurl -X POST -H \"Content-Type: application/json\" -d '{\"text\": \"Will I be making my own pasta for this recipe or using the packaged store bought kind?\"}' http://localhost:80/predict\n```\n\n## Possible Improvements\n\nThe following are some of the possible improvements that can be made to the model and the deployment strategy.\n\n### Model Improvements\n\n- From analysis it is seen that the model accuracy is not very high. This is due to the fact that the model size had to be constrained (larger models take up more memory to load) owing to the 1vCPU and 1GiB memory limit of the free tier EC2 instance. Larger models (like the uncased BERT, and 8 layer Small BERT models) were trained and the accuracy was seen to improve. These models can be found within the [Google Drive](https://drive.google.com/drive/folders/17oGnAMfBkuxeXM_vI4qb3a9DZr2yPuqN?usp=share_link).\n\n- The model accuracy is seen to improve linearly with the number of epochs. However owing to time constraints the model was trained for only 5 epochs. The model can be trained for more epochs to improve the accuracy.\n\n- The current model is a pre-trained BERT model with additional Dense and Dropout layers. A better model is bound to give better results on the dataset.\n\n### Deployment Improvements\n\n- The entire deployment can be containerized on EKS. Helm charts can be used to start a Kubernetes cluster and deploy the application on it. $0.10 per hour is charged for the EKS cluster, however since the challenge required using only free resources this was not done.\n\n- AWS SageMaker could be optionally use to train, test and validate the model along with deploying it for API consumption. SageMaker also handles scaling the resources dedicated to the model to handle increased traffic.\n\n- Currently instances are started in two different zones. This can be extended to all 6 zones to ensure high availability.\n\n- Although healthchecks are in place. Observability and monitoring alerts can be set up to ensure that the application is running smoothly.\n\n## References\n\nThe following papers and resources were used to complete the project.\n\n- Wizard of Tasks: A Novel Conversational Dataset for Solving Real-World Tasks in Conversational Settings by Jason Ingyu Choi, Saar Kuzi, Nikhita Vedula, Jie Zhao, Giuseppe Castellucci, Marcus Collins, Shervin Malmasi, Oleg Rokhlenko and Eugene Agichtein\n\n- Devlin, J., Chang, M.W., Lee, K. and Toutanova, K., 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.\n\n- AWS Documentation\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fronlek%2Falexa-prize-coding-challenge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fronlek%2Falexa-prize-coding-challenge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fronlek%2Falexa-prize-coding-challenge/lists"}