{"id":21761950,"url":"https://github.com/ashutoshdongare/huggingface-model-serving","last_synced_at":"2025-04-13T13:26:38.843Z","repository":{"id":201566001,"uuid":"479754042","full_name":"AshutoshDongare/HuggingFace-Model-Serving","owner":"AshutoshDongare","description":"Quick and easy tutorial to serve HuggingFace sentiment analysis model using torchserve","archived":false,"fork":false,"pushed_at":"2022-04-09T19:00:00.000Z","size":44,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-27T04:33:15.574Z","etag":null,"topics":["bert-models","deployment","devops","huggingface","inference","model","rest-api","sentiment-analysis","torchserve","tutorial"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AshutoshDongare.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-04-09T14:41:32.000Z","updated_at":"2025-03-27T00:19:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"becf8ebc-6251-4f9e-82af-5eede39d7608","html_url":"https://github.com/AshutoshDongare/HuggingFace-Model-Serving","commit_stats":null,"previous_names":["ashutoshdongare/huggingface-model-serving"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AshutoshDongare%2FHuggingFace-Model-Serving","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AshutoshDongare%2FHuggingFace-Model-Serving/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AshutoshDongare%2FHuggingFace-Model-Serving/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AshutoshDongare%2FHuggingFace-Model-Serving/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AshutoshDongare","download_url":"https://codeload.github.com/AshutoshDongare/HuggingFace-Model-Serving/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248719566,"owners_count":21150759,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert-models","deployment","devops","huggingface","inference","model","rest-api","sentiment-analysis","torchserve","tutorial"],"created_at":"2024-11-26T12:10:24.980Z","updated_at":"2025-04-13T13:26:38.819Z","avatar_url":"https://github.com/AshutoshDongare.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HuggingFace-Model-Serving\nQuick and easy tutorial to serve HuggingFace sentiment analysis model using torchserve\n\nFull explanation of all possible configurations to serve any type of model can be found at [Torchserve Github](https://github.com/pytorch/serve)  \n\nHowever, This tutorial can help you to get started quickly on serving your models to production.\nwe will be using a pretrained huggingface model ``` distilbert-base-uncased-finetuned-sst-2-english ``` for serving.\n\nThis model is finetuned for Text Classification (sentiment analysis) task. It is available on [HuggingFace](https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english)\n\n## Dependencies Installation \n\nWe will require following components available for serving. It is a good idea to create and activate a python virtual environment with name of your choice before installing python dependencies. We may want to call it \"torchserve\" as an environment.\n\n- [JDK 11](https://www.oracle.com/in/java/technologies/javase/jdk11-archive-downloads.html) \n  You may need to sign up to oracle to download archived version of JDK to be able to download and install\n  Torchserve uses JDK for HTTP server support.\n\n- [pytorch](https://github.com/pytorch/serve/blob/master/README.md#serve-a-model)\n  Install torchserve and related components using below command \n\n```\npip install torchserve torch-model-archiver torch-workflow-archiver\n```\n\n-[Transformers](https://huggingface.co/docs/transformers/index)\nAs we will be serving Transformer model, we will require to install Transformers using following command\n```\npip install transformers\n```\n\n## Setup\n\nWe will first download the transformer model locally, then archive it to model archive file (.mar) and serve it using Torch Serve\n\n- Step 1 - Lets create and change directory to a local folder named \"sentiment_deployment\".   \n\n- Step 2 - Clone or download and extract serve repo to your machine from [Torch Serve repo](https://github.com/pytorch/serve). we  will require a couple of files from this repo. this will give you \"serve-master\" directory with all the artifacts. \n\n\u003cspan style=\"color:red\"\u003eIf you do not want to download all the files/repo from torch serve, you can download my repo that has only required files. In that case you can skip the steps 3,4 \u0026 5 and directly continue with step 6\u003c/span\u003e \n\n- Step 3 - copy following files from serve-master folder of serve repo to sentiment_deployment folder.\n``` \n/serve-master/examples/Huggingface_Transformers/setup_config.json\n/serve-master/examples/Huggingface_Transformers/Download_Transformer_models.py\n/serve-master/examples/Huggingface_Transformers/Transformer_handler_generalized.py\n/serve-master/examples/Huggingface_Transformers/Seq_classification_artifacts/index_to_name.json\n\n```\n\n\n- Step 4 - Edit ```setup_config.json``` to have following content. \n```\n{\n \"model_name\":\"distilbert-base-uncased-finetuned-sst-2-english\",\n \"mode\":\"sequence_classification\",\n \"do_lower_case\":true,\n \"num_labels\":\"2\",\n \"save_mode\":\"pretrained\",\n \"max_length\":\"128\",\n \"captum_explanation\":false,\n \"embedding_name\": \"distilbert\",\n \"FasterTransformer\":false,\n \"model_parallel\":false\n}\n\n```\n- Step 5 - Edit ```index_to_name.json``` to have following content. \n```\n{\n \"0\":\"Negative\",\n \"1\":\"Positive\"\n}\n\n```\n\n- Step 6 - Let's now download Transformer model using following command\n```\npython Download_Transformer_models.py\n```\nThis will create a new folder ```Transformer_model``` under current directory \u0026 download transformer model mentioned in setup_config.json and  and all required artifacts\n\n- Step 6 - Let's create Model Archive (.mar) using following command. Please ensure that you have all the files at correct places. If you have followed the steps correctly then these files should be in correct places. \n```\ntorch-model-archiver --model-name distilBERTSeqClassification --version 1.0 --serialized-file Transformer_model/pytorch_model.bin --handler ./Transformer_handler_generalized.py --extra-files \"Transformer_model/config.json,./setup_config.json,./index_to_name.json\"\n```\n\n- Step 7 - Create a directory named ```model_store``` under current directory and move your new archived model file to this folder\n```\nmkdir model_store\nmv distilBERTSeqClassification.mar model_store/ \n\n```\n- Step 8 - This is the final step in serving the model. We will run torchserve as below\n```\ntorchserve --start --model-store model_store --models sentiments=distilBERTSeqClassification.mar --ncs\n``` \nIf everything goes well, you should see a message like below in the terminal log\n\n\u003e ***Transformer model from path c:\\..\\.. loaded successfully***\n\nThis confirms that you are now serving pretrained Huggingface sentiment analysis model as a REST API\n\n\n# Running inference  \n\nYou can test the model inference that we just hosted as REST API\n\nQuick way to do this is run below Curl command. Before you run below command you need to create a simple text file named ```sentiment.txt``` with one sentence that will be given for inference.\n```\ncurl -X POST http://127.0.0.1:8080/predictions/sentiments -T sentiment.txt\n```\nYou may as well write a simple python REST client to check model inference. I have added a simple REST API client code in this repo. In a new terminal with \"torchserve\" virtual environment activated,  Please run following client code to get model inference  \n\n ```\npython /client/Sentiment_analysis_client.py\n \n``` \n\nThis should return the sentiment inference as Positive or Negative. \n\nSince this server is running locally and we are only running one sentence not the batch inference, the response time is extremely fast.\n\n \n## Citations\n \nThis tutorial is curated list of steps from Torch Serve Github documentation. \n \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashutoshdongare%2Fhuggingface-model-serving","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fashutoshdongare%2Fhuggingface-model-serving","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashutoshdongare%2Fhuggingface-model-serving/lists"}