{"id":19307430,"url":"https://github.com/artefactory/deploy-fastai-torchserve-aiplatform","last_synced_at":"2025-04-22T12:36:42.314Z","repository":{"id":103035110,"uuid":"327061340","full_name":"artefactory/deploy-fastai-torchserve-aiplatform","owner":"artefactory","description":"Deploy FastAI Trained PyTorch Model in TorchServe and Host in GCP's AI-Platform Prediciton.","archived":false,"fork":false,"pushed_at":"2021-03-31T10:23:59.000Z","size":246,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-02T00:02:25.218Z","etag":null,"topics":["ai-platform","gcp","pytorch","torchserve","ulmfit"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/artefactory.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-05T16:50:34.000Z","updated_at":"2023-05-13T01:08:45.000Z","dependencies_parsed_at":"2023-03-13T15:09:46.959Z","dependency_job_id":null,"html_url":"https://github.com/artefactory/deploy-fastai-torchserve-aiplatform","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fdeploy-fastai-torchserve-aiplatform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fdeploy-fastai-torchserve-aiplatform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fdeploy-fastai-torchserve-aiplatform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/artefactory%2Fdeploy-fastai-torchserve-aiplatform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/artefactory","download_url":"https://codeload.github.com/artefactory/deploy-fastai-torchserve-aiplatform/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250242098,"owners_count":21398066,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-platform","gcp","pytorch","torchserve","ulmfit"],"created_at":"2024-11-10T00:10:55.316Z","updated_at":"2025-04-22T12:36:42.299Z","avatar_url":"https://github.com/artefactory.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deploy FastAI Trained PyTorch Model in TorchServe and Host in GCP AI Platform Prediction\n\n- [Deploy FastAI Trained PyTorch Model in TorchServe and Host in GCP AI Platform Prediction](#deploy-fastai-torchserve-aiplatform)\n  - [Introduction](#introduction)\n  - [Getting Started with A FastAI Model](#getting-started-with-a-fastai-model)\n    - [Installation](#installation)\n  - [PyTorch Transfer Modeling from FastAI](#pytorch-transfer-modeling-from-fastai)\n    - [Export Model Weights from FastAI](#export-model-weights-from-fastai)\n    - [PyTorch Model from FastAI Source Code](#pytorch-model-from-fastai-source-code)\n    - [Weights Transfer](#weights-transfer)\n  - [Deployment to TorchServe](#deployment-to-torchserve)\n    - [Custom Handler](#custom-handler)\n      - [`initialize`](#initialize)\n      - [`preprocess`](#preprocess)\n      - [`inference`](#inference)\n      - [`postprocess`](#postprocess)\n    - [TorchServe in Action](#torchserve-in-action)\n  - [Deployment to GCP AI Platform Prediction](#deployment-to-gcp-aiplatform-prediction)\n    - [Getting Started with GCP AI Platform Prediction](#getting-started-with-gcp-aiplatform-prediction)\n    - [Real-time Inference with Python](#real-time-inference-with-python)\n  - [Conclusion](#conclusion)\n  - [Reference](#reference)\n\n## Introduction\n\nOver the past few years, NLP has had a huge hype of transfer learning. [FastAI](https://www.fast.ai/) is a deep learning library which provides practitioners with high-level components that can quickly and easily provide state-of-the-art results in standard deep learning domains.. It has become one of the most cutting-edge open-source deep learning framework and the go-to choice for many machine learning use cases based on [PyTorch](https://pytorch.org/).\n\nRecently, AWS developed *[TorchServe](https://github.com/pytorch/serve)* in partnership with Facebook. TorchServe makes it easy to deploy PyTorch models at scale in production environments. It removes the heavy lifting of developing your own client server architecture.\n\nMeanwhile, GCP AI Platform Prediction has been a fully managed  [cost-effective](https://cloud.google.com/ai-platform/prediction/pricing) service that allows users to make real-time inferences via a REST API, and save Data Scientists and Machine Learning Engineers from managing their own server instances.\n\nIn this repository we demonstrate how to deploy a FastAI trained PyTorch model in TorchServe eager mode and host it in AI Platform Prediction.\n\n\n## 1 - Installation\n\nThe first step is to install FastAI package, which is covered in its [Github](https://github.com/fastai/fastai) repository.\n\n\u003e If you're using Anaconda then run:\n\u003e ```python\n\u003e conda install -c fastai -c pytorch -c anaconda fastai gh anaconda\n\u003e ```\n\u003e ...or if you're using miniconda) then run:\n\u003e ```python\n\u003e conda install -c fastai -c pytorch fastai\n\u003e ```\n\nFor other installation options, please refer to the FastAI documentation.\n\n## 2 - Reusing fastai model in pytorch\n\n### Export Model Weights from FastAI\n\nFirst, restore the FastAI learner from the export pickle at the last Section, and save its model weights with PyTorch.\n\n#### Text version\n```python\nfrom fastai.text import load_learner\nfrom fastai.text.learner import _get_text_vocab, get_c\n\nimport torch\n\nVOCABZ_SZ = len(_get_text_vocab(dls)) #dls is the dataloader you used for training\nN_CLASSES = get_c(dls)\nCONFIG = awd_lstm_clas_config.copy()\n\nlearn = load_learner(\"fastai_cls.pkl\")\ntorch.save(learn.model.state_dict(), \"fastai_cls_weights.pth\")\n\ntext = \"This was a very good movie\"\npred_fastai = learn.predict(text)\npred_fastai\n\u003e\u003e\u003e\n(Category tensor(1), tensor(1), tensor([0.0036, 0.9964]))\n```\n\nOnce you got the model weights, put the .pth file in model/text or model/image.\n\n#### Image version\n```python\nfrom fastai.vision.all import get_c\n\nIMAGE_SIZE = dls.one_batch()[0].shape[-2:] #dls is the dataloader you used for training\nN_CLASSES = get_c(dls)\n\nimage_path = \"street_view_of_a_small_neighborhood.png\"\npred_fastai = learn.predict(image_path)\npred_fastai[0].numpy()\n\u003e\u003e\u003e\narray([[26, 26, 26, ...,  4,  4,  4],\n       [26, 26, 26, ...,  4,  4,  4],\n       [26, 26, 26, ...,  4,  4,  4],\n       ...,\n       [17, 17, 17, ..., 30, 30, 30],\n       [17, 17, 17, ..., 30, 30, 30],\n       [17, 17, 17, ..., 30, 30, 30]])\n```\n### PyTorch Model from FastAI\n\nNext, we need to define the model in pure PyTorch. In [Jupyter](https://jupyter.org/) notebook, one can investigate the FastAI source code by adding `??` in front of a function name. \n#### Text version\n```python\nfrom fastai.text.models.core import get_text_classifier\nfrom fastai.text.all import AWD_LSTM\n\nmodel_torch = get_text_classifier(AWD_LSTM, VOCABZ_SZ, N_CLASSES, config=CONFIG)\n```\n\nThe important thing here is that get_text_classifier fastai function outputs a torch.nn.modules.module.Module which therefore is a pure PyTorch object.\n\n#### Image version\n\n```python\nfrom fastai.vision.learner import create_unet_model\nfrom fastai.vision.models import resnet50\n\n\nmodel_torch = create_unet_model(resnet50, N_CLASSES, IMAGE_SIZE)\n```\n### Weights Transfer\n\nNow initialize the PyTorch model, load the saved model weights, and transfer that weights to the PyTorch model.\n\n```python\nstate = torch.load(\"fastai_cls_weights.pth\")\nmodel_torch.load_state_dict(state)\nmodel_torch.eval()\n```\n\nIf we take one sample, transform it, and pass it to the `model_torch`, we shall get an identical prediction result as FastAI's.\n\n### Preprocessing inputs\n\n#### Text version\n```python\nimport torch\nfrom fastai.text.core import Tokenizer, SpacyTokenizer\nfrom fastai.text.data import Numericalize\n\nexample = \"Hello, this is a test.\"\n\ntokenizer = Tokenizer(\n    tok=SpacyTokenizer(\"en\")\n)\nnumericalizer = Numericalize(vocab=vocab)\n\nexample_processed = numericalizer(tokenizer(example))\n\u003e\u003e\u003e\ntensor([ 4,  7, 26, 29, 16, 72, 69, 31])\n\ninputs = example_processed.resize(1, len(example_processed))\noutputs = model_torch.forward(inputs)[0]\npreds = torch.softmax(outputs, dim=-1) #You can use any activation function you need\n\u003e\u003e\u003e\ntensor([[0.0036, 0.9964]], grad_fn=\u003cSoftmaxBackward\u003e)\n```\n#### Image version\n\n```python\nfrom torchvision import transforms\nfrom PIL import Image\nimport numpy as np\n\nimage_path = \"street_view_of_a_small_neighborhood.png\"\n\nimage = Image.open(image_path).convert(\"RGB\")\nimage_tfm = transforms.Compose(\n    [\n        transforms.Resize((96, 128)),\n        transforms.ToTensor(),\n        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),\n    ]\n)\n\nx = image_tfm(image).unsqueeze_(0)\n\n# inference on CPU\nraw_out = model_torch_rep(x)\nraw_out.shape\n\u003e\u003e\u003e torch.Size([1, 32, 96, 128])\n\npred_res = raw_out[0].argmax(dim=0).numpy().astype(np.uint8)\npred_res\n\u003e\u003e\u003e\narray([[26, 26, 26, ...,  4,  4,  4],\n       [26, 26, 26, ...,  4,  4,  4],\n       [26, 26, 26, ...,  4,  4,  4],\n       ...,\n       [17, 17, 17, ..., 30, 30, 30],\n       [17, 17, 17, ..., 30, 30, 30],\n       [17, 17, 17, ..., 30, 30, 30]], dtype=uint8)\n```\n\n## 3-  Deployment to TorchServe\n\nIn this section we deploy the PyTorch model to TorchServe. For installation, please refer to TorchServe [Github](https://github.com/pytorch/serve) Repository.\n\nOverall, there are mainly 3 steps to use TorchServe:\n\n1. Archive the model into `*.mar`.\n2. Start the `torchserve`.\n3. Call the API and get the response.\n\nIn order to archive the model, at least 2 files are needed in our case:\n\n1. PyTorch model weights `fastai_cls_weights.pth`.\n2. TorchServe custom handler.\n\n### Custom Handler\n\nAs shown in `/deployment/handler.py`, the TorchServe handler accept `data` and `context`. In our example, we define another helper Python class with 4 instance methods to implement: `initialize`, `preprocess`, `inference` and `postprocess`.\n\nNow it's ready to setup and launch TorchServe.\n\n### TorchServe in Action\n\nStep 1: Archive the model PyTorch\n\n```bash\n\u003e\u003e\u003e torch-model-archiver \\\n  --model-name=fastai_model \\\n  --version=1.0 \\\n  --serialized-file=/home/model-server/fastai_v2_cls_weights.pth \\\n  --extra-files=/home/model-server/config.py,/home/model-server/vocab.json\n  --handler=/home/model-server/handler.py \\\n  --export-path=/home/model-server/model-store/\n```\n\nStep 2: Serve the Model\n\n```bash\n\u003e\u003e\u003e torchserve --start --ncs --model-store model_store --models fastai_model.mar\n```\n\nStep 3: Call API and Get the Response (here we use [httpie](https://httpie.org/)). For a complete response see `sample/sample_output.txt` at [here](sample/sample_output.txt).\n\n```bash\n\u003e\u003e\u003e time http POST http://127.0.0.1:8080/predictions/fastai_model/ @input.txt\n\nHTTP/1.1 200\nCache-Control: no-cache; no-store, must-revalidate, private\nExpires: Thu, 01 Jan 1970 00:00:00 UTC\nPragma: no-cache\nconnection: keep-alive\ncontent-length: 131101\nx-request-id: 96c25cb1-99c2-459e-9165-aa5ef9e3a439\n\n{\n  \"Categories\": \"1\",\n  \"Tensor\": [\n    0.0036,\n    0.9964\n  ]\n}\n\nreal    0m0.979s\nuser    0m0.280s\nsys     0m0.039s\n```\n\nThe first call would have longer latency due to model weights loading defined in `initialize`, but this will be mitigated from the second call onward.\n\n## 4- Deployment to AI Platform Prediction\n\nIn this section we deploy the FastAI trained Scene Segmentation PyTorch model with TorchServe in GCP AI Platform Prediction using customized Docker image. or more details about GCP AI Platform Prediction routines using custom containers please refer to [here](https://cloud.google.com/ai-platform/prediction/docs/use-custom-container).\n\n### Getting Started with GCP AI Platform Prediction\n\nThere are 3 steps to host a model on AI Platform Prediction with TorchServe:\n\n1. Create a new model on AI Platform. Be careful to choose a model with a regional endpoint to have the option to use a custom container.\n```bash\ngcloud beta ai-platform models create $MODEL_NAME \\\n  --region=$REGION \\\n  --enable-logging \\\n  --enable-console-logging\n```\n2. Build a docker image of the torchserve API package and push it to a container registry following the [custom container requirements](https://cloud.google.com/ai-platform/prediction/docs/custom-container-requirements). Be careful to create a regional artifact registry repository if it's doesn't already exist\n```bash\ngcloud beta artifacts repositories create $ARTIFACT_REGISTRY_NAME \\\n --repository-format=docker \\\n --location=$REGION\n ```\n```bash\n docker build -t $REGION-docker.pkg.dev/$GCP_PROJECT_ID/$ARTIFACT_REGISTRY_NAME/$DOCKER_NAME:$DOCKER_TAG . -f TextDockerfile\n```\n```bash\n docker push $REGION-docker.pkg.dev/$GCP_PROJECT_ID/$ARTIFACT_REGISTRY_NAME/$DOCKER_NAME:$DOCKER_TAG\n```\n\n3. [Optional] Run your docker locally and try to send a prediction\n\n```bash\ndocker run -d -p 8080:8080 --name local_imdb  $REGION-docker.pkg.dev/$GCP_PROJECT_ID/$ARTIFACT_REGISTRY_NAME/$DOCKER_NAME:$DOCKER_TAG\n```\n```bash\ncurl -X POST -H \"Content-Type: application/json\" -d '[\"this was a bad movie\"]' 127.0.0.1:8080/predictions/fastai_model\n```\nIf everything is working okay, you should receive a response from the server in your console\n```bash\n[\n  {\n    \"Categories\": \"0\",\n    \"Tensor\": [\n      0.9999990463256836,\n      9.371918849865324e-07\n    ]\n  }\n]\n```\n\n4. Create AI Platform version model using the docker image of the torchserve API package.\n```bash\ngcloud beta ai-platform versions create $VERSION_NAME  --region=$REGION --model=$MODEL_NAME --image=$REGION-docker.pkg.dev/$GCP_PROJECT_ID/$ARTIFACT_REGISTRY_NAME/$DOCKER_NAME:$DOCKER_TAG  --ports=8080   --health-route=/ping   --predict-route=/predictions/fastai_model\n```\n\n\n## Conclusion\n\nThis repository presented an end-to-end demonstration of deploying a FastAI text classifier model on GCP AI Platform Prediction. It allows a user to serve a fastai model without going through the self-maintaining effort to build and manage a customized inference server. This repository was inspired by another project that aimed to deploy a fastai image classifier on AWS SageMaker Inference Endpoint [here](https://github.com/aws-samples/amazon-sagemaker-endpoint-deployment-of-fastai-model-with-torchserve).\n\nIf you have questions please create an issue or submit Pull Request on the [GitHub](https://github.com/artefactory/deploy-fastai-torchserve-aiplatform) repository.\n\n## Reference\n\n- [fast.ai · Making neural nets uncool again](https://www.fast.ai/)\n- [TORCHSERVE](https://pytorch.org/serve/)\n- [Deploying PyTorch models for inference at scale using TorchServe host in AWS SageMaker](https://github.com/aws-samples/amazon-sagemaker-endpoint-deployment-of-fastai-model-with-torchserve)\n- [Deploying PyTorch models for inference at scale using TorchServe](https://aws.amazon.com/blogs/machine-learning/deploying-pytorch-models-for-inference-at-scale-using-torchserve/)\n- [Serving PyTorch models in production with the Amazon SageMaker native TorchServe integration](https://aws.amazon.com/blogs/machine-learning/serving-pytorch-models-in-production-with-the-amazon-sagemaker-native-torchserve-integration/)\n- [Building, training, and deploying fastai models with Amazon SageMaker](https://aws.amazon.com/blogs/machine-learning/building-training-and-deploying-fastai-models-with-amazon-sagemaker/)\n- [Running TorchServe on Amazon Elastic Kubernetes Service](https://aws.amazon.com/blogs/opensource/running-torchserve-on-amazon-elastic-kubernetes-service/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartefactory%2Fdeploy-fastai-torchserve-aiplatform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fartefactory%2Fdeploy-fastai-torchserve-aiplatform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fartefactory%2Fdeploy-fastai-torchserve-aiplatform/lists"}