{"id":30224232,"url":"https://github.com/seldonio/cassava-example","last_synced_at":"2026-03-18T01:31:04.149Z","repository":{"id":160536400,"uuid":"635341420","full_name":"SeldonIO/cassava-example","owner":"SeldonIO","description":"Example mlserver and seldon deployment for a cassava leaf classifier","archived":false,"fork":false,"pushed_at":"2023-05-17T14:22:15.000Z","size":18947,"stargazers_count":10,"open_issues_count":0,"forks_count":7,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-14T13:13:25.847Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SeldonIO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-02T13:41:54.000Z","updated_at":"2023-07-26T11:33:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"ddbfadcf-f367-49c1-83a3-786eeb2c8b2e","html_url":"https://github.com/SeldonIO/cassava-example","commit_stats":null,"previous_names":["seldonio/cassava-example"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SeldonIO/cassava-example","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeldonIO%2Fcassava-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeldonIO%2Fcassava-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeldonIO%2Fcassava-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeldonIO%2Fcassava-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SeldonIO","download_url":"https://codeload.github.com/SeldonIO/cassava-example/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeldonIO%2Fcassava-example/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30640014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-18T00:09:27.587Z","status":"ssl_error","status_checked_at":"2026-03-18T00:09:26.123Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-14T13:07:42.360Z","updated_at":"2026-03-18T01:31:04.140Z","avatar_url":"https://github.com/SeldonIO.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deploying a Custom Tensorflow Model with MLServer and Seldon Core\n\n## Background\n\n### Intro\n\nThis tutorial walks through the steps required to take a python ML model from your machine to a production deployment on Kubernetes. More specifically we'll cover:\n- Running the model locally\n- Turning the ML model into an API\n- Containerizing the model\n- Storing the container in a registry\n- Deploying the model to Kubernetes (with Seldon Core)\n- Scaling the model\n\nThe tutorial comes with an accompanying video which you might find useful as you work through the steps:\n[![video_play_icon](img/video_play.png)](https://youtu.be/3bR25_qpokM)\n\nThe slides used in the video can be found [here](img/slides.pdf).\n\n### The Use Case\n\nFor this tutorial, we're going to use the [Cassava dataset](https://www.tensorflow.org/datasets/catalog/cassava) available from the Tensorflow Catalog. This dataset includes leaf images from the cassava plant. Each plant can be classified as either \"healthly\" or as having one of four diseases (Mosaic Disease, Bacterial Blight, Green Mite, Brown Streak Disease).\n\n![cassava_examples](img/cassava_examples.png)\n\nWe won't go through the steps of training the classifier. Instead, we'll be using a pre-trained one available on TensorFlow Hub. You can find the [model details here](https://tfhub.dev/google/cropnet/classifier/cassava_disease_V1/2). \n\n## Getting Set Up\n\nThe easiest way to run this example is to clone the repository. Once you've done that, you can just run:\n\n```Python\npip install -r requirements.txt\n```\n\nAnd it'll set you up with all the libraries required to run the code.\n\n## Running The Python App\n\nThe starting point for this tutorial is python script `app.py`. This is typical of the kind of python code we'd run standalone or in a jupyter notebook. Let's familiarise ourself with the code:\n\n```Python\nfrom helpers import plot, preprocess\nimport tensorflow as tf\nimport tensorflow_datasets as tfds\nimport tensorflow_hub as hub\n\n# Fixes an issue with Jax and TF competing for GPU\ntf.config.experimental.set_visible_devices([], 'GPU')\n\n# Load the model\nmodel_path = './model'\nclassifier = hub.KerasLayer(model_path)\n\n# Load the dataset and store the class names\ndataset, info = tfds.load('cassava', with_info=True)\nclass_names = info.features['label'].names + ['unknown']\n\n# Select a batch of examples and plot them\nbatch_size = 9\nbatch = dataset['validation'].map(preprocess).batch(batch_size).as_numpy_iterator()\nexamples = next(batch)\nplot(examples, class_names)\n\n# Generate predictions for the batch and plot them against their labels\npredictions = classifier(examples['image'])\npredictions_max = tf.argmax(predictions, axis=-1)\nprint(predictions_max)\nplot(examples, class_names, predictions_max)\n```\n\nFirst up, we're importing a couple of functions from our `helpers.py` file:\n- `plot` provides the visualisation of the samples, labels and predictions.\n- `preprocess` is used to resize images to 224x224 pixels and normalize the RGB values.\n\nThe rest of the code is fairly self-explanatory from the comments. We load the model and dataset, select some examples, make predictions and then plot the results.\n\nTry it yourself by running:\n\n```Bash\npython app.py\n```\n\nHere's what our setup currently looks like:\n![step_1](img/step_1.png)\n\n## Creating an API for The Model\n\nThe problem with running our code like we did earlier is that it's not accessible to anyone who doesn't have the python script (and all of it's dependencies). A good way to solve this is to turn our model into an API. \n\nTypically people turn to popular python web servers like [Flask](https://github.com/pallets/flask) or [FastAPI](https://github.com/tiangolo/fastapi). This is a good approach and gives us lots of flexibility but it also requires us to do a lot of the work ourselves. We need to impelement routes, set up logging, capture metrics and define an API schema among other things. A simpler way to tackle this problem is to use an inference server. For this tutorial we're going to use the open source [MLServer](https://github.com/SeldonIO/MLServer) framework. \n\nMLServer supports a bunch of [inference runtimes](https://mlserver.readthedocs.io/en/stable/runtimes/index.html) out of the box, but it also supports [custom python code](https://mlserver.readthedocs.io/en/stable/user-guide/custom.html) which is what we'll use for our Tensorflow model.\n\n### Setting Things Up\n\nIn order to get our model ready to run on MLServer we need to wrap it in a single python class with two methods, `load()` and `predict()`. Let's take a look at the code (found in `model/serve-model.py`):\n\n```Python\nfrom mlserver import MLModel\nfrom mlserver.codecs import decode_args\nimport numpy as np\nimport tensorflow as tf\nimport tensorflow_hub as hub\n\n# Define a class for our Model, inheriting the MLModel class from MLServer\nclass CassavaModel(MLModel):\n\n  # Load the model into memory\n  async def load(self) -\u003e bool:\n    tf.config.experimental.set_visible_devices([], 'GPU')\n    model_path = '.'\n    self._model = hub.KerasLayer(model_path)\n    self.ready = True\n    return self.ready\n\n  # Logic for making predictions against our model\n  @decode_args\n  async def predict(self, payload: np.ndarray) -\u003e np.ndarray:\n    # convert payload to tf.tensor\n    payload_tensor = tf.constant(payload)\n\n    # Make predictions\n    predictions = self._model(payload_tensor)\n    predictions_max = tf.argmax(predictions, axis=-1)\n\n    # convert predictions to np.ndarray\n    response_data = np.array(predictions_max)\n\n    return response_data\n```\n\nThe `load()` method is used to define any logic required to set up our model for inference. In our case, we're loading the model weights into `self._model`. The `predict()` method is where we include all of our prediction logic. \n\nYou may notice that we've slightly modified our code from earlier (in `app.py`). The biggest change is that it is now wrapped in a single class `CassavaModel`.\n\nThe only other task we need to do to run our model on MLServer is to specify a `model-settings.json` file:\n\n```Json\n{\n    \"name\": \"cassava\",\n    \"implementation\": \"serve-model.CassavaModel\"\n}\n```\n\nThis is a simple configuration file that tells MLServer how to handle our model. In our case, we've provided a name for our model and told MLServer where to look for our model class (`serve-model.CassavaModel`).\n\n### Serving The Model\n\nWe're now ready to serve our model with MLServer. To do that we can simply run:\n\n```bash\nmlserver start model/\n```\n\nMLServer will now start up, load our cassava model and provide access through both a REST and gRPC API.\n\n### Making Predictions Using The API\n\nNow that our API is up and running. Open a new terminal window and navigate back to the root of this repository. We can then send predictions to our api using the `test.py` file by running:\n\n```bash\npython test.py --local\n```\n\nOur setup has now evloved and looks like this:\n![step_2](img/step_2.png)\n\n## Containerizing The Model\n\n[Containers](https://en.wikipedia.org/wiki/Containerization_(computing)) are an easy way to package our application together with it's runtime and dependencies. More importantly, containerizing our model allows it to run in a variety of different environments. \n\n\u003e **Note:** you will need [Docker](https://www.docker.com/) installed to run this section of the tutorial. You'll also need a [docker hub](https://hub.docker.com/) account or another container registry.\n\nTaking our model and packaging it into a container manually can be a pretty tricky process and requires knowledge of writing Dockerfiles. Thankfully MLServer removes this complexity and provides us with a simple `build` command.\n\nBefore we run this command, we need to provide our dependencies in either a `requirements.txt` or a `conda.env` file. The requirements file we'll use for this example is stored in `model/requirements.txt`:\n\n```\ntensorflow==2.12.0\ntensorflow-hub==0.13.0\n```\n\n\u003e Notice that we didn't need to include `mlserver` in our requirements? That's because the builder image has mlserver included already.\n\nWe're now ready to build our container image using:\n\n```bash\nmlserver build model/ -t [YOUR_CONTAINER_REGISTRY]/[IMAGE_NAME]\n```\n\nMake sure you replace `YOUR_CONTAINER_REGISTRY` and `IMAGE_NAME` with your dockerhub username and a suitable name e.g. \"bobsmith/cassava\".\n\nMLServer will now build the model into a container image for us. We can check the output of this by running:\n\n```bash\ndocker images\n```\n\nFinally, we want to send this container image to be stored in our container registry. We can do this by running:\n\n```bash\ndocker push [YOUR_CONTAINER_REGISTRY]/[IMAGE_NAME]\n```\n\nOur setup now looks like this. Where our model has been packaged and sent to a container registry:\n![step_3](img/step_3.png)\n\n## Deploying to Kubernetes\n\nNow that we've turned our model into a production-ready API, containerized it and pushed it to a registry, it's time to deploy our model.\n\nWe're going to use a popular open source framework called [Seldon Core](https://github.com/seldonio/seldon-core) to deploy our model. Seldon Core is great because it combines all of the awesome cloud-native features we get from [Kubernetes](https://kubernetes.io/) but it also adds machine-learning specific features.\n\n*This tutorial assumes you already have a Seldon Core cluster up and running. If that's not the case, head over the [installation instructions](https://docs.seldon.io/projects/seldon-core/en/latest/nav/installation.html) and get set up first. You'll also need to install the `kubectl` command line interface.*\n\n### Creating the Deployment\n\nTo create our deployment with Seldon Core we need to create a small configuration file that looks like this:\n\n*You can find this file named `deployment.yaml` in the base folder of this tutorial's repository.*\n\n```yaml\napiVersion: machinelearning.seldon.io/v1\nkind: SeldonDeployment\nmetadata:\n  name: cassava\nspec:\n  protocol: v2\n  predictors:\n    - componentSpecs:\n        - spec:\n            containers:\n              - image: YOUR_CONTAINER_REGISTRY/IMAGE_NAME\n                name: cassava\n                imagePullPolicy: Always\n      graph:\n        name: cassava\n        type: MODEL\n      name: cassava\n```\n\nMake sure you replace `YOUR_CONTAINER_REGISTRY` and `IMAGE_NAME` with your dockerhub username and a suitable name e.g. \"bobsmith/cassava\".\n\nWe can apply this configuration file to our Kubernetes cluster just like we would for any other Kubernetes object using:\n\n```bash\nkubectl create -f deployment.yaml\n```\n\nTo check our deployment is up and running we can run:\n\n```bash\nkubectl get pods\n```\n\nWe should see `STATUS = Running` once our deployment has finalized.\n\n### Testing the Deployment\n\nNow that our model is up and running on a Kubernetes cluster (via Seldon Core), we can send some test inference requests to make sure it's working.\n\nTo do this, we simply run the `test.py` file in the following way:\n\n```bash\npython test.py --remote\n```\n\nThis script will randomly select some test samples, send them to the cluster, gather the predictions and then plot them for us.\n\n**A note on running this yourself:**\n*This example is set up to connect to a kubernetes cluster running locally on your machine. If yours is local too, you'll need to make sure you [port forward](https://docs.seldon.io/projects/seldon-core/en/latest/install/kind.html#local-port-forwarding) before sending requests. If your cluster is remote, you'll need to change the `inference_url` variable on line 21 of `test.py`.*\n\nHaving deployed our model to kubernetes and tested it, our setup now looks like this:\n![step_4](img/step_4.png)\n\n## Scaling the Model\n\nOur model is now running in a production environment and able to handle requests from external sources. This is awesome but what happens as the number of requests being sent to our model starts to increase? Eventually, we'll reach the limit of what a single server can handle. Thankfully, we can get around this problem by scaling our model [horizontally](https://en.wikipedia.org/wiki/Scalability#Horizontal_or_scale_out).\n\nKubernetes and Seldon Core make this really easy to do by simply running:\n\n```bash\nkubectl scale sdep cassava --replicas=3\n```\n\nWe can replace the `--replicas=3` with any number we want to scale to. \n\nTo watch the servers scaling out we can run:\n\n```bash\nkubectl get pods --watch\n```\n\nOnce the new replicas have finished rolling out, our setup now looks like this:\n![step_5](img/step_5.png)\n\n\nIn this tutorial we've scaled the model out manually to show how it works. In a real environment we'd want to set up [auto-scaling](https://docs.seldon.io/projects/seldon-core/en/latest/graph/scaling.html#autoscaling-seldon-deployments) to make sure our prediction API is always online and performing as expected.\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseldonio%2Fcassava-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fseldonio%2Fcassava-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseldonio%2Fcassava-example/lists"}