{"id":15654058,"url":"https://github.com/hanneshapke/deploying-deep-learning-models","last_synced_at":"2025-04-23T16:42:28.016Z","repository":{"id":69455220,"uuid":"137128028","full_name":"hanneshapke/Deploying-Deep-Learning-Models","owner":"hanneshapke","description":"Strategies to deploy deep learning models","archived":false,"fork":false,"pushed_at":"2018-07-18T03:26:32.000Z","size":4317,"stargazers_count":27,"open_issues_count":0,"forks_count":4,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-03-30T01:36:42.085Z","etag":null,"topics":["deep-learning","deep-learning-tutorial","devops","machine-learning"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hanneshapke.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-12T21:09:17.000Z","updated_at":"2023-04-05T11:59:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"a2115822-081e-412d-ab8f-005f953aaa2b","html_url":"https://github.com/hanneshapke/Deploying-Deep-Learning-Models","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanneshapke%2FDeploying-Deep-Learning-Models","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanneshapke%2FDeploying-Deep-Learning-Models/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanneshapke%2FDeploying-Deep-Learning-Models/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanneshapke%2FDeploying-Deep-Learning-Models/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hanneshapke","download_url":"https://codeload.github.com/hanneshapke/Deploying-Deep-Learning-Models/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250472049,"owners_count":21436071,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","deep-learning-tutorial","devops","machine-learning"],"created_at":"2024-10-03T12:49:20.966Z","updated_at":"2025-04-23T16:42:27.998Z","avatar_url":"https://github.com/hanneshapke.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deploying Deep Learning Models\n### Strategies to deploy deep learning models\n\nThe slides of my OSCON Tensorflow Day 2018 are available [here](https://github.com/hanneshapke/Deploying-Deep-Learning-Models/blob/master/OSCON%20Tensorflow%20Day%20-%20Deploying%20Deep%20Learning%20Models.pdf).\nI will add a link to the conference video once it is published.\n\n## Sample Project\n\n### Model Structure\n\nLet's predict Amazon product ratings based on the comments with a small LSTM network.\n```python\nmodel_input = Input(shape=(MAX_TOKENS,))\nx = Embedding(input_dim=len(CHARS), output_dim=10, input_length=MAX_TOKENS)(model_input)\nx = LSTM(128)(text_input)\noutput = Dense(5, activation='softmax')(x)\nmodel = Model(inputs=text_input, outputs=output)\noptimizer = RMSprop(lr=0.01)\nmodel.compile(loss='categorical_crossentropy', optimizer=optimizer)\n```\n\n### Testing our Model\n\n#### Negative Review\n\n```python\n\u003e\u003e test_sentence = \"horrible book, don't buy it\"\n\u003e\u003e test_vector = clean_data(test_sentence, max_tokens=MAX_TOKENS, sup_chars=CHARS)\n\u003e\u003e model.predict(test_vector.reshape(1, MAX_TOKENS, len(CHARS)))\n[[0.5927979  0.23748466 0.10798287 0.03301411 0.02872046]]\n```\n\n#### Positive Review\n\n```python\n\u003e\u003e test_sentence = \"Awesome product.\"\n\u003e\u003e test_vector = clean_data(test_sentence, max_tokens=MAX_TOKENS, sup_chars=CHARS)\n\u003e\u003e model.predict(test_vector.reshape(1, MAX_TOKENS, len(CHARS)))\n[[0.03493131 0.0394276  0.08326671 0.2957105  0.5466638 ]]\n```\n\n## Steps to Deploy the Sample Project\n\n#### Export the model as protobuf\n\n```python\nimport os\nfrom keras import backend as K\nimport tensorflow as tf\n\ntf.app.flags.DEFINE_integer('training_iteration', 1000, 'number of training iterations.')\ntf.app.flags.DEFINE_integer('model_version', 1, 'version number of the model.')\ntf.app.flags.DEFINE_string('work_dir', '/tmp', 'Working directory.')\nFLAGS = tf.app.flags.FLAGS\n\nexport_path_base = '/tmp/amazon_reviews'\nexport_path = os.path.join(tf.compat.as_bytes(export_path_base), \n               tf.compat.as_bytes(str(FLAGS.model_version)))\n\nbuilder = tf.saved_model.builder.SavedModelBuilder(export_path)\n\nsignature = tf.saved_model.signature_def_utils.predict_signature_def(\n    inputs={'input': model.input}, outputs={'rating_prob': model.output})\n\nbuilder.add_meta_graph_and_variables(\n    sess=K.get_session(), tags=[tf.saved_model.tag_constants.SERVING],\n    signature_def_map={\n        tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: signature })\nbuilder.save()\n\n```\n\n#### Set up the Tensorflow Serving\n\n```terminal\n$ git clone git@github.com:hanneshapke/Deploying_Deep_Learning_Models.git\n\n$ docker build --pull -t $USER/tensorflow-serving-devel-cpu \\\n                      -f {path to repo}/\\\n                      Deploying_Deep_Learning_Models/\\\n                      examples/Dockerfile .\n\n$ docker run -it -p 8500:8500 -p 8501:8501 \n             -v {model_path}/exported_models/amazon_review/:/models \n             $USER/tensorflow-serving-devel-cpu:latest /bin/bash\n\n$[docker bash] tensorflow_model_server --port=8500 \n                                       --model_name={model_name}\n                                       --model_base_path=/models/{model_name}\n\n```\n\n#### Setup a client (either gRPC or REST based)\n\n```python\nfrom grpc.beta import implementations\nfrom tensorflow_serving.apis import prediction_service_pb2\n\ndef get_stub(host='127.0.0.1', port='8500'):\n    channel = implementations.insecure_channel(host, int(port))\n    stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)\n    return stub\n\ndef get_model_prediction(model_input, stub, \n                         model_name='amazon_reviews', \n                         signature_name='serving_default'):\n\n    request = predict_pb2.PredictRequest()\n    request.model_spec.name = model_name\n    request.model_spec.signature_name = signature_name\n    \n    request.inputs['input'].CopyFrom(\n        tf.contrib.util.make_tensor_proto(\n            model_input.reshape(1, 50), \n            verify_shape=True, shape=(1, 50)))\n \n    response = stub.Predict.future(request, 5.0)  # wait max 5s\n    return response.result().outputs[\"rating_prob\"].float_val\n```\n\n```python\n\u003e\u003e\u003e sentence = \"this product is really helpful\"\n\u003e\u003e\u003e model_input = clean_data_encoded(sentence)\n\n\u003e\u003e\u003e get_model_prediction(model_input, stub)\n[0.0250927172601223, 0.03738045319914818, 0.09454590082168579, \n0.33069494366645813, 0.5122858881950378]\n```\n\n\n## Deployment Options\n\n### On Premise\n\n* Kubeflow\n* Tensorflow Serving\n* MLflow\n* Seldon.io\n\n### Cloud base\n\n* Google Cloud Platform\n* Microsoft Azure ML\n* Amazon SageMaker\n\nMore details [here](https://github.com/hanneshapke/Deploying-Deep-Learning-Models/blob/master/OSCON%20Tensorflow%20Day%20-%20Deploying%20Deep%20Learning%20Models.pdf)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhanneshapke%2Fdeploying-deep-learning-models","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhanneshapke%2Fdeploying-deep-learning-models","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhanneshapke%2Fdeploying-deep-learning-models/lists"}