{"id":15664217,"url":"https://github.com/deep-diver/semantic-segmentation-ml-pipeline","last_synced_at":"2025-04-15T16:22:00.357Z","repository":{"id":58617029,"uuid":"529908151","full_name":"deep-diver/semantic-segmentation-ml-pipeline","owner":"deep-diver","description":"Machine Learning Pipeline for Semantic Segmentation with TensorFlow Extended (TFX) and various GCP products","archived":false,"fork":false,"pushed_at":"2023-06-22T10:51:33.000Z","size":157060,"stargazers_count":95,"open_issues_count":3,"forks_count":22,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-28T22:12:58.064Z","etag":null,"topics":["mlops","semantic-segmentation","tensorflow","tfx","vertex-ai"],"latest_commit_sha":null,"homepage":"https://blog.tensorflow.org/2023/01/end-to-end-pipeline-for-segmentation-tfx-google-cloud-hugging-face.html","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deep-diver.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-28T15:53:36.000Z","updated_at":"2025-01-23T02:20:03.000Z","dependencies_parsed_at":"2024-10-23T08:55:48.588Z","dependency_job_id":"f58fffc2-a35f-45f8-a51c-51e5c481c785","html_url":"https://github.com/deep-diver/semantic-segmentation-ml-pipeline","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deep-diver%2Fsemantic-segmentation-ml-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deep-diver%2Fsemantic-segmentation-ml-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deep-diver%2Fsemantic-segmentation-ml-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deep-diver%2Fsemantic-segmentation-ml-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deep-diver","download_url":"https://codeload.github.com/deep-diver/semantic-segmentation-ml-pipeline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249105499,"owners_count":21213543,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mlops","semantic-segmentation","tensorflow","tfx","vertex-ai"],"created_at":"2024-10-03T13:41:41.175Z","updated_at":"2025-04-15T16:22:00.336Z","avatar_url":"https://github.com/deep-diver.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Python](https://img.shields.io/badge/python-3.9-blue.svg) [![TFX](https://img.shields.io/badge/TFX-1.9.1-orange)](https://www.tensorflow.org/tfx)\n[![Validity Check for Training Pipeline](https://github.com/deep-diver/semantic-segmentation-ml-pipeline/actions/workflows/ci.yml/badge.svg)](https://github.com/deep-diver/semantic-segmentation-ml-pipeline/actions/workflows/ci.yml) [![Trigger Training Pipeline](https://github.com/deep-diver/semantic-segmentation-ml-pipeline/actions/workflows/cd-training-pipeline.yml/badge.svg)](https://github.com/deep-diver/semantic-segmentation-ml-pipeline/actions/workflows/cd-training-pipeline.yml)\n\n# Semantic Segmentation model within ML pipeline\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://i.ibb.co/7kF5kCK/semantic-seg-overview.png\" width=\"80%\" /\u003e\n\u003c/p\u003e\n\nThis repository shows how to build a Machine Learning Pipeline for [Semantic Segmentation](https://paperswithcode.com/task/semantic-segmentation) with [TensorFlow Extended (TFX)](https://www.tensorflow.org/tfx) and various GCP products such as [Vertex Pipeline](https://cloud.google.com/vertex-ai/docs/pipelines), [Vertex Training](https://cloud.google.com/vertex-ai/docs/training/custom-training), [Vertex Endpoint](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api). Also, the ML pipeline contains a custom TFX component that is integrated with Hugging Face 🤗 Hub - `HFPusher`. `HFPusher` pushes a trained model to 🤗 Model Hub and, optionally `Gradio` application to 🤗 Space Hub with the latest model out of the box.\n\n**NOTE**: We use U-NET based TensorFlow model from the [official tutorial](https://www.tensorflow.org/tutorials/images/segmentation). Since we implement an ML pipeline, U-NET like model could be a good starting point. Other SOTA models like [SegFormer from 🤗 `Transformers`](https://huggingface.co/transformers/v4.12.5/model_doc/segformer.html) or [DeepLabv3+](https://keras.io/examples/vision/deeplabv3_plus/) will be explored later.\n\n**NOTE**: The aim of this project is not to serve the most SoTA segmentation model. Our main focus is to demonstrate how to build an end-to-end ML pipeline for semantic segmentation task instead.\n\n**Update 17/02/2023**: This project received the [#TFCommunitySpotlight award](https://twitter.com/TensorFlow/status/1626629821022208020).\n\n**Update 18/01/2023**: We published a blogpost on the TensorFlow blog discussing this project: [End-to-End Pipeline for Segmentation with TFX, Google Cloud, and Hugging Face](https://blog.tensorflow.org/2023/01/end-to-end-pipeline-for-segmentation-tfx-google-cloud-hugging-face.html).\n\n# Project structure\n\n\n```bash\nproject\n│\n└───notebooks\n│   │   gradio_demo.ipynb \n│   │   inference_from_SavedModel.ipynb # test inference w/ Vertex Endpoint\n│   │   parse_tfrecords_pets.ipynb # test TFRecord parsing\n│   │   tfx_pipeline.ipynb # build TFX pipeline within a notebook\n│\n└───tfrecords\n│   │   create_tfrecords_pets.py # script to create TFRecords of PETS dataset\n│\n└───training_pipeline\n    └───apps # Gradio app template codebase    \n    └───models # contains files related to model    \n    └───pipeline # definition of TFX pipeline\n```\n\nInside `training_pipeline` the entrypoints for the pipeline runners are defined in\n`kubeflow_runner.py` and `local_runner.py`.\n\n# Instructions\n\nThe TFX pipeline is designed to be run on both of local and GCP environments. \n\n## On local environment\n\n```\n$ cd training_pipeline\n$ tfx pipeline create --pipeline-path=local_runner.py \\\n                      --engine=local\n$ tfx pipeline compile --pipeline-path=local_runner.py \\\n                       --engine=local\n$ tfx run create --pipeline-name=segformer-training-pipeline \\ \n                 --engine=local\n```\n\n## On Vertex AI environment\n\nThere are two ways to run TFX pipeline on GCP environment(Vertex AI). \n\nFirst, you can run it manually with the following CLIs. In this case, you should replace `GOOGLE_CLOUD_PROJECT` to your GCP project ID in `training_pipeline/pipeline/configs.py` beforehand.\n\n```\n$ cd training_pipeline\n$ tfx pipeline create --pipeline-path=kubeflow_runner.py \\\n                      --engine=vertex\n$ tfx pipeline compile --pipeline-path=kubeflow_runner.py \\\n                       --engine=vertex\n$ tfx run create --pipeline-name=segformer-training-pipeline \\ \n                 --engine=vertex \\ \n                 --project=$GCP_PROJECT_ID \\\n                 --regeion=$GCP_REGION\n```\n\n## Using GitHub Actions\n\nYou can use `workflow_dispatch` feature of GitHub Action to run the pipeline on Vertex AI environment as well. In this case, go to the action tab, then select `Trigger Training Pipeline` on the left pane, then `Run workflow` on the branch of your choice. The GCP project ID in the input parameters will automatically replace the `GOOGLE_CLOUD_PROJECT` in `training_pipeline/pipeline/configs.py`. Also it will be injected to the `tfx run create` CLI.\n\n![](https://i.ibb.co/MkTWLZS/dispatch.png)\n\nFor further understading about how GitHub Action is implemented, please refer to [its README document](.github/workflows/README.md).\n\n# To-do\n\n- [X] Notebook to prepare input dataset in `TFRecord` format\n- [X] Upload the input dataset into the GCS bucket\n- [X] Implement and include [UNet](https://www.tensorflow.org/tutorials/images/segmentation) model in the pipeline\n- [X] Implement Gradio app template\n- [X] Make a complete TFX pipeline with `ExampleGen`, `SchemaGen`, `Resolver`, `Trainer`, `Evaluator`, and `Pusher` components\n- [X] Add necessary configurations to the [configs.py](https://github.com/deep-diver/semantic-segmentation-ml-pipeline/blob/main/training_pipeline/pipeline/configs.py)\n- [X] Add `HFPusher` component to the TFX pipeline\n- [X] Replace `SchemaGen` with `ImportSchemaGen` for better TFRecords parsing capability\n- [X] (Optional) Integrate `Dataflow` in `ImportExampleGen` to handle a large amount of dataset. This feature is included in the code as a reference, but it is not used after we switched the Sidewalk to PETS dataset.\n\n## Misc notes\n\n### On the use of two different datasets\n\nInitially, we started our work with the [Sidewalks dataset](https://huggingface.co/datasets/segments/sidewalk-semantic). This\ndataset contains different stuff and things and is also very high-resolution in nature. To keep the runtime of our pipeline\nfaster and to experiment quicker, we settled with a shallow UNet architecture (from [this tutorial](https://www.tensorflow.org/tutorials/images/segmentation)). This is why, we also downsampled the Sidewalks dataset quite a bit (128x128, 256x256, etc.). But this \nled to poor quality models. \n\nTo circumvent around this, we used the [PETS dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/) where the effects of downsampling\nweren't that visible compared to Sidewalks. \n\nBut do note that the approaches showcases in our pipeline can easily be extended to high-resolution segmentation datasets and different\nmodel architectures (as long as they can be serialized as a `SavedModel`).\n\n## Acknowledgements\n\nWe are thankful to the ML Developer Programs team at Google that provided GCP support.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeep-diver%2Fsemantic-segmentation-ml-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeep-diver%2Fsemantic-segmentation-ml-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeep-diver%2Fsemantic-segmentation-ml-pipeline/lists"}