{"id":50758858,"url":"https://github.com/nasim-raj-laskar/sagemaker-lane-segmentation","last_synced_at":"2026-06-11T08:01:15.675Z","repository":{"id":354894520,"uuid":"1178171755","full_name":"nasim-raj-laskar/Sagemaker-Lane-Segmentation","owner":"nasim-raj-laskar","description":"U-Net lane segmentation pipeline with SageMaker training, MLflow tracking, and registry-gated deployment.","archived":false,"fork":false,"pushed_at":"2026-04-30T18:25:56.000Z","size":25070,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-30T20:13:39.441Z","etag":null,"topics":["amazon-sagemaker"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nasim-raj-laskar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-10T19:02:13.000Z","updated_at":"2026-04-30T18:28:12.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/nasim-raj-laskar/Sagemaker-Lane-Segmentation","commit_stats":null,"previous_names":["nasim-raj-laskar/sagemaker-lane-segmentation"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/nasim-raj-laskar/Sagemaker-Lane-Segmentation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nasim-raj-laskar%2FSagemaker-Lane-Segmentation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nasim-raj-laskar%2FSagemaker-Lane-Segmentation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nasim-raj-laskar%2FSagemaker-Lane-Segmentation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nasim-raj-laskar%2FSagemaker-Lane-Segmentation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nasim-raj-laskar","download_url":"https://codeload.github.com/nasim-raj-laskar/Sagemaker-Lane-Segmentation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nasim-raj-laskar%2FSagemaker-Lane-Segmentation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34188272,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-sagemaker"],"created_at":"2026-06-11T08:01:14.247Z","updated_at":"2026-06-11T08:01:15.656Z","avatar_url":"https://github.com/nasim-raj-laskar.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eLane Segmentation MLOps Pipeline on Amazon SageMaker\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://tensorflow.org/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/TensorFlow-2.11.0-FF6F00?style=flat\u0026logo=tensorflow\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://aws.amazon.com/sagemaker/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/AWS-SageMaker-FF9900?style=flat\u0026logo=amazon-aws\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://mlflow.org/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/MLflow-2.8.1-0194E2?style=flat\u0026logo=mlflow\" /\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://python.org/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Python-3.9+-3776AB?style=flat\u0026logo=python\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/arch.png\" width=\"900\"/\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  Pixel-level binary semantic segmentation of lane boundaries using a fully-convolutional U-Net encoder-decoder trained on 289 annotated road images. The pipeline integrates SageMaker Training Jobs, SageMaker Model Registry with threshold-gated approval, MLflow experiment tracking, and a Streamlit inference frontend backed by TFSMLayer-wrapped SavedModel artifacts.\n\u003c/p\u003e\n\n\u003ch2 align=\"center\"\u003eArchitecture\u003c/h2\u003e\n\n\u003ch3 align=\"center\"\u003eModel\u003c/h3\u003e\n\nSymmetric encoder-decoder (U-Net) with lateral skip connections between mirrored resolution stages. Skip connections concatenate encoder feature maps directly into the decoder path, preserving high-frequency spatial detail lost during max-pooling downsampling.\n\n\u003cp align=\"center\"\u003e\n\n  | Component | Specification |\n  |---|---|\n  | Input tensor | `(N, 256, 832, 3)` — float32, normalized to `[0, 1]` |\n  | Encoder depth | 4 stages — filter progression `[64, 128, 256, 512]` |\n  | Bottleneck | 1024 filters, no spatial downsampling |\n  | Decoder depth | 4 stages — filter progression `[512, 256, 128, 64]` |\n  | Upsampling | Bilinear interpolation (`unpool='bilinear'`) |\n  | Output | `(N, 256, 832, 1)` — sigmoid activation, binary mask |\n  | Loss | Sørensen–Dice coefficient: `L = 1 - (2·|X∩Y| + ε) / (|X| + |Y| + ε)` |\n  | Optimizer | Adam, `lr=1e-4`, default β₁=0.9, β₂=0.999 |\n  | Metrics | Binary accuracy, Mean IoU (`num_classes=2`) |\n\n\u003c/p\u003e\n\nDice loss is preferred over binary cross-entropy here due to severe foreground/background class imbalance — lane pixels constitute a small fraction of total image area, causing BCE to converge to a degenerate all-background solution.\n\n\u003ch3 align=\"center\"\u003eInfrastructure\u003c/h3\u003e\n\n```yaml\nCompute:\n  instance_type: ml.g4dn.xlarge       # 4 vCPU, 16 GiB RAM, 1x NVIDIA T4 (16 GiB VRAM)\n  framework_version: \"2.11.0\"          # TF 2.11 — last version with Keras 2 API\n  container: AWS Deep Learning Container (763104351884.dkr.ecr.\u003cregion\u003e.amazonaws.com)\n\nStorage:\n  training_data: s3://\u003cbucket\u003e/raw-data/          # 289 RGB images + 289 binary masks\n  model_artifacts: s3://\u003cbucket\u003e/model-artifacts/ # versioned tar.gz SavedModel archives\n  experiment_logs: SageMaker MLflow Tracking Server\n\nOrchestration:\n  training: SageMaker Training Jobs (managed spot optional)\n  registry: SageMaker Model Registry (ModelPackageGroup: lane-segmentation-models)\n  tracking: SageMaker MLflow Apps (OIDC-authenticated tracking server)\n```\n\n\u003ch2 align=\"center\"\u003eRepository Structure\u003c/h2\u003e\n\n```\nlane-segmentation-pipeline/\n├── src/\n│   ├── train.py              # Training loop, checkpointing, S3 artifact upload, registry registration\n│   ├── model.py              # U-Net graph construction via keras-unet-collection\n│   ├── data_loader.py        # tf.data pipeline: decode → resize → normalize → augment → batch\n│   ├── mlflow_config.py      # MLflow client init, run context manager, param/metric logging\n│   ├── model_registry.py     # SageMaker boto3 calls: create_model_package, list_model_packages\n│   └── requirements.txt\n├── config/\n│   ├── model.yaml            # Hyperparameters, data config, approval threshold\n│   └── train.yaml            # SageMaker instance config, S3 paths, job name prefix\n├── dataset/\n│   ├── image/                # 289 × RGB road frames (variable resolution, resized to 256×832)\n│   └── mask/                 # 289 × binary lane masks (uint8, values ∈ {0, 255})\n├── assets/\n│   ├── ui.png\n│   └── output.mp4\n├── models/                   # Local SavedModel cache (populated by app.py on first load)\n├── app.py                    # Streamlit frontend: inference, registry status, job launcher\n├── main.py                   # SageMaker Estimator configuration and .fit() invocation\n├── model_registry_utils.py   # CLI wrapper: list packages, patch approval status\n└── MODEL_REGISTRY.md\n```\n\n\u003ch2 align=\"center\"\u003eEnvironment Setup\u003c/h2\u003e\n\n**Requirements:** AWS account with `sagemaker:*`, `s3:*`, `iam:PassRole` permissions; Python ≥ 3.9; AWS CLI v2.\n\n```bash\ngit clone https://github.com/nasim-raj-laskar/lane-segmentation-pipeline.git\ncd lane-segmentation-pipeline/\npip install -r src/requirements.txt\n```\n\nCreate a `.env` file:\n\n```bash\nAWS_ACCOUNT_ID=\u003caccount-id\u003e\nAWS_REGION=\u003cregion\u003e\nS3_BUCKET=\u003cbucket\u003e\nSAGEMAKER_ROLE=SageMakerExecutionRole\nMLFLOW_ARN=arn:aws:sagemaker:\u003cregion\u003e:\u003caccount\u003e:mlflow-tracking-server/\u003cserver-name\u003e\n```\n\n```bash\n# Sync raw dataset to S3 input channel\naws s3 sync dataset/ s3://\u003cbucket\u003e/raw-data/\n```\n\n\u003ch2 align=\"center\"\u003eTraining\u003c/h2\u003e\n\n\u003ch3 align=\"center\"\u003eLaunch SageMaker Training Job\u003c/h3\u003e\n\n```bash\npython main.py\n```\n\nThis instantiates a `sagemaker.tensorflow.TensorFlow` estimator targeting `ml.g4dn.xlarge`, injects `config/model.yaml` hyperparameters as `--hyperparameters`, and calls `.fit()` with the S3 data channel. Training artifacts are written to `/opt/ml/model/` inside the container and automatically uploaded to S3 on job completion.\n\n\u003ch3 align=\"center\"\u003eHyperparameter Reference (config/model.yaml)\u003c/h3\u003e\n\n```yaml\nepochs: 15\nbatch_size: 4                  # constrained by T4 VRAM at 256×832 resolution\nlearning_rate: 0.0001\naccuracy_threshold: 0.85       # minimum val_binary_accuracy for auto-approval\nimg_height: 256\nimg_width: 832\nnormalization_factor: 255.0\nmask_threshold: 255            # binarization cutoff for mask preprocessing\ntest_size: 0.2\nrandom_state: 42\ns3_bucket: \u003cbucket\u003e\ns3_model_prefix: model-artifacts/lane_segmentation_model\ntimestamp_format: '%Y%m%d_%H%M%S'\n```\n\n\u003ch3 align=\"center\"\u003eInfrastructure Configuration (config/train.yaml)\u003c/h3\u003e\n\n```yaml\nsagemaker:\n  instance_type: ml.g4dn.xlarge\n  instance_count: 1\n  framework_version: \"2.11.0\"\n  py_version: py39\n\ns3:\n  bucket: \u003cbucket\u003e\n  data_path: raw-data\n  model_artifacts_path: model-artifacts\n  code_location: code\n\ntraining:\n  job_name_prefix: lane-segmentation-training\n```\n\n\u003ch2 align=\"center\"\u003eModel Registry\u003c/h2\u003e\n\n\u003ch3 align=\"center\"\u003eApproval Gate Logic\u003c/h3\u003e\n\nPost-training, `src/model_registry.py` calls `sagemaker:CreateModelPackage`. Approval status is determined by comparing `final_val_accuracy` against `accuracy_threshold`:\n\n```python\ndef register_model(self, model_s3_uri, metrics, accuracy_threshold=0.8):\n    val_accuracy = metrics.get('final_val_accuracy', 0)\n    approval_status = \"Approved\" if val_accuracy \u003e= accuracy_threshold else \"PendingManualApproval\"\n\n    self.sagemaker.create_model_package({\n        'ModelPackageGroupName': 'lane-segmentation-models',\n        'ModelApprovalStatus': approval_status,\n        'InferenceSpecification': {\n            'Containers': [{\n                'Image': f'763104351884.dkr.ecr.{region}.amazonaws.com/tensorflow-inference:2.12-cpu',\n                'ModelDataUrl': model_s3_uri\n            }]\n        }\n    })\n```\n\nFor stricter multi-criteria gating (val_loss + Mean IoU):\n\n```python\napproval_status = \"Approved\" if (\n    val_accuracy \u003e= accuracy_threshold and\n    val_loss \u003c 0.3 and\n    mean_iou \u003e 0.7\n) else \"PendingManualApproval\"\n```\n\n\u003ch3 align=\"center\"\u003eCLI Operations\u003c/h3\u003e\n\n```bash\n# Enumerate all model package versions with approval status and metrics\npython model_registry_utils.py\n\n# Patch approval status on a specific model package ARN\npython model_registry_utils.py approve \\\n  arn:aws:sagemaker:\u003cregion\u003e:\u003caccount\u003e:model-package/lane-segmentation-models/1\n```\n\n\u003ch3 align=\"center\"\u003eS3 Artifact Layout\u003c/h3\u003e\n\n```\ns3://\u003cbucket\u003e/model-artifacts/lane_segmentation_model/\n├── v1/\n│   ├── 20240430_143022.tar.gz   # TensorFlow SavedModel (saved_model.pb + variables/)\n│   └── metrics.json\n├── v2/\n│   ├── 20240430_150145.tar.gz\n│   └── metrics.json\n└── v3/\n    ├── 20240430_152301.tar.gz\n    └── metrics.json\n```\n\n`metrics.json` schema:\n\n```json\n{\n  \"final_train_loss\": 0.6542,\n  \"final_val_loss\": 0.4382,\n  \"final_train_accuracy\": 0.3605,\n  \"final_val_accuracy\": 0.8366,\n  \"epochs\": 15,\n  \"batch_size\": 4,\n  \"learning_rate\": 0.0001,\n  \"timestamp\": \"20240430_164055\",\n  \"version\": 3\n}\n```\n\n\u003ch2 align=\"center\"\u003eInference Application\u003c/h2\u003e\n\n\u003ch3 align=\"center\"\u003eModel Loading\u003c/h3\u003e\n\n`app.py` resolves the latest `Approved` model package ARN via `list_model_packages`, downloads the SavedModel artifact from S3, and wraps it in a `TFSMLayer` to maintain Keras 3.x functional API compatibility (Keras 3 dropped native `tf.saved_model.load` integration):\n\n```python\n@st.cache_resource\ndef load_model():\n    registry = ModelRegistry()\n    model_s3_uri, model_package_arn = registry.get_latest_approved_model()\n\n    model_layer = tf.keras.layers.TFSMLayer(\n        'models/approved/1',\n        call_endpoint='serving_default'\n    )\n    inputs = tf.keras.Input(shape=(256, 832, 3))\n    outputs = model_layer(inputs)\n    return tf.keras.Model(inputs=inputs, outputs=outputs), model_package_arn\n```\n\n\u003ch3 align=\"center\"\u003eLaunch\u003c/h3\u003e\n\n```bash\nstreamlit run app.py --server.port 8501 --server.address 0.0.0.0\n```\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/ui.png\" alt=\"Streamlit inference UI — image upload, binary mask overlay, registry status panel, and training job launcher\" width=\"900\"/\u003e\n  \u003cbr/\u003e\n  \u003cem\u003eStreamlit frontend: image upload → TFSMLayer inference → binary mask overlay. Registry status and training job launcher rendered in the sidebar.\u003c/em\u003e\n\u003c/p\u003e\n\n\u003ch2 align=\"center\"\u003eMLOps Lifecycle\u003c/h2\u003e\n\n```mermaid\ngraph TD\n    A[SageMaker Training Job] --\u003e B[Epoch Metrics Logged to MLflow]\n    B --\u003e C[val_binary_accuracy evaluated against threshold]\n    C --\u003e|\u003e= threshold| D[ModelApprovalStatus: Approved]\n    C --\u003e|\u003c threshold| E[ModelApprovalStatus: PendingManualApproval]\n    D --\u003e F[app.py resolves latest Approved ARN]\n    E --\u003e G[Manual review via model_registry_utils.py]\n    G --\u003e|approve| F\n    G --\u003e|reject| H[Package remains in PendingManualApproval]\n    F --\u003e I[TFSMLayer inference serving]\n```\n\n\u003ch2 align=\"center\"\u003eExperiment Tracking\u003c/h2\u003e\n\nMLflow run context is opened in `src/train.py` before the Keras `.fit()` call. Hyperparameters are logged once; per-epoch metrics are logged with `step=epoch` for time-series visualization in the MLflow UI:\n\n```python\nwith mlflow.start_run(run_name=f\"lane_seg_{timestamp}\"):\n    mlflow.log_params({\n        'epochs': config['epochs'],\n        'batch_size': config['batch_size'],\n        'learning_rate': config['learning_rate']\n    })\n    for epoch in range(epochs):\n        mlflow.log_metrics({\n            'train_loss': history.history['loss'][epoch],\n            'val_loss': history.history['val_loss'][epoch],\n            'train_accuracy': history.history['binary_accuracy'][epoch],\n            'val_accuracy': history.history['val_binary_accuracy'][epoch]\n        }, step=epoch)\n```\n\nTracked metrics: Dice loss, binary cross-entropy, binary accuracy, Mean IoU, epoch wall-clock time, GPU utilization, peak VRAM allocation, total parameter count, SavedModel size on disk, and batch inference latency (p50/p95).\n\n\u003ch2 align=\"center\"\u003eInteractive Demo (Hugging Face Spaces)\u003c/h2\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/demo.png\" width=\"900\"/\u003e\n  \u003cbr/\u003e\n  \u003cem\u003eLightweight interactive dashboard deployed on Hugging Face Spaces for real-time inference and visualization. Includes adjustable thresholding, overlay tuning, and performance metrics.\u003c/em\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  👉 \u003ca href=\"https://huggingface.co/spaces/Nasim435/Lane-seg\"\u003eTry Live Demo\u003c/a\u003e\n\u003c/p\u003e\n\n\u003ch2 align=\"center\"\u003eReferences\u003c/h2\u003e\n\n- [SageMaker `CreateModelPackage` API](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModelPackage.html)\n- [MLflow Tracking Server](https://mlflow.org/docs/latest/tracking.html#mlflow-tracking-servers)\n- [TensorFlow SavedModel format](https://www.tensorflow.org/guide/saved_model)\n- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/abs/1505.04597)\n- [V-Net / Dice Loss](https://arxiv.org/abs/1606.04797)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnasim-raj-laskar%2Fsagemaker-lane-segmentation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnasim-raj-laskar%2Fsagemaker-lane-segmentation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnasim-raj-laskar%2Fsagemaker-lane-segmentation/lists"}