{"id":15137026,"url":"https://github.com/lilili-bot/mlops-infra-ta","last_synced_at":"2026-01-04T06:40:30.154Z","repository":{"id":256174216,"uuid":"854509279","full_name":"lilili-bot/mlops-infra-ta","owner":"lilili-bot","description":"fastapi service observability with Grafana Cloud and OpenTelemetry, gke for infra.","archived":false,"fork":false,"pushed_at":"2024-09-09T19:14:59.000Z","size":339,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-30T18:47:33.370Z","etag":null,"topics":["fastapi","gke-terraform","grafana","observability","opentelemetry"],"latest_commit_sha":null,"homepage":"https://lilili-bot.github.io/mlops-infra-ta/","language":"Smarty","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lilili-bot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-09T09:54:40.000Z","updated_at":"2024-09-09T19:17:25.000Z","dependencies_parsed_at":"2024-09-09T12:03:08.550Z","dependency_job_id":"796640aa-21d0-4c32-ac9e-7b0828107bdb","html_url":"https://github.com/lilili-bot/mlops-infra-ta","commit_stats":null,"previous_names":["lilili-bot/mlops-infra-ta"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lilili-bot%2Fmlops-infra-ta","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lilili-bot%2Fmlops-infra-ta/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lilili-bot%2Fmlops-infra-ta/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lilili-bot%2Fmlops-infra-ta/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lilili-bot","download_url":"https://codeload.github.com/lilili-bot/mlops-infra-ta/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237821751,"owners_count":19371819,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastapi","gke-terraform","grafana","observability","opentelemetry"],"created_at":"2024-09-26T06:43:01.566Z","updated_at":"2025-10-23T11:32:09.270Z","avatar_url":"https://github.com/lilili-bot.png","language":"Smarty","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FastAPI OpenTelemetry Project\n\n## Summary\n\nThis project demonstrates how to effectively integrate FastAPI with OpenTelemetry to achieve enhanced observability. By combining FastAPI with OpenTelemetry and Grafana, you gain several advantages:\n\n- **Simplified Instrumentation:** The project includes minimal customized instrumentation with OpenTelemetry, showcasing how easy it is to get started.\n- **Flexibility:** The setup allows for flexible and extensible monitoring infrastructure management, enabling easy adjustments and scalability.\n- **Enhanced Observability:** Leverage distributed tracing and monitoring to gain deep insights into your application's performance and behavior.\n- **Near Real-time Visualization:** Use Grafana dashboards to visualize metrics and traces in real-time, providing quick feedback and alerting capabilities.\n\n## Table of Contents\n\n- [Prerequisites](#prerequisites)\n- [Infrastructure Setup](#infrastructure-setup)\n- [Seceret management](#secret-management)\n-  [oTelCol deployment](#deploy-otel-collector-daemonset)\n- [Service Deployment](#service-deployment)\n- [Monitoring and Tracing](#monitoring-and-tracing)\n- [To do](#to-do)\n\n## Prerequisites\n\nBefore you begin, ensure you have the following installed:\n\n- [Terraform](https://www.terraform.io/downloads.html)\n- [Docker](https://www.docker.com/products/docker-desktop)\n- [GCP SDK](https://cloud.google.com/sdk/install)\n- [kubectl](https://kubernetes.io/docs/tasks/tools/)\n- [Google Artifactory](https://cloud.google.com/artifact-registry/docs)\n- [GCS](https://cloud.google.com/storage/docs)\n\n## Infrastructure Setup\n\n1. **Authenticate with GCP:**\n    ```sh\n    gcloud auth login\n    gcloud auth application-default login\n    ```\n2. **Set Environment Variables:**\n\n    Export the necessary environment variables for Terraform:\n    ```sh\n    export TF_VAR_gcp_credentials_json=\"path/to/your/credentials.json\"\n    export TF_VAR_project_id=\"your-gcp-project-id\"\n    export TF_VAR_region=\"your-gcp-region\"\n    ```\n3. **Initialize Terraform:**\n    ```sh\n    terraform init -backend-config=\"gke-infra/terraform/backend_configs/staging-backend-config.hcl\"\n    ```\n\n3. **Create Infrastructure:**\n    ```sh\n    terraform apply -var-file=\"gke-infra/terraform/envs/staging.tfvars\"\n    ```\n\n    This will create the necessary GKE cluster, deploy the OpenTelemetry Collector and the Grafana instance.\n\n## Secret Management\n\nBefore deploying the FastAPI service, create the necessary secrets to pass them as variables to the service.\n\n1. **Create Secrets** (example using `kubectl`):\n    In this repo, all the gke resources are build namespace otel-col. You could use your own namespace, such as monitoring-demo.\n    ```sh\n    kubectl create secret generic otel-endpoint --from-literal=OTEL_EXPORTER_OTLP_ENDPOINT=\u003cyour otel-collector endpoind\u003e\n    --namespace=otel-col\n\n    kubectl create secret generic grafana-auth --from-literal=username=\"\u003cyour-grafana-cloud-instant-name\u003e\" --from-literal=password=\"\u003cyour grafana access token\u003e\" -n \u003cyour-name-space\u003e\n    ```\n    note: the otel-col endpoint is \u003cotel-collector-service-name\u003e.\u003cyour-namespace\u003e.svc.cluster.local\n\n## Deploy otel-collector daemonset\n    ```\n    helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts\n    helm install otel-collector open-telemetry/opentelemetry-collector -f values.yaml\n    ```\n    Verify if the pods are created.\n    ```\n    ./check_pod_all_logs.sh \n    ```\n\n## Service Deployment\n\n1. **Build and Push Docker Image:**\n    ```sh\n    docker build --platform linux/amd64 -t fastapi-otel:v2.0 .\n    docker tag fastapi-otel:v2.0 \u003cyour-artifact-registry-rep\u003e/\u003cyour-service-name:version\u003e\n    docker push \u003cyour-artifact-registry-rep\u003e/\u003cyour-service-name:version\u003e\n    ```\n\n2. **Deploy to GKE Cluster:**\n    Update the `deployment.yaml` file to include the correct image path from Google Artifactory.\n    ```sh\n    cd gke-infra/deploy/fast_api\n    kubectl apply -f deployment.yaml\n    ```\n    deploy debug pod with traffic simulation script\n    ```\n    kubectl apply -f alpine-pod.yaml\n    kubectl apply -f traffic-simulation-configmap.yaml\n    ```\n    login to the pod\n    ```\n    kubectl exec -it apline-curl -n otel-col -- sh\n    cd script\n    traffic_simulation.sh\n    ```\n    You should see some 200 and 422 responses from service. To check details you could check the log of the service pod. The metics which we custimised with open-telemetry export also to pod log, for debuging purpose.\n    ```\n    kubectl logs -f \u003cname-of-fast-api-service-pod\u003e  -n otel-col\n    ```\n\n## Monitoring and Tracing\n\nThe FastAPI service is instrumented with OpenTelemetry, allowing for distributed tracing and monitoring.\n\n1. **Configure OpenTelemetry Collector:**\n\n    See gke-infra/deploy/otel-collector/values.yaml\n\n2. **Access Metrics in Grafana cloud:**\n\n    Assume you have already Grafana Cloud set up. Stack created, credential set up, enabled OpenTelemetry Connection\n    Reference documentation: https://grafana.com/docs/grafana-cloud/send-data/otlp/send-data-otlp/\n\n4. **Configure Grafana Datasource:**\n    \n    * There will be datasource automatically available based on the connection set in pervious step.\n    * For metrics, go to default datasource connection, details see below image.\n    ![Image description](readme.png)\n\n5. **Create Dashboards:**\n    You can now create custom dashboards in Grafana to visualize the metrics collected by OpenTelemetry.\n\n## To do\n\n* Set up alert via Grafana API.\n* Customised more instrumentation for fastapi as well as for possible backends, such as postgress, redis, rocksdb.\n* Migrate all GCP deployment to terraform.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flilili-bot%2Fmlops-infra-ta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flilili-bot%2Fmlops-infra-ta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flilili-bot%2Fmlops-infra-ta/lists"}