{"id":18836307,"url":"https://github.com/obitech/micro-obs","last_synced_at":"2025-04-14T05:30:55.481Z","repository":{"id":57533419,"uuid":"159844190","full_name":"obitech/micro-obs","owner":"obitech","description":"Microservice observability with Go","archived":false,"fork":false,"pushed_at":"2023-02-16T09:09:00.000Z","size":10776,"stargazers_count":55,"open_issues_count":5,"forks_count":8,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-27T19:22:55.323Z","etag":null,"topics":["docker","elk","go","golang","jaeger","kubernetes","microservices","opentracing","prometheus"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/obitech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-30T15:42:19.000Z","updated_at":"2024-10-11T09:28:44.000Z","dependencies_parsed_at":"2024-06-20T08:22:34.230Z","dependency_job_id":"e88eca88-4ee6-4494-b48e-099f9fa282f5","html_url":"https://github.com/obitech/micro-obs","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obitech%2Fmicro-obs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obitech%2Fmicro-obs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obitech%2Fmicro-obs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obitech%2Fmicro-obs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/obitech","download_url":"https://codeload.github.com/obitech/micro-obs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248826487,"owners_count":21167698,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","elk","go","golang","jaeger","kubernetes","microservices","opentracing","prometheus"],"created_at":"2024-11-08T02:21:21.614Z","updated_at":"2025-04-14T05:30:55.014Z","avatar_url":"https://github.com/obitech.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# [micro-obs](https://github.comobitech/microservices-observability)\n\n[![Build Status](https://travis-ci.org/obitech/micro-obs.svg?branch=master)](https://travis-ci.org/obitech/micro-obs) [![Go Report Card](https://goreportcard.com/badge/github.com/obitech/micro-obs)](https://goreportcard.com/report/github.com/obitech/micro-obs) [![](https://img.shields.io/docker/automated/jrottenberg/ffmpeg.svg)](https://hub.docker.com/r/obitech/micro-obs)\n\n## Contents\n\n  - [Overview](#overview)\n  - [Build it](#build-it)\n  - [Deploy it](#deploy-it)\n    - [Docker](#docker)\n    - [Kubernetes](#kubernetes)\n  - [Use it](#use-it)\n    - [Docker](#docker-1)\n      - [Preparation](#preparation)\n      - [Jaeger](#jaeger)\n      - [Prometheus](#prometheus)\n      - [ELK](#elk)\n    - [Kubernetes](#kubernetes-1)\n      - [Preparation](#preparation)\n      - [Jaeger](#jaeger-1)\n      - [Prometheus](#prometheus-1)\n      - [ELK](#elk-1)\n  - [item](#item)\n  - [order](#order)\n  - [util](#util)\n  - [License](#license)\n\n## Overview\n\nExample of instrumenting a Go microservices application:\n\n- Structured logging via [zap](https://github.com/uber-go/zap)\n- Automatic endpoint monitoring exposing metrics to [Prometheus](https://github.com/prometheus/prometheus)\n- Internal \u0026 distributed tracing via [Jaeger](https://github.com/jaegertracing/jaeger)\n\nDeployments can be made via Docker and on Kubernetes. Additional instrumentation:\n\n- Monitoring of demo app and internal Kubernetes components via [Prometheus Operator](https://github.com/coreos/prometheus-operator)\n- Automatic container log aggregation via [ELK Stack](https://www.elastic.co/elk-stack)\n\nThe example application consists of two services: `item` and `order`. Both service have a seperate [redis](https://redis.io) instance as their primary data store. When an order gets placed, `order` contacts `item` to check if the wished items with the requested quantities are in stock:\n\n![Application overview](static/micro-obs-overview.png)\n\nAPI endpoints of both services are instrumented via Prometheus and export the following metrics:\n\n- `in_flight_requests`: a gauge of requests currently being served by the wrapped handler\n- `api_requests_total`: a counter for requests to the wrapped handler\n- `request_duration_seconds`: a histogram for request latencies\n- `response_size_bytes`: a histogram for response sizes\n\nAdditionally, all requests are traced via Jaeger:\n\n![Observability overview](static/micro-obs-observability.png)\n\nOn Kubernetes, next to the application and node-specific metrics exposed via [node_exporter](https://github.com/prometheus/node_exporter), internal Kubernetes components are monitored as well:\n\n- `kube-apiserver`\n- `kube-dns`\n- `etcd`\n- `kube-controller-manager`\n- `kube-scheduler`\n- `kubelet`\n- [`kube-state-metrics`](https://github.com/kubernetes/kube-state-metrics)\n\nProemtheus is fully managed via the Prometheus Operator, allowing flexible deployment of new instances, rules, alertmanagers as well as convenient monitoring of Services via ServiceMonitors. A [Mailhog](https://github.com/mailhog/MailHog) instance is deployed to test the alerting pipeline.\n\nFor the ELK Stack, a Filebeat DaemonSet is watching all container logs on the nodes.\n\n![Kubernetes overview, arrows are omitted for visibility](static/micro-obs-kubernetes.png)\n\n## Build it\n\nTo build it from source you need [Go 1.11+](https://golang.org/dl/) installed.\n\nThis project uses [Go Modules](https://github.com/golang/go/wiki/Modules) so you can clone the repo to anywhere:\n\n```bash\ngit clone https://github.com/obitech/micro-obs.git\ncd micro-obs/\n```\n\nRun `make` to test \u0026 build it:\n\n```bash\nmake\n# ...\n```\n\nOr `make docker` to build a Docker image:\n\n```bash\nmake docker TAG=yourname/yourimage\n# ...\n```\n\n## Deploy it\n\n### Docker\n\nMake sure [Docker](https://docs.docker.com/install/) and [Docker Compose](https://docs.docker.com/compose/install/) are installed.\n\nDeploy the stack:\n\n```bash\ncd deploy/docker\ndocker-compose up -d\n```\n\n### Kubernetes\n\nMake sure [Docker](https://docs.docker.com/install/) and Kubernets (via [Minikube](https://github.com/kubernetes/minikube) or [microk8s](https://microk8s.io), for example) are installed.\n\nDeploy the `monitoring` stack first:\n\n```bash\nkubectl create -f deploy/k8s/000-monitoring --recursive\n```\n\nIt might take a while for the [Prometheus Operator](https://github.com/coreos/prometheus-operator) CRDs to be created. If the deployment fails, try running above command again.\n\nCheck if everything is up:\n\n```\n$ kubectl get pods -n monitoring\nNAME                                   READY     STATUS    RESTARTS   AGE\nalertmanager-main-0                    2/2       Running   0          1m\nelasticsearch-f8dc4db44-l7ddn          2/2       Running   0          1m\nfilebeat-pnx76                         2/2       Running   0          1m\ngrafana-5bfbfb9665-nf9c9               1/1       Running   0          1m\njaeger-deployment-ff5c4dccc-tfxns      1/1       Running   0          1m\nkibana-86c4b5577b-dxtpf                1/1       Running   0          1m\nkube-state-metrics-58dcbb8579-hsw78    4/4       Running   0          1m\nlogstash-74479c885f-5lfcw              1/1       Running   0          1m\nmailhog-778bd8484d-8vqfg               1/1       Running   0          1m\nnode-exporter-swtd6                    2/2       Running   0          1m\nprometheus-core-0                      3/3       Running   0          1m\nprometheus-operator-85bb8dc95b-wqpx6   1/1       Running   0          1m\n```\n\nNext deploy `micro-obs` stack:\n\n```bash\nkubectl create -f deploy/k8s/100-micro-obs/ --recursive\n```\n\nCheck if everything is up:\n\n```\n$ kubectl get pods -n micro-obs\nNAME                           READY     STATUS    RESTARTS   AGE\nitem-54d9d7d554-2thtl          1/1       Running   0          1m\norder-6549584969-k2cp8         1/1       Running   0          1m\nredis-item-5bcf99c9f7-zdf2r    2/2       Running   0          1m\nredis-order-68869c7986-4s7w2   2/2       Running   0          1m\n```\n\n## Use it\n\nFirst build the `dummy` CLI application:\n\n```make\nmake build-dummy\n```\n\n### Docker\n\nService|Location\n---|---\nitem API|http://localhost:8080/\norder API|http://localhost:8090/\nJaeger Query|http://localhost:16686/\nPrometheus|http://localhost:9090/\nGrafana|http://localhost:3000/\n\n#### Preparation\n\nCreate some dummy data:\n\n```\n./bin/dummy data all\n```\n\n#### Jaeger\n\nAfter creating dummy data, those transactions can be found in the Jaeger Query UI at http://localhost:16686:\n\n![Jaeger UI start screen](static/jaeger1.PNG)\n![Jaeger sample trace](static/jaeger2.PNG)\n\n#### Prometheus\n\nSend some dummy requests:\n\n```\n./bin/dummy requests all / /aksdasd /items /orders /delay /error -n 100 -c 5 \n```\n\nStart Grafana and upload the `deploy/docker/dashboards/micro-obs.json` dashboard:\n\n![Grafana micro-obs dashboard example](static/grafana2.png)\n\n#### ELK\n\nAfter sending some requests, logs can be queried via Kibana running on http://localhost:5601:\n\n![Kibana micro-obs error query](static/kibana1.png)\n\n### Kubernetes\n\nService|Location|Internal FQDN\n---|---|---\nitem API|http://localhost:30808|http://item.micro-obs.service.cluster.local:8080\nitem Redis|.|redis-item.micro-obs.service.cluster.local:3879\norder API|http://localhost:30809|http://order.micro-obs.service.cluster.local:8090\norder Redis|.|http://redis-order.micro-obs.service.cluster.local:3879\nJaeger Query|http://localhost:30686|.\nPrometheus|http://localhost:30900|http://prometheus.monitoring.svc.cluster.local:9090\nGrafana|http://localhost:30300|.\nElasticSearch|.|http://elasticsearch.monitoring.svc.cluster.local:9200\nKibana|http://localhost:30601|.\nMailhog|http://localhost:32025|mailhog.svc.cluster.local:1025\n\n#### Preparation\n\nCreate some dummy data:\n\n```\n./bin/dummy data all -i http://localhost:30808 -o http://localhost:30809\n```\n\n#### Jaeger\n\nAfter creating the dummy data, those transactions can be found in the Jaeger Query UI at http://localhost:30686:\n\n![Jaeger UI start screen](static/jaeger1.PNG)\n![Jaeger sample trace](static/jaeger2.PNG)\n\n#### Prometheus\n\nBoth the Kubernetes' internal components as well as the `micro-obs` application (TODO) is being monitored by Prometheus. Some pre-installed Dashboards can be found in Grafana via http://localhost:30300 (default login is admin:admin):\n\n![Grafana Kubernetes dashboard](static/grafana1.PNG)\n\n#### ELK\n\nTODO\n\n## [item](https://godoc.org/github.com/obitech/micro-obs/item)\n[![godoc reference for item](https://img.shields.io/badge/godoc-reference-blue.svg)](https://godoc.org/github.com/obitech/micro-obs/item) \n\nMethod|Endpoint|Comment\n---|---|---\nGET|`/healthz`|Returns `OK` as string\nGET|`/ping`|Returns a standard API response\nGET|`/items`|Returns all items\nGET|`/items/{id:[a-zA-Z0-9]+}`|Returns a single item by ID\nDELETE|`/items/{id:[a-zA-Z0-9]+}`|Deletes a single item by ID\nPOST|`/items`|Sends a JSON body to create a new item. Will not update if item already exists\nPUT|`/items`|Sends a JSON body to create or update an item. Will update existing item\n\nRequest:\n\n```json\nPOST http://localhost:8080/items\n[\n\t{\n\t\t\"name\": \"banana\",\n\t\t\"desc\": \"a yello fruit\",\n\t\t\"qty\": 5\n\t},\n\t{\n\t\t\"name\": \"water\",\n\t\t\"desc\": \"bottles of water\",\n\t\t\"qty\": 10\n\t},\n\t{\n\t\t\"name\": \"apple\",\n\t\t\"desc\": \"delicious\",\n\t\t\"qty\": 15\n\t}\n]\n```\n\nResponse:\n\n```json\n{\n    \"status\": 201,\n    \"message\": \"items BxYs9DiGaIMXuakIxX, GWkUo1hE3u7vTxR, JAQU27CQrTkQCNr,  created\",\n    \"count\": 3,\n    \"data\": [\n        {\n            \"name\": \"banana\",\n            \"id\": \"BxYs9DiGaIMXuakIxX\",\n            \"desc\": \"a yello fruit\",\n            \"qty\": 5\n        },\n        {\n            \"name\": \"water\",\n            \"id\": \"GWkUo1hE3u7vTxR\",\n            \"desc\": \"bottles of water\",\n            \"qty\": 10\n        },\n        {\n            \"name\": \"apple\",\n            \"id\": \"JAQU27CQrTkQCNr\",\n            \"desc\": \"delicious\",\n            \"qty\": 15\n        }\n    ]\n}\n```\n\n## [order](https://godoc.org/github.com/obitech/micro-obs/order)\n[![godoc reference for ](https://img.shields.io/badge/godoc-reference-blue.svg)](https://godoc.org/github.com/obitech/micro-obs/order) \n\nMethod|Endpoint|Comment\n---|---|---\nGET|`/healthz`|Returns `OK` as string\nGET|`/ping`|Returns a standard API response\nGET|`/orders`|Returns all orders\nGET|`/orders/{id:[0-9]+}`|Returns a single order by ID\nDELETE|`/orders/{id:[a-zA-Z0-9]+}`|Deletes a single order by ID\nPOST|`/orders/create`|Creates a new order. Will query the `item` service first to check if passed items exist and are present in the wished quantity\n\nRequest:\n\n```json\nPOST http://localhost:8090/orders/create\n{\n\t\"items\": [\n\t\t{\n\t\t\t\"id\": \"BxYs9DiGaIMXuakIxX\",\n\t\t\t\"qty\": 2\n\t\t},\n\t\t{\n\t\t\t\"id\": \"GWkUo1hE3u7vTxR\",\n\t\t\t\"qty\": 8\n\t\t}\n\t]\n}\n```\n\nResponse:\n\n```json\n{\n    \"status\": 201,\n    \"message\": \"order 1 created\",\n    \"count\": 1,\n    \"data\": [\n        {\n            \"id\": 1,\n            \"items\": [\n                {\n                    \"id\": \"BxYs9DiGaIMXuakIxX\",\n                    \"qty\": 2\n                },\n                {\n                    \"id\": \"GWkUo1hE3u7vTxR\",\n                    \"qty\": 8\n                }\n            ]\n        }\n    ]\n}\n```\n\n## [util](https://godoc.org/github.com/obitech/micro-obs/util)\n[![godoc reference for util](https://img.shields.io/badge/godoc-reference-blue.svg)](https://godoc.org/github.com/obitech/micro-obs/util) \n\n## License\n\n[MIT](https://spdx.org/licenses/MIT.html)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobitech%2Fmicro-obs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fobitech%2Fmicro-obs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobitech%2Fmicro-obs/lists"}