{"id":37403477,"url":"https://github.com/oslokommune/dataplatform-probe","last_synced_at":"2026-01-16T05:48:24.325Z","repository":{"id":37087372,"uuid":"302338511","full_name":"oslokommune/dataplatform-probe","owner":"oslokommune","description":"Monitoring service for dataplatform services and events ","archived":false,"fork":false,"pushed_at":"2026-01-13T07:51:34.000Z","size":188,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2026-01-13T10:36:31.261Z","etag":null,"topics":["dataplatform"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oslokommune.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-10-08T12:50:08.000Z","updated_at":"2026-01-13T07:51:37.000Z","dependencies_parsed_at":"2024-04-17T11:47:16.067Z","dependency_job_id":"5e7da5bc-dd84-4b8c-96ea-8d0f40f99af3","html_url":"https://github.com/oslokommune/dataplatform-probe","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oslokommune/dataplatform-probe","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslokommune%2Fdataplatform-probe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslokommune%2Fdataplatform-probe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslokommune%2Fdataplatform-probe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslokommune%2Fdataplatform-probe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oslokommune","download_url":"https://codeload.github.com/oslokommune/dataplatform-probe/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslokommune%2Fdataplatform-probe/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28477419,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T03:13:13.607Z","status":"ssl_error","status_checked_at":"2026-01-16T03:11:47.863Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataplatform"],"created_at":"2026-01-16T05:48:24.230Z","updated_at":"2026-01-16T05:48:24.295Z","avatar_url":"https://github.com/oslokommune.png","language":"Python","readme":"# dataplatform-probe\nMonitoring service for dataplatform services.\n\nContinuously sends GET requests to the dataplatform metadata-api in order to keep an eye on Keycloak.\n\n## Metrics\nThis app uses the [prometheus_client](https://github.com/prometheus/client_python) library to expose\nmetrics to Prometheus regarding pipeline latency through a http server on port `8000`.\n\n| Name                       | Type      | Description                                 |\n|----------------------------|-----------|---------------------------------------------|\n| `probe_requests_created`   | `Counter` | Number of created requests                  |\n| `probe_requests_succeeded` | `Counter` | Number of succeeded requests                |\n| `probe_requests_failed`    | `Counter` | Number of failed requests                   |\n| `probe_request_duration`   | `Gauge`   | The duration of the last succeeded requests |\n\n## Configuration\n\nThe app is configurable by setting the following environment variables (* = required, no default):\n\n| Name                         | Description                                                  | Default |\n|------------------------------|--------------------------------------------------------------|---------|\n| `DATASET_ID`*                | Dataset ID                                                   |         |\n| `TASK_INTERVAL_SECONDS`      | Interval in seconds between requests                         | `30`    |\n| `BETTERUPTIME_HEARTBEAT_URL` | Send heartbeat to Better Uptime on successful task execution |         |\n\nIn addition, `OKDATA_CLIENT_ID`, `OKDATA_CLIENT_SECRET`, and `OKDATA_ENVIRONMENT`, must also be set when deploying.\n\n## Development\n\nA makefile with various commands is provided for convenience and to ease development. These are:\n\n```sh\n$ make init # Install dependencies\n$ make format # Format using black\n$ make lint # Lint using flake8\n$ make run # Run application (as described below)\n```\n\n### Running\n\nBy issuing the command `make run`, a local environment is configured comprising of the following services (see `local-compose.yaml`):\n\n* [**Prometheus**](https://hub.docker.com/r/prom/prometheus) | `http://localhost:9090` \\\n  Monitoring and alerting toolkit.\n\n* [**Grafana**](https://hub.docker.com/r/grafana/grafana) (+ [tns-db](https://hub.docker.com/r/grafana/tns-db)) | `http://localhost:3000`  \\\n  Observability and data visualization platform. Includes a provisioned datasource and dashboard for the application. Default username/password: `admin`/`admin`.\n\n* **HTTP server** | `http://localhost:8081` \\\n  Accepts requests from tasks, currently emulating [okdata-metadata-api](https://github.com/oslokommune/okdata-metadata-api). By default configured to introduce some (more or less random) latency (between 0-3 seconds) for ~10 percent of requests, as well as failing with `401` for ~5 percent (configurable in `local/http_server/http_server.py`).\n\nThe `run` target sets `LOCAL_RUN=true` and `LOCAL_SERVICES_ONLY=true`. While the first environment variable enables \"debug mode\", the latter tells the application to use the \"dummy\" HTTP server mentioned above.\n\nTo test against real dataplatform services (while still running the application locally), set the appropriate environment variables listed above (i.e. credentials and dataset id) and use `make run-dp`.\n\n```sh\n$ docker compose -f local-compose.yaml ps # Check services\n$ docker compose -f local-compose.yaml logs --follow --tail=10 # Tail service logs\n```\n\n## Deploy\n\nDeploy to both dev and prod is automatic via GitHub Actions on push to master.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foslokommune%2Fdataplatform-probe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foslokommune%2Fdataplatform-probe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foslokommune%2Fdataplatform-probe/lists"}