{"id":18524264,"url":"https://github.com/datakitchen/dataops-observability","last_synced_at":"2026-04-01T22:47:45.919Z","repository":{"id":234517899,"uuid":"787887332","full_name":"DataKitchen/dataops-observability","owner":"DataKitchen","description":"DataOps Observability is part of DataKitchen's Open Source Data Observability.  DataOps Observability monitors every data journey from data source to customer value, from any team development environment into production, across every tool, team, environment, and customer so that problems are detected, localized, and understood immediately.","archived":false,"fork":false,"pushed_at":"2026-03-22T03:17:10.000Z","size":4383,"stargazers_count":50,"open_issues_count":5,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-03-22T18:41:49.353Z","etag":null,"topics":["data","data-engineering","data-observability","data-science","dataops","pipleine-monitoring"],"latest_commit_sha":null,"homepage":"https://datakitchen.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataKitchen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-04-17T11:23:27.000Z","updated_at":"2026-03-22T03:17:17.000Z","dependencies_parsed_at":"2024-04-22T17:59:37.440Z","dependency_job_id":"29689394-da55-4ab5-9c95-e90a7c17a5ef","html_url":"https://github.com/DataKitchen/dataops-observability","commit_stats":null,"previous_names":["datakitchen/dataops-observability"],"tags_count":33,"template":false,"template_full_name":null,"purl":"pkg:github/DataKitchen/dataops-observability","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataKitchen%2Fdataops-observability","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataKitchen%2Fdataops-observability/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataKitchen%2Fdataops-observability/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataKitchen%2Fdataops-observability/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataKitchen","download_url":"https://codeload.github.com/DataKitchen/dataops-observability/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataKitchen%2Fdataops-observability/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31292696,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T21:15:39.731Z","status":"ssl_error","status_checked_at":"2026-04-01T21:15:34.046Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","data-engineering","data-observability","data-science","dataops","pipleine-monitoring"],"created_at":"2024-11-06T17:40:13.797Z","updated_at":"2026-04-01T22:47:45.400Z","avatar_url":"https://github.com/DataKitchen.png","language":"Python","readme":"# DataOps Observability\n![apache 2.0 license Badge](https://img.shields.io/badge/License%20-%20Apache%202.0%20-%20blue) ![PRs Badge](https://img.shields.io/badge/PRs%20-%20Welcome%20-%20green) [![Docker Pulls](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2F\u0026query=pull_count\u0026style=flat\u0026label=docker%20pulls\u0026color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-observability) [![Documentation](https://img.shields.io/badge/docs-On%20datakitchen.io-06A04A?style=flat)](https://docs.datakitchen.io/observability/what-is-observability/)\n[![Latest Version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-observability-be%2Ftags%2F\u0026query=results%5B0%5D.name\u0026label=latest%20version\u0026color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-observability-be)\n[![Static Badge](https://img.shields.io/badge/Slack-Join%20Discussion-blue?style=flat\u0026logo=slack)](https://data-observability-slack.datakitchen.io/join)\n\n*\u003cp style=\"text-align: center;\"\u003eDataOps Observability is part of DataKitchen's Open Source Data Observability. DataOps Observability monitors every data journey from data source to customer value, from any team development environment into production, across every tool, team, environment, and customer so that problems are detected, localized, and understood immediately.\u003c/p\u003e*\n\n[![DatKitchen Open Source Data Observability](https://datakitchen.io/wp-content/uploads/2024/04/both-products.png)](https://datakitchen.storylane.io/share/g01ss0plyamz)\n[Interactive Product Tour](https://datakitchen.storylane.io/share/g01ss0plyamz)\n\n## Developer Setup\n\n**This repository requires `Python 3.12` at minimum -- prefer the latest `3.12.X`.**\n\nA local Kubernetes cluster requires\n\n* [docker](https://www.docker.com/)\n* [helm](https://helm.sh/)\n* [minikube](https://minikube.sigs.k8s.io/docs/)\n\n\n### Installation\n\nPrefer using a virtual Python environment when installing. Tools such as\n[virtualenv](https://virtualenv.pypa.io) can be used to set up the environment\nusing a specific Python version. [pyenv](https://github.com/pyenv/pyenv) can be\nused to install the desired Python version if your choice of OS does not provide\nit for you.\n\nExample install\n```bash\npython3.12 -m venv venv\nsource venv/bin/activate\n# Install platform and developer extra packages\npip install --editable '.[dev]'\n```\n\n### Testing\n\n`pytest` is used to run test.\n```bash\ncd /to/observability\npytest # runs both unit and integration tests\n```\n\n### Invoke Testing\nWhile tests can be run with ``pytest \u003cOPTIONS\u003e`` there is an invoke handler to run tests using common patterns. The\ntests are run in parallel by default (which can help determine if there are any unexpected dependencies between tests)\nso that running the tests locally takes less time.\n\n**NOTE:** *Requires* ``pytest-xdist`` *package to be installed. This is specified as a* **dev** *dependency when you\nperform initial environment setup. If you set up your local environment before the invoke commands were added, you\nmay need to install this package.*\n\n#### Commands\n\n| Command | Purpose |\n|---------|---------|\n| `invoke test.all`           | Run all tests                             |\n| `invoke test.unit`          | Run all tests marked as unittests         |\n| `invoke test.integration`   | Run all tests marked as integration tests |\n\n#### Arguments\n\nAll of the ``invoke text.\u003cCMD\u003e`` commands have a few common arguments you may pass.\n\n- ``--level=\u003cVALUE\u003e `` [str] (DEBUG, INFO, WARNING, ERROR, CRITICAL) Set logging output level. DEFAULT: DEBUG\n- ``--maxfail=\u003cVALUE\u003e`` [int] Maximum number of tests allowed to fail before aborting test run. DEFAULT: 10\n- ``--processes=\u003cVALUE\u003e`` [int] Number of test processes to run in parallel. DEFAULT: 5\n\nExample:\n\n```shell\n$ invoke test.all --processes=2 --level=\"INFO\" --maxfail=50\n```\n\n## Running the App\n\nAfter installing the [required tools](#developer-setup), run `invoke deploy.local` for an initial local installation of\nthe Observability backend. It creates a minikube node in a docker instance (i.e. in a separate logical machine) running the\nrequired infrastructure along with the Observability services. Destroy the installation with `invoke deploy.nuke`.\n\n[More `invoke` info](#additional-tools).\n\n### Useful commands\n| Command | Purpose |\n|---------|---------|\n| `minikube ssh`                               | SSH into minikube machine                         |\n| `minikube service list`                      | List all services and the endpoints to reach them |\n| `minikube image build \u003cdocker build params\u003e` | Build docker image inside minikube machine        |\n| `minikube image load \u003cimage\u003e`                | Push docker image from host to minikube machine   |\n\n\n## Developer Experience\n\n### Pre-commit + Linting\n\nWe enforce the use of certain linting tools. To not get caught by the build-system's checks, you should use\n`pre-commit` to scan your commits before they go upstream.\n\nThe following hooks are enabled in pre-commit:\n\n- `black`: The black formatter is enforced on the project. We use a basic configuration. Ideally this should solve any and all\nformatting questions we might encounter.\n- `isort`: the isort import-sorter is enforced on the project. We use it with the `black` profile.\n\nTo enable pre-commit from within your virtual environment, simply run:\n\n```bash\npip install pre-commit\npre-commit install\n```\n\n### Additional tools\n\nThese tools should be used by the developer because the build-system will enforce that the code complies with them.\nThese tools are pinned in the `dev` extra-requirements of `pyproject.toml`, so you can acquire them with\n\n```sh\n# within environment\npip install .[dev]\n```\n\nWe use the following additional tools:\n\n- `pytest`: This tool is used to run the test e.g. `pytest .`\n- `mypy`: This is a static and dynamic type-checking tool. This also checks for unreachable and non-returning code. See `pyproject.toml` for its settings. This\ntool is not included in pre-commit because doing so would require installing this repo's package and additional stubs into the pre-commit environment, which\nis inadvised by pre-commit, and poorly supported.\n- `invoke` (shorthand `inv`): This is a `make` replacement.\n  - Run `invoke --list` to see available commands and e.g. `invoke deploy.restart --help` for additional info on command `restart`.\n  - [Shell tab completion](https://docs.pyinvoke.org/en/stable/invoke.html#shell-tab-completion)\n\n\n### FAQ: mypy errors\n\n#### I've encountered 'Unused \"type: ignore\" comment'\n\nGood news, this means that `mypy` has found symbols for the thing which you are ignoring. That means its time to enable\ntype-checking on these code-paths.\n\nTo resolve this error, do two things:\n\n1. Remove the ignore and fix any type errors.\n2. run `mypy . --install-types` and add any newly installed `types-*` packages installed to our `dev` dependencies.\n\n## Community\n\n### Getting Started Guide\nWe recommend you start by going through the [Data Observability Overview Demo](https://docs.datakitchen.io/tutorials/quickstart-demo/).\n\n### Support\nFor support requests, [join the Data Observability Slack](https://data-observability-slack.datakitchen.io/join) and ask post on #support channel.\n\n### Connect\nTalk and Learn with other data practitioners who are building with DataKitchen. Share knowledge, get help, and contribute to our open-source project.\n\nJoin our community here:\n\n* 🌟 [Star us on GitHub](https://github.com/DataKitchen/data-observability-installer)\n\n* 🐦 [Follow us on Twitter](https://twitter.com/i/flow/login?redirect_after_login=%2Fdatakitchen_io)\n\n* 🕴️ [Follow us on LinkedIn](https://www.linkedin.com/company/datakitchen)\n\n* 📺 [Get Free DataOps Fundamentals Certification](https://info.datakitchen.io/training-certification-dataops-fundamentals)\n\n* 📚 [Read our blog posts](https://datakitchen.io/blog/)\n\n* 👋 [Join us on Slack](https://data-observability-slack.datakitchen.io/join)\n\n* 🗃 [Sign The DataOps Manifesto](https://DataOpsManifesto.org)\n\n* 🗃 [Sign The Data Journey Manifesto](https://DataJourneyManifesto.org)\n\n\n### Contributing\nFor details on contributing or running the project for development, check out our contributing guide (coming soon!).\n\n### License\nDataKitchen DataOps Observability is Apache 2.0 licensed.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatakitchen%2Fdataops-observability","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatakitchen%2Fdataops-observability","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatakitchen%2Fdataops-observability/lists"}