{"id":18524246,"url":"https://github.com/datakitchen/dataops-observability-agents","last_synced_at":"2025-04-09T11:31:59.952Z","repository":{"id":234517425,"uuid":"787485345","full_name":"DataKitchen/dataops-observability-agents","owner":"DataKitchen","description":"DataOps Observability Integration Agents are part of DataKitchen's Open Source Data Observability. They connect to various ETL, ELT, BI, data science, data visualization, data governance, and data analytic tools. They provide logs, messages, metrics, overall run-time start/stop, subtask status, and scheduling information to DataOps Observability.","archived":false,"fork":false,"pushed_at":"2025-03-19T14:59:46.000Z","size":250,"stargazers_count":27,"open_issues_count":3,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-19T15:41:59.849Z","etag":null,"topics":["data-engineering","data-observability","integrations","monitoring-plugins"],"latest_commit_sha":null,"homepage":"https://datakitchen.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataKitchen.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.d/template.jinja","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-16T16:00:04.000Z","updated_at":"2025-03-19T14:59:50.000Z","dependencies_parsed_at":"2024-04-22T18:02:21.726Z","dependency_job_id":"378e9476-ab8c-4fd2-9ab0-b095e1d369c9","html_url":"https://github.com/DataKitchen/dataops-observability-agents","commit_stats":null,"previous_names":["datakitchen/dataops-observability-agents"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataKitchen%2Fdataops-observability-agents","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataKitchen%2Fdataops-observability-agents/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataKitchen%2Fdataops-observability-agents/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataKitchen%2Fdataops-observability-agents/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataKitchen","download_url":"https://codeload.github.com/DataKitchen/dataops-observability-agents/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248031623,"owners_count":21036441,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering","data-observability","integrations","monitoring-plugins"],"created_at":"2024-11-06T17:40:10.870Z","updated_at":"2025-04-09T11:31:58.538Z","avatar_url":"https://github.com/DataKitchen.png","language":"Python","readme":"\n# DataOps Observability Integration Agents ![apache 2.0 license Badge](https://img.shields.io/badge/License%20-%20Apache%202.0%20-%20blue) ![PRs Badge](https://img.shields.io/badge/PRs%20-%20Welcome%20-%20green)\n\n*\u003cp style=\"text-align: center;\"\u003eDataOps Observability Integration Agents are part of DataKitchen's Open Source Data Observability. They connect to various ETL, ELT, BI, data science, data visualization, data governance, and data analytic tools. They provide logs, messages, metrics, overall run-time start/stop, subtask status, and scheduling information to DataOps Observability.\u003c/p\u003e*\n\n![DatKitchen Open Source Data Observability](https://datakitchen.io/wp-content/uploads/2024/04/Screenshot-2024-04-22-at-12.40.59 PM.png)\n\n## Current List of Integration Agents\n\n\u003cp align=\"center\"\u003e\n\u003cimg alt=\"DatKitchen Open Source Data Observability Integration Agents List\" src=\"https://datakitchen.io/wp-content/uploads/2024/07/Datakitchen-agent-list-q2-24.png\" width=\"70%\" \u003e\n\u003c/p\u003e\n\n## Setup\n\nBefore you begin you will need to clone the repository.\n\n```sh\n# Mac with Homebrew\nbrew install git\n# Ubuntu\nsudo apt install git\n\ngit clone \u003cURL\u003e\n```\n\n### Developer Environment\n\nCreate a development environment by running the following:\n\n```shell\n# this project requires python 3.11\ncd /your/cloned/repo/\npython3.11 -m venv venv\nsource venv/bin/activate\n# Linux\npip install .[dev,release]\n# Note: when running on MacOS with zsh shell, enclose the directive in single quotes to stop the shell\n#       interpreting it as a wild card.\npip install '.[dev,release]'\n```\n\nNext, you should set up `pre-commit`.\n\n```shell\npre-commit install\n```\n\n## Running tests\n\n### pytest\n\nTests will work by default from the terminal when run from root.\n\ntests can be run from the root with the following command.\n\n```shell\npytest .\n```\n\n### PyCharm\n\nSome fixtures are specified as plugins from the root `conftest.py`. PyCharm by default sets the root to `tests` or the\ntest directory. You will need to manually set the root for tests that use the plugin.\n\n`Edit Configurations...` --\u003e `Edit Configuration Templates` --\u003e `Python Tests` --\u003e `Pytest` --\u003e `Working directory`.\n\nChange it to be the root of the repo.\n\nA configuration has been saved in `.run/Template Python tests.run.xml`. Hopefully pycharm will pick that up\nautomatically.\n\n\n### Sharing data fixtures\n\nA lot of tests require data that be shared. By default, all testlib/**/fixtures files are installed as pytest\nplugins. That means the fixture is accessible just by referencing it, and allows us to reference just a single\nset of data.\n\nPlease use it.\n\nSee `\u003croot\u003e/conftest.py` for details.\n\n\n## Running the quality tools\n\nAfter setting up your environment and entering `mypy` can be from `invoke`\n\n```shell\ninv mypy\n```\n\nThe rest of the tools can be run as a part of pre-commit, or through `invoke`\n\n```shell\ninv lint\n```\n\nTIP: You can run these both together like so:\n\n```shell\ninv lint mypy\n```\n\n## Building an image\n\nTo test that the docker image builds, you can run\n\n```shell\n# --tag may be specified multiple times.\ninv build-image [--tag tag]\n```\n\n## Creating a release note\n\nA release note must be created in order to merge into main. You can do so manually after installing the development\ntools, or you can use the `invoke mk-note` command which will prompt you for the correct information.\n\nYou'll need your ticket number and release type.\n\nValid release note categories are:\n\n* fixed\n* added\n* deprecated\n* removed\n* changed\n* chore\n\nthese follow the [Keep a Changlog](https://keepachangelog.com/en/1.1.0/) format.\n\nRun the following:\n\n```shell\n# e.g., towncrier create AG-17.fixed.md --edit\ntowncrier create your-ticket.type.md\n# or; and follow the prompts\ninv mk-note\n```\n\nThese notes will be consumed by leadership and tech-comms. Be thorough and describe _what_ you did!\nNote: `chore` types won't appear on changelog. They also do not require a ticket, just press `enter` when prompted for board name.\n\n\n## Doing a release\n\nReleases should be done strictly through the build-system. Tagging should be done through the build-system.\n\nOn either a build of `main` or a manual build of main, you should see a series of manual jobs in the pipeline.\n\n- Trigger Breaking Release\n- Trigger Feature Release\n- Trigger Bugfix Release\n\nChoose one (and only one) of these jobs and hit the \"play\" button.\n\nThese will increment the versions and push them to the appropriate release channels.\n\n## Community\n\n### Getting Started Guide\nWe recommend you start by going through the [Data Observability Overview Demo](https://docs.datakitchen.io/articles/open-source-data-observability/data-observability-overview).\n\n### Support\nFor support requests, [join the Data Observability Slack](https://data-observability-slack.datakitchen.io/join) and ask post on #support channel.\n\n### Connect\nTalk and Learn with other data practitioners who are building with DataKitchen. Share knowledge, get help, and contribute to our open-source project. \n\nJoin our community here:\n\n* 🌟 [Star us on GitHub](https://github.com/DataKitchen/data-observability-installer)\n\n* 🐦 [Follow us on Twitter](https://twitter.com/i/flow/login?redirect_after_login=%2Fdatakitchen_io)\n\n* 🕴️ [Follow us on LinkedIn](https://www.linkedin.com/company/datakitchen)\n\n* 📺 [Get Free DataOps Fundamentals Certification](https://info.datakitchen.io/training-certification-dataops-fundamentals)\n\n* 📚 [Read our blog posts](https://datakitchen.io/blog/)\n\n* 👋 [Join us on Slack](https://data-observability-slack.datakitchen.io/join)\n\n* 🗃 [Sign The DataOps Manifesto](https://DataOpsManifesto.org)\n\n* 🗃 [Sign The Data Journey Manifesto](https://DataJourneyManifesto.org)\n\n\n### Contributing\nFor details on contributing a new Agent or running the project for development, check out our contributing guide.\n\n### License\nDataKitchen DataOps Observability Agents are Apache 2.0 licensed.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatakitchen%2Fdataops-observability-agents","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatakitchen%2Fdataops-observability-agents","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatakitchen%2Fdataops-observability-agents/lists"}