https://github.com/bilbyai/observability
Documentation on our observability dashboards
https://github.com/bilbyai/observability
documentation elasticsearch excalidraw kibana
Last synced: 2 months ago
JSON representation
Documentation on our observability dashboards
- Host: GitHub
- URL: https://github.com/bilbyai/observability
- Owner: bilbyai
- Created: 2025-07-17T03:25:40.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2025-07-18T09:53:15.000Z (3 months ago)
- Last Synced: 2025-07-18T13:39:00.280Z (3 months ago)
- Topics: documentation, elasticsearch, excalidraw, kibana
- Language: Shell
- Homepage: https://bilby.kb.asia-southeast1.gcp.elastic-cloud.com/s/official-china/app/dashboards#/list
- Size: 7.91 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# Data Observability on Elastic and Kibana
This is a documentation repository that provides information about how to access
our data through Kibana, as well as observability dashboards that monitor,
explore, and ensure the quality of our data pipelines built on Elastic and
Kibana.
## Overview
Our observability solution consists of three dashboards and the Kibana Discover
page designed to provide visibility into our data:
| Page | Documentation Link | Purpose | Description |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| [Pipeline Health Dashboard]() | [Documentation](https://github.com/bilbyai/observability/tree/main/pipeline-health) | Quickly see the status of our pipelines. | This dashboard answers the critical question within 30 seconds: "Is the pipeline running ok?" |
| ~~Data QA Dashboard~~ | (Not implemented yet due to lack of data) | Determine the quality of our data pipelines. | This dashboard will help answer: "Is the data produced by the pipeline ok?" |
| [Data Exploration Dashboard]() | [Documentation](https://github.com/bilbyai/observability/tree/main/data-exploration) | Explore and understand our data. | Get insights into data patterns, distributions, and characteristics to answer: "What does the data look like?" |
| [Kibana Discover](https://bilby.kb.asia-southeast1.gcp.elastic-cloud.com/s/official-china/app/discover#/) | [User Guide](https://github.com/bilbyai/observability/tree/main/kibana-discover) | For in-depth investigations | This is used to find specific data points via filters. |
> 🚨 For in-depth investigations, please use
> [Kibana Discover](https://bilby.kb.asia-southeast1.gcp.elastic-cloud.com/s/official-china/app/discover#/)
> to directly query the data.
### 1. ⚠️ Pipeline Health Dashboard
Monitor the operational status of your data pipelines in real-time.
This dashboard answers the critical question: "Is the pipeline running ok?"
### 2. 🎯 Data QA Dashboard (Coming Soon)
Ensure data quality by validating the output of your pipelines.
This dashboard will help answer: "Is the data produced by the pipeline ok?"
### 3. 📊 Data Exploration Dashboard
Explore and understand your data through interactive visualizations. Get
insights into data patterns, distributions, and characteristics to answer: "What
does the data look like?"
### 4. 🔍 Kibana Discover
For detailed data exploration and custom queries, use
[Kibana Discover](https://bilby.kb.asia-southeast1.gcp.elastic-cloud.com/s/official-china/app/discover#/)
to directly search and analyze your data.
## Architecture
We've decided to go with Elasticsearch and Kibana for the observability stack.
This allows us to create visualizations with our date pretty easily, and data
analysts and non-technical users can build visualizations on their own without
needing to think about data pipeline and infrastructure.
## Contributing
Please refer to individual dashboard documentation for specific implementation
details, as shown above.
If you'd like to build your own dashboard, please feel free! Tell us about it!