https://github.com/mizcausevic-dev/search-observability-warehouse
SQL warehouse for search console, crawl, index coverage, freshness drift, and URL-level observability diagnostics.
https://github.com/mizcausevic-dev/search-observability-warehouse
analytics-engineering crawl-analysis data-modeling indexation python search-observability sql technical-seo warehouse
Last synced: 7 days ago
JSON representation
SQL warehouse for search console, crawl, index coverage, freshness drift, and URL-level observability diagnostics.
- Host: GitHub
- URL: https://github.com/mizcausevic-dev/search-observability-warehouse
- Owner: mizcausevic-dev
- Created: 2026-05-11T18:41:45.000Z (27 days ago)
- Default Branch: project/search-observability-warehouse
- Last Pushed: 2026-05-12T21:42:03.000Z (26 days ago)
- Last Synced: 2026-05-12T22:30:37.324Z (26 days ago)
- Topics: analytics-engineering, crawl-analysis, data-modeling, indexation, python, search-observability, sql, technical-seo, warehouse
- Language: HTML
- Homepage: https://mizcausevic.com/skills/
- Size: 528 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# Search Observability Warehouse
Search Observability Warehouse is a warehouse-first project for modeling search console, crawl, index coverage, and URL health signals in one analytics layer. Instead of keeping search performance and crawl risk in separate reports, the project organizes them into shared fact tables, rollup views, and operator-friendly queries.
## Executive Summary
This repo shows how search observability can be treated like a real analytics engineering problem. It models site performance, crawl behavior, index coverage, URL status, freshness drift, and anomaly flags in a single warehouse shape that supports diagnostics instead of isolated reporting.
## Portfolio Takeaway
- explicit SQL and warehouse modeling signal
- search console, crawl, and indexation facts under one schema
- rollup views for page-group and site-level diagnostics
- validation harness proving the schema, seed data, and views execute cleanly
## Overview
| Area | Details |
| --- | --- |
| Warehouse dialect | SQLite-compatible ANSI SQL |
| Validation harness | Python 3 standard library |
| Core layers | schema, seeds, views, operator queries |
| Core domains | sites, page groups, crawl facts, search facts, index coverage, anomaly flags |
| Outputs | SQL rollups, JSON proof, screenshot pack |
## Warehouse Structure
- `sql/schema.sql`
- `sql/seed.sql`
- `sql/views.sql`
- `sql/queries.sql`
- `docs/erd.mmd`
## Core Tables
- `sites`
- `page_groups`
- `source_freshness`
- `crawl_daily`
- `search_console_daily`
- `index_coverage_daily`
- `url_signals`
- `anomaly_flags`
## Request Flow
```mermaid
flowchart LR
A["Search Console daily facts"] --> F["Warehouse views"]
B["Crawl observations"] --> F
C["Index coverage snapshots"] --> F
D["URL-level status signals"] --> F
E["Freshness monitors"] --> F
F --> G["Operator queries and anomaly review"]
```
## Example Questions This Warehouse Answers
- Which page groups are losing CTR without losing impressions?
- Which templates are showing crawl and index mismatches?
- Which sites are falling behind freshness SLAs?
- Which anomaly flags are concentrated in the same business owner lane?
## Validation Output
The included validation harness loads the warehouse into SQLite, executes the schema, seeds, and views, then prints proof queries for:
- site overview
- freshness lag
- crawl vs. index mismatch
- page-group performance
- active anomalies
## Screenshots
### Warehouse Control View
### ERD and Fact Layout
### KPI Rollups
### Validation Proof
## Run Locally
```powershell
cd search-observability-warehouse
python .\scripts\run_demo.py
python -m unittest discover -s tests
```
## Validation
```powershell
cd search-observability-warehouse
python .\scripts\run_demo.py
python -m unittest discover -s tests
```
## Tech Stack
[](https://sqlite.org/lang.html)
[](https://www.python.org/)
[](https://mermaid.js.org/)
## Portfolio Links
- [Kinetic Gain](https://kineticgain.com/)
- [LinkedIn](https://www.linkedin.com/in/mirzacausevic)
- [Skills Page](https://mizcausevic.com/skills/)
- [Medium](https://medium.com/@mizcausevic)
- [GitHub](https://github.com/mizcausevic-dev)