https://github.com/jakthom/hercules

A DuckDB-powered Prometheus exporter
https://github.com/jakthom/hercules

duckdb metrics monitoring observability observability-data prometheus prometheus-exporter prometheus-metrics prometheus-utility

Last synced: about 2 months ago
JSON representation

A DuckDB-powered Prometheus exporter

• Host: GitHub
• URL: https://github.com/jakthom/hercules
• Owner: jakthom
• License: mit
• Created: 2024-10-16T15:41:38.000Z (8 months ago)
• Default Branch: main
• Last Pushed: 2024-12-05T05:27:07.000Z (6 months ago)
• Last Synced: 2025-03-28T20:46:21.417Z (3 months ago)
• Topics: duckdb, metrics, monitoring, observability, observability-data, prometheus, prometheus-exporter, prometheus-metrics, prometheus-utility
• Language: Go
• Homepage:
• Size: 6.78 MB
• Stars: 38
• Watchers: 4
• Forks: 2
• Open Issues: 13
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        


  

  


  Hercules

  




 Write SQL. Get Prometheus Metrics.




![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/jakthom/hercules)

[![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)

[![test](https://github.com/jakthom/hercules/actions/workflows/test.yml/badge.svg)](https://github.com/jakthom/hercules/actions/workflows/test.yml)

[![lint](https://github.com/jakthom/hercules/actions/workflows/lint.yml/badge.svg)](https://github.com/jakthom/hercules/actions/workflows/lint.yml)



# ⚡ Quickstart

Launching Hercules in a Codespace is the easiest way to get started.

A `make run` from inside your codespace will get things spinning.

[![Launch GitHub Codespace](https://github.com/codespaces/badge.svg)](https://github.com/codespaces/new?hide_repo_select=true&ref=main&repo=873715049)

# Sources

Hercules generates Prometheus metrics by querying:

- **Local files** (parquet, json, csv, xlsx, etc)

- **Object storage** (GCS, S3, Azure Blob)

- **HTTP endpoints**

- **Databases** (PostgreSQL, MySQL, SQLite)

- **Data lakes** (Iceberg, Delta)

- **Data warehouses** (BigQuery)

- **Arrow IPC buffers**

Sources can be cached and periodically refreshed, or act as views to the underlying data.

Metrics from multiple sources can be materialized using a single exporter.

# Metrics

### Definition

Metric definitions are `yml` and use `sql` in a number of supported dialects to aggregate, enrich, and materialize metric values.

Hercules supports Prometheus **gauges, counters, summaries, and histograms.**

### Enrichment

Sources and metrics can be *externally enriched*, leading to more ***thorough***, ***accurate*** (or is it precise?), ***properly-labeled*** metrics.

Integrate, calculate, enrich, and label on the edge.

# Macros

Metric definitions can be kept DRY using SQL macros.

Macros are useful for:

- Parsing log lines

- Reading useragent strings

- Schematizing unstructured data

- Enriching results with third-party tooling

- Tokenizing attributes

#  Labels

Hercules propagates global labels to all configured metrics. So you don't have to guess where a metric came from.

Labels are propagated from configuration or sourced from environment variables.

# Packages

Hercules extensions, sources, metrics, and macros can be logically grouped and distributed by the use of **packages**.

Examples can be found in the [hercules-packages](/hercules-packages/) directory.

# Bonus

- Calculate prometheus-compatible metrics from geospatial data

- Coerce unwieldy files to useful statistics using full-text search

- Use modern [pipe sql syntax](https://research.google/pubs/sql-has-problems-we-can-fix-them-pipe-syntax-in-sql/) or [prql](https://prql-lang.org/) for defining and transforming your metrics

- You don't need to [start queries with `select`](https://jvns.ca/blog/2019/10/03/sql-queries-don-t-start-with-select/).

# Further Resources

More to come.