{"id":18136150,"url":"https://github.com/llandy3d/pytheus","last_synced_at":"2025-04-04T12:06:38.013Z","repository":{"id":44362607,"uuid":"512217389","full_name":"Llandy3d/pytheus","owner":"Llandy3d","description":"experimenting with a new prometheus client for python","archived":false,"fork":false,"pushed_at":"2025-03-09T01:15:00.000Z","size":986,"stargazers_count":96,"open_issues_count":9,"forks_count":8,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-28T11:06:31.311Z","etag":null,"topics":["instrumentation","metrics","monitoring","prometheus","prometheus-client-library","python"],"latest_commit_sha":null,"homepage":"https://pythe.us/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Llandy3d.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-09T15:17:12.000Z","updated_at":"2025-03-10T10:59:09.000Z","dependencies_parsed_at":"2024-11-01T14:11:06.806Z","dependency_job_id":"623b20c2-c005-4a21-88c1-f95a8c54b0e9","html_url":"https://github.com/Llandy3d/pytheus","commit_stats":{"total_commits":334,"total_committers":3,"mean_commits":"111.33333333333333","dds":0.05688622754491013,"last_synced_commit":"819018fd3cfddd42fda8726c81d1e9702106e57a"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Llandy3d%2Fpytheus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Llandy3d%2Fpytheus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Llandy3d%2Fpytheus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Llandy3d%2Fpytheus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Llandy3d","download_url":"https://codeload.github.com/Llandy3d/pytheus/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247174406,"owners_count":20896076,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["instrumentation","metrics","monitoring","prometheus","prometheus-client-library","python"],"created_at":"2024-11-01T14:10:56.844Z","updated_at":"2025-04-04T12:06:37.985Z","avatar_url":"https://github.com/Llandy3d.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"https://user-images.githubusercontent.com/16627175/185823115-b33905c3-f389-40e1-b830-2197889a936a.png\" height=\"400\"\u003e\n\n# pytheus\n\n*playing with metrics*\n\n\n[![ci](https://github.com/Llandy3d/pytheus/workflows/ci/badge.svg?event=push)](https://github.com/Llandy3d/pytheus/actions?query=event%3Apush+branch%3Amain+workflow%3Aci)\n[![pypi](https://img.shields.io/pypi/v/pytheus.svg)](https://pypi.python.org/pypi/pytheus)\n[![versions](https://img.shields.io/pypi/pyversions/pytheus.svg)](https://github.com/Llandy3d/pytheus)\n[![license](https://img.shields.io/github/license/Llandy3d/pytheus.svg)](https://github.com/Llandy3d/pytheus/blob/main/LICENSE)\n[![downloads](https://pepy.tech/badge/pytheus/month)](https://pepy.tech/project/pytheus)\n\n---\n\n**Documentation**: https://pythe.us\n\n---\n\npytheus is a modern python library for collecting [prometheus](https://prometheus.io/docs/introduction/overview/) metrics built with multiprocessing in mind.\n\nSome of the features are:\n\n  - multiple multiprocess support:\n    - redis backend ✅\n    - Rust powered backend 🧪\n    - bring your own ✅\n  - support for default labels value ✅\n  - partial labels value (built in an incremental way) ✅\n  - customizable registry support ✅\n  - registry prefix support ✅\n\n---\n## Philosophy\n\nSimply put is to let you work with metrics the way you want.\n\nBe extremely flexible, allow for customization from the user for anything they might want to do without having to resort to hacks and most importantly offer the same api for single \u0026 multi process scenarios, the switch should be as easy as loading a different backend without having to change anything in the code.\n\n- What you see is what you get.\n- No differences between `singleprocess` \u0026 `multiprocess`, the only change is loading a different backend and everything will work out of the box.\n- High flexibility with an high degree of `labels` control and customization.\n\n---\n## Requirements\n\n- Python 3.8+\n- redis \u003e= 4.0.0 (**optional**: for multiprocessing)\n- pytheus-backend-rs (**optional**: for Rust powered multiprocessing 🦀)\n\n\n---\n\n**Architecture**\n\nA small piece of the architecture is described in the [ARCHITECTURE.md file](ARCHITECTURE.md)\n\n---\n\n**Install**\n\n```\npip install pytheus\n```\n\nOptionally if you want to use the Redis backend (for multiprocess support) you will need the redis library:\n```python\npip install redis\n# or\npip install pytheus[redis]\n```\n\nIf you want to try the Rust based backend (for multiprocess support):\n```python\npip install pytheus-backend-rs\n```\n\n---\n\n**Partial labels support:**\n\n```python\nfrom pytheus.metrics import Counter\n\n# without labels\nmy_metric = Counter('metric_name', 'desc')\nmy_metric.inc()  # example for counter\n\n# with labels\nmy_metric = Counter('metric_name', 'desc', required_labels=['req1', 'req2'])\n\nmy_metric.labels({'req1': '1', 'req2': '2'}).inc()  # you can pass all the labels at once\npartial_my_metric = my_metric.labels({'req1': '1'})  # a cacheable object with one of the required labels already set\nobservable_my_metric = partial_my_metric.labels({'req2': '2'})  # finish setting the remaining values before observing\nobservable_my_metric.inc()\n\n# alternatively to using a `dict` you can use `kwargs` (keyword arguments)\nmy_metric.labels(req1='1', req2='2')\n\n```\n\n---\n\n**Default labels support:**\n\n```python\nfrom pytheus.metrics import Counter\n\n# with default labels\nmy_metric = Counter('metric_name', 'desc', required_labels=['req1', 'req2'], default_labels={'req2': 'me_set!'})\n\nmy_metric.labels({'req1': '1'}).inc()  # as we have req2 as a default label we only need to set the remaining labels for observing\nmy_metric.labels({'req1': '1', 'req2': '2'})  # you can still override default labels!\n\n```\n\n---\n\n**Exposing metrics:**\n\nYou can use the `generate_metrics` function from `pytheus.exposition` to generate the metrics and serve them as an endpoint with your favourite web framework.\n\nAlternatively you can use the `make_wsgi_app` function that creates a simple wsgi app to serve the metrics.\n\n---\n\n## Quickstart / Example\n\nThe `example.py` file starts a flask application with three endpoints:\n  - `/`: just returns a phrase while observing the time taken for the request to complete\n  -  `/slow`: same as before but will sleep so that values will only end up in higher buckets\n  -  `/metrics`: the endpoint used by prometheus to scrape the metrics\n\nIt uses two histograms, one without labels, and one with labels required and a default label that makes it observable.\nTo expose the metrics the `generate_metrics()` function is used.\nnote: the example file is using the redis backend but you can try without and set up prometheus yourself.\n\n### Redis version\n\nFor the redis version you can just clone the repository and run `docker-compose up` to start both redis and prometheus scraping on localhost:8080.\nThen you can start the local server with `python example.py`. (flask is required for it to work)\n\nNow you can visit the described endpoints and by visiting `localhost:9090` you can query prometheus, for example by looking for all the slow requests buckets: `page_visits_latency_seconds_labeled_bucket{speed=\"slow\"}`\n\n\u003cimg width=\"1693\" alt=\"image\" src=\"https://user-images.githubusercontent.com/16627175/206577287-06bf89c3-7ab6-4a70-b14c-415be32ea890.png\"\u003e\n\n### Default version\n\nFor the default single process version you can create your python server like this:\n\n```python\nimport time\nfrom flask import Flask\nfrom pytheus.metrics import Histogram\nfrom pytheus.exposition import generate_metrics\n\napp = Flask(__name__)\n\nhistogram = Histogram('page_visits_latency_seconds', 'used for testing')\n\n# this is the endpoint that prometheus will use to scrape the metrics\n@app.route('/metrics')\ndef metrics():\n    return generate_metrics()\n\n# track time with the context manager\n@app.route('/')\ndef home():\n    with histogram.time():\n        return 'hello world!'\n\n# you can also track time with the decorator shortcut\n@app.route('/slow')\n@histogram\ndef slow():\n    time.sleep(3)\n    return 'hello world! from slow!'\n\napp.run(host='0.0.0.0', port=8080)\n```\n\nand if you have prometheus installed configure it to scrape on localhost:8080 or you can still use the included `docker-compose.yml` file.\n\n---\n\n## Metric types\n\n### Counter\n\nThe Counter is a metric that only increases and can resets to 0. (For example if a service restart, it will start again from zero)\n\n```python\nfrom pytheus.metrics import Counter\n\ncounter = Counter(name=\"my_counter\", description=\"My description\")\n\n# increase by 1\ncounter.inc()\n\n# increase by x\ncounter.inc(7)\n\n# it is possible to count exceptions\nwith counter.count_exceptions():\n    raise ValueError  # increases counter by 1\n\n# you can specify which exceptions to watch for\nwith counter.count_exceptions((IndexError, ValueError)):\n    raise ValueError. # increases counter by 1\n\n# it is possible to use the counter as a decorator as a shortcut to count exceptions\n@counter\ndef test():\n    raise ValueError  # increases counter by 1 when called\n\n# specifying which exceptions to look for also works with the decorator\n@counter(exceptions=(IndexError, ValueError))\ndef test():\n    raise ValueError  # increases counter by 1 when called\n```\n\n---\n\n### Gauge\n\nThe Gauge can increase and decrease its value. It is also possible to set a specific value.\n\n```python\nfrom pytheus.metrics import Gauge\n\ngauge = Gauge(name=\"my_gauge\", description=\"My description\")\n\n# increase by 1\ngauge.inc()\n\n# increase by x\ngauge.inc(7)\n\n# decrease by 1\ngauge.dec()\n\n# set a specific value\ngauge.set(7)\n\n# set to current unix timestamp\ngauge.set_to_current_time()\n\n# it is possible to track progress so that when entered increases the value by 1, and when exited decreases it\nwith gauge.track_inprogress():\n    do_something()\n\n# you can also time a piece of code, will set the duration in seconds to value when exited\nwith gauge.time():\n    do_something()\n\n# tracking time can also be done as a decorator\n@gauge\ndef do_something():\n    ...\n\n# tracking progress is also available via decorator with a flag\n@gauge(track_inprogress=True)\ndef do_something():\n    ...\n```\n\n---\n\n### Histogram\n\nA histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values. ([taken from prometheus docs](https://prometheus.io/docs/concepts/metric_types/#histogram))\n\n```python\nfrom pytheus.metrics import Histogram\n\nhistogram = Histogram(name=\"my_histogram\", description=\"My description\")\n# by default it will have the following buckets: (.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10)\n# note: the +Inf bucket will be added automatically, this is float('inf') in python\n\n# create a histogram specifying buckets\nhistogram = Histogram(name=\"my_histogram\", description=\"My description\", buckets=(0.2, 1, 3))\n\n# observe a value\nhistogram.observe(0.4)\n\n# you can also time a piece of code, will set the duration in seconds to value when exited\nwith histogram.time():\n    do_something()\n\n# tracking time can also be done as a decorator\n@histogram\ndef do_something():\n    ...\n```\n\n## Summary\n\nA Summary is a metric that captures individual observations and tracks the total size \u0026 number of events observed. It can be useful to track latencies for example.\n\n```python\nfrom pytheus.metrics import Summary\n\nsummary = Summary(name=\"my_summary\", description=\"My description\")\n\n# observe a value\nsummary.observe(0.4)\n\n# you can also time a piece of code, will set the duration in seconds to value when exited\nwith summary.time():\n    do_something()\n\n# tracking time can also be done as a decorator\n@summary\ndef do_something():\n    ...\n```\n\n## Custom Collectors\n\nIt is possible to use a custom collector in cases you can't directly instrument the code.\nYou will need to inherit from `CustomCollector` and define the `collect` method.\nAlso be sure to disable the automatic registering of a newly created metric into the default registry.\n\n```python\nfrom pytheus.metrics import Counter, CustomCollector\nfrom pytheus.registry import REGISTRY\n\nclass MyCustomCollector(CustomCollector):\n    def collect():\n        counter = Counter('name', 'desc', registry=None)  # note that we are disabling automatic registering\n        counter.inc()\n        yield counter\n\nREGISTRY.register(MyCustomCollector())\n```\n\nNote: if one of the yield metrics have a name already registered with the registry you are trying to register to, the custom collector will be ignored.\n\n\n## How to use different backends\n\nThings work out of the box, using the SingleProcessBackend:\n\n```python\nfrom pytheus.metrics import Counter\n\ncounter = Counter(\n    name=\"my_metric\",\n    description=\"My description\",\n    required_labels=[\"label_a\", \"label_b\"],\n)\nprint(counter._metric_value_backend.__class__)\n# \u003cclass 'pytheus.backends.base.SingleProcessBackend'\u003e\nprint(counter._metric_value_backend.config)\n# {}\n```\n\nYou can define environment configuration to have different defaults, using two environment variables:\n\n```bash\nexport PYTHEUS_BACKEND_CLASS=\"pytheus.backends.redis.MultiProcessRedisBackend\"\nexport PYTHEUS_BACKEND_CONFIG=\"./config.json\"\n```\n\nNow, create the config file, `./config.json`:\n\n```json\n{\n  \"host\": \"127.0.0.1\",\n  \"port\":  6379\n}\n```\n\nNow we can try the same snippet as above:\n\n```python\nfrom pytheus.metrics import Counter\n\ncounter = Counter(\n    name=\"my_metric\",\n    description=\"My description\",\n    required_labels=[\"label_a\", \"label_b\"],\n)\nprint(counter._metric_value_backend.__class__)\n# \u003cclass 'pytheus.backends.redis.MultiProcessRedisBackend'\u003e\nprint(counter._metric_value_backend.config)\n# {\"host\": \"127.0.0.1\", \"port\":  6379}\n```\n\nYou can also pass the values directly in Python, which would take precedence over the environment\nsetup we have just described:\n\n```python\n\nfrom pytheus.metrics import Counter\nfrom pytheus.backends import load_backend\nfrom pytheus.backends.redis import MultiProcessRedisBackend\n\nload_backend(\n    backend_class=MultiProcessRedisBackend,\n    backend_config={\n      \"host\": \"127.0.0.1\",\n      \"port\":  6379\n    }\n)\n# Notice that if you simply call load_backend(), it would reload config from the environment.\n\n# load_backend() is called automatically at package import, that's why we didn't need to call it\n# directly in the previous example\n\ncounter = Counter(\n    name=\"my_metric\",\n    description=\"My description\",\n    required_labels=[\"label_a\", \"label_b\"],\n)\nprint(counter._metric_value_backend.__class__)\n# \u003cclass 'pytheus.backends.redis.MultiProcessRedisBackend'\u003e\nprint(counter._metric_value_backend.config)\n# {'host': '127.0.0.1', 'port': 6379}\n```\n\n## Create your own Backend\n\n### Custom Backend\n\nYou can create your own backend by implementing a class that fulfills the `Backend` protocol.\n\n```python\nclass Backend(Protocol):\n    def __init__(\n        self,\n        config: BackendConfig,\n        metric: \"_Metric\",\n        histogram_bucket: str | None = None,\n    ) -\u003e None:\n        ...\n\n    def inc(self, value: float) -\u003e None:\n        ...\n\n    def dec(self, value: float) -\u003e None:\n        ...\n\n    def set(self, value: float) -\u003e None:\n        ...\n\n    def get(self) -\u003e float:\n        ...\n```\n\n### Initialization hook\n\nIt's possible that you want to initialize your custom backend or there are one time steps that you want to happen on import.\n\nTo achieve that you can use the class method hook called `_initialize` that accepts a `BackendConfig` parameter.\n\n```python\n@classmethod\ndef _initialize(cls, config: \"BackendConfig\") -\u003e None:\n    # initialization steps\n```\n\n### Client patching\n\nAn **highly experimental** feature is to patch in-place the official `prometheus_client` so that you can test out the multiprocessing mode of this library without committing to it or just to try it out.\n\n```python\nimport prometheus_client\nfrom pytheus.experimental.compatibility import patch_client\n\npatch_client(prometheus_client)\n```\n\nThis has to be done at the start/initialization of your application and you can load a different backend before applying with patch with `patch_client` by using `load_backend`.\n\nThis supports all the metrics types `Counter`, `Gauge`, `Summary` \u0026 `Histogram`. It supports the default `REGISTRY` and `generate_latest`. It also works with the `start_http_server` function.\n\nOf course it's extremely limited and you might be using a feature that is not yet supported, but feel free to try! :)\n\n(This was used to test an internal library without having to migrate all of it, so since I worked on it might as well share it! )\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllandy3d%2Fpytheus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fllandy3d%2Fpytheus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllandy3d%2Fpytheus/lists"}