{"id":19860324,"url":"https://github.com/nivanorge/nivacloud-logging","last_synced_at":"2025-11-09T03:03:53.596Z","repository":{"id":57446282,"uuid":"177112395","full_name":"NIVANorge/nivacloud-logging","owner":"NIVANorge","description":"A set of shared utils for setting up logging in a consistent way in the nivacloud ecosystem.","archived":false,"fork":false,"pushed_at":"2021-05-03T09:26:02.000Z","size":85,"stargazers_count":2,"open_issues_count":5,"forks_count":0,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-04-22T15:55:29.867Z","etag":null,"topics":["aiohttp","flask","gunicorn","logging","mit","python","requests","stackdriver","starlette","tracing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NIVANorge.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-22T09:37:07.000Z","updated_at":"2024-09-16T09:05:10.000Z","dependencies_parsed_at":"2022-09-02T22:11:39.808Z","dependency_job_id":null,"html_url":"https://github.com/NIVANorge/nivacloud-logging","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NIVANorge%2Fnivacloud-logging","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NIVANorge%2Fnivacloud-logging/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NIVANorge%2Fnivacloud-logging/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NIVANorge%2Fnivacloud-logging/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NIVANorge","download_url":"https://codeload.github.com/NIVANorge/nivacloud-logging/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251986606,"owners_count":21675950,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiohttp","flask","gunicorn","logging","mit","python","requests","stackdriver","starlette","tracing"],"created_at":"2024-11-12T15:03:59.592Z","updated_at":"2025-10-19T09:03:59.245Z","avatar_url":"https://github.com/NIVANorge.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Nivacloud-logging\n\nA set of shared utilities for setting up logging and tracing in a\nconsistent way across [NIVA](https://www.niva.no/)'s Python-based\ncloud services.\n\nWe're currently stuffing both regular application and system logs and\ntraces into [Console Logs](https://console.cloud.google.com/logs/) in\nGoogle Cloud, so this is for making sure that we're adding information\nthat makes it easy to trace our development.\n\n## Usage\n\nNormally, you would just call `setup_logging()` and start logging and\nset the `NIVACLOUD_PLAINTEXT_LOGS` if you want plaintext (human-readable)\nlogs instead of JSON. Default is JSON.\n\nIf the `GIT_COMMIT_ID` environment variable is set, a `git_commit_id`\ndefault context containing this will be added to all threads when\n`setup_logging()` is executed.\n\nBy default it will override all loggers to make sure that we get all the\nlogs through our handler. (So that everything is formatted as JSON and\nends up on stdout for Docker logs.) This feels slightly hacky, but I\nthink it's okay for our use-case. If you need to disable this, set\n`NIVACLOUD_OVERRIDE_LOGGERS` to `0`.\n\n```python\nimport logging\nfrom nivacloud_logging.log_utils import setup_logging, LogContext, log_context, log_exceptions\n\nsetup_logging()\nlogging.info(\"something happened\")\n\nwith LogContext(my_id=123):\n    logging.info(\"something happened with some context attached\")\n\nwith LogContext(fjas=\"xyzzy\"), log_exceptions():\n    raise Exception(\"Log this exception, preserving context, then re-raise\")\n\n@log_context(something=\"foo\")\ndef myfun(x):\n    logging.info(\"I'm adding 1 to X and outputting 'something'!\")\n    return x + 1\n```\n\n### Runtime configuration\n\nIf you want to tweak the log level of a running service, you can\nsend `SIGUSR1` to set `INFO` level debugging and `SIGUSR2` to set\n`DEBUG` level debugging.\n\nNB! If running locally on Windows, this feature will not be available.\n\n### Running with Gunicorn\n\nIf you want access logs, pass `--access-logfile -` to Gunicorn. If\nyou want to override Gunicorn's log format before it starts\noutputting logs, you can supply the `Logger` class from\n`gunicorn_logger`, like this:\n\n```\ngunicorn --logger-class nivacloud_logging.gunicorn_logger.Logger\n```\n\n#### --preload\n\nDon't use `--preload`, because we have a bunch of code like this:\n\n```python\ndb = MyDb.connect()\n\n@app.route(\"/\")\ndef something():\n    db.get(\"foo\")\n```\n\nIn cases like this, when you run Gunicorn with more than one worker\nprocess, they may be sharing the file descriptors (sockets, in this\ncase) inherited from the parent process, and there will be no\nsynchronization between them, so in the worst case this may cause data\ncorruption. (It doesn't matter if the library used claims to be\nthread-safe, because these are processes, not threads, so they don't\nknow about each other.)\n\n### Tracing with Requests\n\nIn order to add `Span-Id`, `User-Id` and `Trace-Id` headers to outgoing requests,\nthere is an adapter that will pick up `trace_id`/`span_id` from the\n*LogContext*, alternatively generating `Span-Id` if one doesn't exist.\n\n```python\nsession = requests.Session()\nsession.mount('http://', TracingAdapter())\nsession.mount('https://', TracingAdapter())\nr = session.get(\"https://httpbin.org/headers\")\nprint(f\"Span-Id is {r.json()['headers'].get('Span-Id')}\")\n```\n\n### Tracing with aiohttp client\n\nTo add `Trace-Id` and `Span-Id` headers to outgoing requests, add a\n`TraceConfig` to your session that adds trace IDs and span IDs to your\nheaders in the same way that the Requests tracing adapter does.\n\n```python\nfrom nivacloud_logging.aiohttp_trace import create_client_trace_config\n\nasync with aiohttp.ClientSession(trace_configs=[create_client_trace_config()]) as session, \\\n        LogContext(trace_id='abc123'), \\\n        session.get('https://httpbin.org/headers') as response:\n    r = await response.json()\n    print(f\"Trace-ID is {r['headers'].get('Trace-Id')}\")\n```\n\n### Tracing with Flask\n\nTo set `trace_id`, `user_id` and `span_id` in `LogContext` for incoming requests\nbased on the value of the `Trace-Id`, `User-Id` and `Span-Id` headers, use the\n`TracingMiddleware` like so:\n\n```python\napp = Flask(__name__)\napp.wsgi_app = TracingMiddleware(app.wsgi_app)\n```\n\nThis will also generate log entries for each requests. (In addition to\naccess log entries. This should be improved at some point.)\n\n### Tracing with Starlette\n\nTo get the same functionality as for Flask (see above), do this:\n\n```python\napp = Starlette()\napp.add_middleware(StarletteTracingMiddleware)\n```\n\n## Running tests\n\n```\n$ python setup.py test\n```\n\nOr just run `pytest` if you have all packages installed.\n\n### Quirks\n\nWith [pytest](https://docs.pytest.org/en/latest/) you would normally\nuse *caplog* to check log messages, but we're testing the logging\nitself here, so it makes more sense to use *capsys* to read the\nactual text output.\n\n## Intended audience\n\nThis repository is primarily intended for internal use within\n[Norwegian Institute for Water Research](https://www.niva.no/).\n\n# Modules\n\n## Structured logging\n\nApplications that run in *nivacloud* run on Google Kubernetes\nEngine. All logs from applications are aggregated via stdout to\n[Logs](https://console.cloud.google.com/logs/) running in Google\nCloud. Formerly known as StackDriver.\n\nWhen logs are aggregated from multiple sources via stdout, multi-line\nlogs end up as multiple log entries. One approach to this problem is to\nlog all log statements in json format. More details can be seen in the\n[Google Cloud Structured Logging\ndocumentation](https://cloud.google.com/logging/docs/structured-logging).\n\n### StackdriverJsonFormatter\n\nA opinionated log formatter which logs all log statements in a\nstructured JSON format, example:\n\n```json\n{\n    \"message\": \"something happened\",\n    \"filename\": \"log_utils.py\",\n    \"lineno\": 52,\n    \"timestamp\": \"2019-03-22T09:26:21.084950\",\n    \"severity\": \"INFO\",\n    \"thread\": 139978714621760,\n    \"pid\": 4984,\n    \"my_id\": 123\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnivanorge%2Fnivacloud-logging","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnivanorge%2Fnivacloud-logging","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnivanorge%2Fnivacloud-logging/lists"}