{"id":19168393,"url":"https://github.com/bloomberg/datalake-query-db-consumer","last_synced_at":"2026-03-07T08:34:05.371Z","repository":{"id":39708125,"uuid":"426319167","full_name":"bloomberg/datalake-query-db-consumer","owner":"bloomberg","description":null,"archived":false,"fork":false,"pushed_at":"2022-05-27T12:40:13.000Z","size":79,"stargazers_count":3,"open_issues_count":0,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-10-04T05:02:53.972Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bloomberg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-09T17:13:48.000Z","updated_at":"2025-03-07T01:29:46.000Z","dependencies_parsed_at":"2022-09-26T18:41:08.556Z","dependency_job_id":null,"html_url":"https://github.com/bloomberg/datalake-query-db-consumer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bloomberg/datalake-query-db-consumer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fdatalake-query-db-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fdatalake-query-db-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fdatalake-query-db-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fdatalake-query-db-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bloomberg","download_url":"https://codeload.github.com/bloomberg/datalake-query-db-consumer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bloomberg%2Fdatalake-query-db-consumer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30209951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T05:23:27.321Z","status":"ssl_error","status_checked_at":"2026-03-07T05:00:17.256Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T09:42:30.565Z","updated_at":"2026-03-07T08:34:05.352Z","avatar_url":"https://github.com/bloomberg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Datalake Query DB Consumer\n\n`datalake-query-pg-consumer` is a python microservice that consumes datalake query events from a Kafka topic and stores them inside a relational database.\n\n## Menu\n\n- [Rationale](#rationale)\n- [Quick start](#quick-start)\n- [Building](#building)\n- [Installation](#installation)\n- [Contributions](#contributions)\n- [License](#license)\n- [Code of Conduct](#code-of-conduct)\n- [Security Vulnerability Reporting](#security-vulnerability-reporting)\n\n## Rationale\n\nThe purpose of this service is to provide a way of consuming Kafka messages produced by [datalake-query-ingester](https://github.com/bloomberg/datalake-query-ingester) and storing them in a relational database for long-term storage and analysis.\n\nThis is part of a datalake query metadata ingestion and analysis pipeline.\n\n## Quick Start\n\nTo run the service locally, along with supporting services for testing, just run `docker-compose up datalakequerydbconsumer`.\nSimilarly, for tests run `docker-compose run tests`.\n\n## Building\n:warning: WARNING: This follows sqlalchemy's approach of not packaging DbAPIs, instead letting the user install\nthe appropiate ones for their use case. This is specified in Dockerfile by the `SQLALCHEMY_DEPENDENCIES`\nargument.\n\nTo build, run `docker build --build-arg SQLALCHEMY_DEPENDENCIES=psycopg2-binary -f Dockerfile -t bloomberg/datalakequerydbconsumer:latest-postgresql .`\n\nOR\n\nRun `docker-compose build datalakequerydbconsumer`\n\n## Installation\n\nThis is meant to be used with [Trino](https://github.com/trinodb/trino) and models data based on Trino's query metrics. This has been tested with [Trino 363](https://github.com/trinodb/trino/releases/tag/363), backwards or forwards compatibility is not guaranteed.\n\nThe service is ment to be deployed with k8s. Configuration is passed with environment variables:\n- KAFKA_BROKERS\n- DATALAKEQUERYDBCONSUMER_KAFKA_TOPIC\n- DATALAKEQUERYDBCONSUMER_KAFKA_GROUP_ID\n- DATALAKEQUERYDBCONSUMER_DB_URL\n\nAn example config can be found in `docker-compose.yaml \u003e datalakequerydbconsumer`.\n\n## Contributions\n\nWe :heart: contributions.\n\nHave you had a good experience with this project? Why not share some love and contribute code, or just let us know about any issues you had with it?\n\nWe welcome issue reports [here](../../issues); be sure to choose the proper issue template for your issue, so that we can be sure you're providing the necessary information.\n\nBefore sending a [Pull Request](../../pulls), please make sure you read our\n[Contribution Guidelines](https://github.com/bloomberg/.github/blob/master/CONTRIBUTING.md).\n\n## License\n\nPlease read the [LICENSE](LICENSE) file.\n\n## Code of Conduct\n\nThis project has adopted a [Code of Conduct](https://github.com/bloomberg/.github/blob/master/CODE_OF_CONDUCT.md).\nIf you have any concerns about the Code, or behavior which you have experienced in the project, please\ncontact us at opensource@bloomberg.net.\n\n## Security Vulnerability Reporting\n\nIf you believe you have identified a security vulnerability in this project, please send email to the project\nteam at opensource@bloomberg.net, detailing the suspected issue and any methods you've found to reproduce it.\n\nPlease do NOT open an issue in the GitHub repository, as we'd prefer to keep vulnerability reports private until\nwe've had an opportunity to review and address them.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbloomberg%2Fdatalake-query-db-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbloomberg%2Fdatalake-query-db-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbloomberg%2Fdatalake-query-db-consumer/lists"}