{"id":16626174,"url":"https://github.com/xlfe/pyjdbq","last_synced_at":"2025-08-25T07:16:05.293Z","repository":{"id":142075717,"uuid":"66448967","full_name":"xlfe/pyjdbq","owner":"xlfe","description":"The easiest way to ship journald logs to Google BigQuery","archived":false,"fork":false,"pushed_at":"2017-01-21T04:49:34.000Z","size":26,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-11T13:25:05.751Z","etag":null,"topics":["bigquery","journald","journald-logs","logging","security"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xlfe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-08-24T09:05:01.000Z","updated_at":"2024-05-30T15:20:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"a3511231-4df6-40f8-b1be-209ac4830dc1","html_url":"https://github.com/xlfe/pyjdbq","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/xlfe/pyjdbq","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlfe%2Fpyjdbq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlfe%2Fpyjdbq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlfe%2Fpyjdbq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlfe%2Fpyjdbq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xlfe","download_url":"https://codeload.github.com/xlfe/pyjdbq/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlfe%2Fpyjdbq/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272022138,"owners_count":24860119,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-25T02:00:12.092Z","response_time":1107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","journald","journald-logs","logging","security"],"created_at":"2024-10-12T04:08:59.950Z","updated_at":"2025-08-25T07:16:05.260Z","avatar_url":"https://github.com/xlfe.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## pyjdbq\n\n## Python journald to Google Big Query remote logger\n\n### Installation\n\n\nCreate a Big Query table using schema.txt (default is to use the database \"logs\" and table \"pyjdbq\")\n\nThese installation instructions should work on CentOS/RHEL\n\nPrepare the system by installing dependencies and creating a pyjdbq user\nwith home directory /opt/pyjdbq and adding them to the systemd-journal group\n(which means they can read all journald entries)\n\n```bash\nsudo dnf install python-systemd unzip\nsudo useradd -d /opt/pyjdbq -G systemd-journal -m -s /sbin/nologin pyjdbq\nsudo -u pyjdbq pip install google-api-python-client pytz tzlocal -t /opt/pyjdbq/\n```\n\nDownload pyjdbq and move it into /opt/pyjdbq\nPut your JSON Google Big Query service account credentials.json file into the same directory\n\n```bash\n\ncurl -LOk https://github.com/xlfe/pyjdbq/archive/master.zip\nunzip master.zip\nsudo cp pyjdbq-master/* /opt/pyjdbq\nsudo mv credentials.json /opt/pyjdbq\nsudo chown -R pyjdbq: /opt/pyjdbq\n```\n\nFinally install the systemd script (you should edit it to set options)\n\nAvailable options:\n\n```bash\n[user@server ~]$ python pyjdbq.py --help\nusage: pyjdbq.py [-h] [--dataset DATASET] [--table TABLE] [--creds CREDS]\n                 [--cursor CURSOR] [--count COUNT] [--timeout TIMEOUT]\n                 [--debug]\n\npyjdbq (journald to big query)\n\nstream journald logs into a biq query table\n\noptional arguments:\n  -h, --help         show this help message and exit\n  --dataset DATASET  BigQuery dataset\n  --table TABLE      BigQuery table\n  --creds CREDS      JSON credentials file for BigQuery\n  --cursor CURSOR    Full path to store the cursor file\n  --count COUNT      Events to store before inserting\n  --timeout TIMEOUT  Seconds to wait for COUNT logs before inserting anyway\n  --debug            Extra verbose\n```\n\nInstall the systemd unit file and check your journald logs to make sure there's no errors\n\n```bash\n\nsudo mv /opt/pyjdbq/pyjdbq.service /etc/systemd/system/\nsudo systemctl enable pyjdbq\nsudo systemctl start pyjdbq; sudo journalctl -f\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxlfe%2Fpyjdbq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxlfe%2Fpyjdbq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxlfe%2Fpyjdbq/lists"}