https://github.com/xlfe/pyjdbq
The easiest way to ship journald logs to Google BigQuery
https://github.com/xlfe/pyjdbq
bigquery journald journald-logs logging security
Last synced: 10 months ago
JSON representation
The easiest way to ship journald logs to Google BigQuery
- Host: GitHub
- URL: https://github.com/xlfe/pyjdbq
- Owner: xlfe
- License: agpl-3.0
- Created: 2016-08-24T09:05:01.000Z (almost 10 years ago)
- Default Branch: master
- Last Pushed: 2017-01-21T04:49:34.000Z (over 9 years ago)
- Last Synced: 2025-03-11T13:25:05.751Z (over 1 year ago)
- Topics: bigquery, journald, journald-logs, logging, security
- Language: Python
- Homepage:
- Size: 25.4 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## pyjdbq
## Python journald to Google Big Query remote logger
### Installation
Create a Big Query table using schema.txt (default is to use the database "logs" and table "pyjdbq")
These installation instructions should work on CentOS/RHEL
Prepare the system by installing dependencies and creating a pyjdbq user
with home directory /opt/pyjdbq and adding them to the systemd-journal group
(which means they can read all journald entries)
```bash
sudo dnf install python-systemd unzip
sudo useradd -d /opt/pyjdbq -G systemd-journal -m -s /sbin/nologin pyjdbq
sudo -u pyjdbq pip install google-api-python-client pytz tzlocal -t /opt/pyjdbq/
```
Download pyjdbq and move it into /opt/pyjdbq
Put your JSON Google Big Query service account credentials.json file into the same directory
```bash
curl -LOk https://github.com/xlfe/pyjdbq/archive/master.zip
unzip master.zip
sudo cp pyjdbq-master/* /opt/pyjdbq
sudo mv credentials.json /opt/pyjdbq
sudo chown -R pyjdbq: /opt/pyjdbq
```
Finally install the systemd script (you should edit it to set options)
Available options:
```bash
[user@server ~]$ python pyjdbq.py --help
usage: pyjdbq.py [-h] [--dataset DATASET] [--table TABLE] [--creds CREDS]
[--cursor CURSOR] [--count COUNT] [--timeout TIMEOUT]
[--debug]
pyjdbq (journald to big query)
stream journald logs into a biq query table
optional arguments:
-h, --help show this help message and exit
--dataset DATASET BigQuery dataset
--table TABLE BigQuery table
--creds CREDS JSON credentials file for BigQuery
--cursor CURSOR Full path to store the cursor file
--count COUNT Events to store before inserting
--timeout TIMEOUT Seconds to wait for COUNT logs before inserting anyway
--debug Extra verbose
```
Install the systemd unit file and check your journald logs to make sure there's no errors
```bash
sudo mv /opt/pyjdbq/pyjdbq.service /etc/systemd/system/
sudo systemctl enable pyjdbq
sudo systemctl start pyjdbq; sudo journalctl -f
```