{"id":13936637,"url":"https://github.com/techjacker/systemdlogger","last_synced_at":"2025-03-25T19:32:31.005Z","repository":{"id":50214409,"uuid":"77930273","full_name":"techjacker/systemdlogger","owner":"techjacker","description":"Exports systemd logs to an external service, eg cloudwatch, elasticsearch","archived":false,"fork":false,"pushed_at":"2021-06-01T21:44:14.000Z","size":91,"stargazers_count":90,"open_issues_count":3,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-18T19:52:43.034Z","etag":null,"topics":["aws","cloudwatch","elasticsearch","kibana","logging","logs","systemd"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/techjacker.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-01-03T15:42:09.000Z","updated_at":"2024-01-04T16:10:14.000Z","dependencies_parsed_at":"2022-09-26T20:52:31.097Z","dependency_job_id":null,"html_url":"https://github.com/techjacker/systemdlogger","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techjacker%2Fsystemdlogger","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techjacker%2Fsystemdlogger/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techjacker%2Fsystemdlogger/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/techjacker%2Fsystemdlogger/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/techjacker","download_url":"https://codeload.github.com/techjacker/systemdlogger/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245530448,"owners_count":20630563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloudwatch","elasticsearch","kibana","logging","logs","systemd"],"created_at":"2024-08-07T23:02:52.624Z","updated_at":"2025-03-25T19:32:30.629Z","avatar_url":"https://github.com/techjacker.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/techjacker/systemdlogger.svg?branch=master)](https://travis-ci.org/techjacker/systemdlogger)\n\n# systemdlogger\n\n- Lightweight, python only, logstash alternative\n- Exports systemd journald logs to an external service, eg cloudwatch, elasticsearch\n- Use with a cron job\n- Python 3.4+\n\n-----------------------------------------------------------\n## Installation\n\n#### Install Dependencies\n[python-systemd](https://github.com/systemd/python-systemd)\n```\n# Fedora/RHEL/CentOS\n$ dnf install python-systemd python3-systemd\n```\nOR\n```\n# Debian/Ubuntu/Mint\n$ apt-get install python-systemd python3-systemd\n```\n\n\n#### Install from pip\n```$ pip install systemdlogger```\n\n-----------------------------------------------------------\n## Usage\n\n```Shell\n$ systemdlogger -h\nusage: systemdlogger [-h] config\n\nExports systemd logs to different storage backends, eg\ncloudwatch/elasticsearch.\n\npositional arguments:\n  config      path to config file\n\noptional arguments:\n  -h, --help  show this help message and exit\n```\n\n#### Example usage\n\nSee example config files below.\n\n```$ systemdlogger config.json```\n\n\n#### Recommended Usage - Cron Job Runing Every Minute\n\n```*/1 * * * * . /etc/webserver.env; export ENV=$NODE_ENV; /usr/local/bin/systemdlogger /path/to/config.json \u003e/etc/logs/systemdlogger.log 2\u003e\u00261```\n\n\n-----------------------------------------------------------\n## Config\n\n[Full example config](tests/fixtures/config.json) that includes extra optional properties.\n\n#### Example Cloudwatch Config - just required properties\n\n```JavaScript\n{\n    \"systemd\": {\n        \"unit\": \"webserver\"\n    },\n    \"backends\": {\n        \"cloudwatch\": {\n            \"log_group_name\": \"log_group_name\",\n            \"log_stream_name\": \"log_stream_name\"\n        }\n    }\n}\n```\n\n#### Example Elasticsearch Config - just required properties\n\nIf the elasticsearch endpoint does not use port `9200` you need to explicitly state it as shown below:\n\n```JavaScript\n{\n    \"systemd\": {\n        \"unit\": \"webserver\"\n    },\n    \"backends\": {\n        \"elasticsearch\": {\n            \"doctype\": \"webserver\",\n            \"hosts\": [\"search-applogs-blahiy7jyhmqwerfnrfg9trdz4.eu-west-1.es.amazonaws.com:80\"]\n        }\n    }\n}\n```\n\n\n#### Environment Variable Interpolation\n\nEnvironment variables will be interpolated when the config is loaded.\n\n```$ ENV=uat FOO=bar systemdlogger config.json```\n\n```JavaScript\n// config.json\n{\n    \"systemd\": {\n        \"unit\": \"webserver\"\n    },\n    \"backends\": {\n        \"cloudwatch\": {\n            \"log_group_name\": \"$ENV-myapp\",\n            \"log_stream_name\": \"$FOO-myservice\"\n        }\n    }\n}\n```\n\nIs loaded as:\n\n```JavaScript\n        ...\n        \"cloudwatch\": {\n            \"log_group_name\": \"uat-myapp\",\n            \"log_stream_name\": \"bar-myservice\"\n        }\n```\n\n\n-----------------------------------------------------------\n## Cloudwatch Backend\n\nAppends EC2 instance id to logstream name, eg:\n\n```\nLog Groups \u003e webapp-uat \u003e webapp-i-06e2a5d847e0d532f\n```\n\n-----------------------------------------------------------\n## Elasticseach Backend\n\nCreates daily indexes for logs in `logs-YYYY-MM-DD` format, eg:\n```\nlogs-16-08-25\nlogs-16-08-26\n```\n\n### Kibana Setup\n\nConfigure Kibana index pattern:\n\n- Go to Kibana index settings URL eg, `http://\u003cyour_elasticsearch_domain\u003e/_plugin/kibana/#/settings/indices/?_g=()`\n- Select both checkboxes.\n- Set `Index pattern interval` to `Daily`\n- Enter `[logs-]YYYY-MM-DD` for the index name\n- Select `timestamp` for the timestamp field\n- Click on `Create`\n\n![Kibana Index Setup](docs/kibana_index_setup.png?raw=true \"Kibana Index Setup\")\n\n\n-----------------------------------------------------------\n## Development Setup\n\n```\n$ make setup\n$ source env/bin/activate\n$ make deps\n```\n\n-----------------------------------------------------------\n\n## Unit Tests\n\n#### Install dependencies\n```$ pip install -r requirements.txt```\n\n#### Run Tests\n```$ make test```\n\n-----------------------------------------------------------\n\n## Integration Tests\n\nRun against elasticsearch docker container.\n\n```\n$ docker-compose up -d\n$ make test-integration\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechjacker%2Fsystemdlogger","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftechjacker%2Fsystemdlogger","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechjacker%2Fsystemdlogger/lists"}