{"id":15970997,"url":"https://github.com/rainbowdashlabs/log-aggregator","last_synced_at":"2025-06-10T13:33:59.185Z","repository":{"id":205295964,"uuid":"713898489","full_name":"rainbowdashlabs/log-aggregator","owner":"rainbowdashlabs","description":"simple log aggregator that dumps json strings received via udp into a mongodb. Supports simple validation and log retention","archived":false,"fork":false,"pushed_at":"2023-11-03T18:24:36.000Z","size":39,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-04T15:12:37.483Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rainbowdashlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-03T13:25:37.000Z","updated_at":"2023-11-03T14:58:42.000Z","dependencies_parsed_at":"2024-10-30T04:29:11.533Z","dependency_job_id":null,"html_url":"https://github.com/rainbowdashlabs/log-aggregator","commit_stats":null,"previous_names":["rainbowdashlabs/log-appender","rainbowdashlabs/log-aggregator"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rainbowdashlabs%2Flog-aggregator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rainbowdashlabs%2Flog-aggregator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rainbowdashlabs%2Flog-aggregator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rainbowdashlabs%2Flog-aggregator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rainbowdashlabs","download_url":"https://codeload.github.com/rainbowdashlabs/log-aggregator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rainbowdashlabs%2Flog-aggregator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259085350,"owners_count":22803185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-07T20:05:58.851Z","updated_at":"2025-06-10T13:33:59.162Z","avatar_url":"https://github.com/rainbowdashlabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Log Aggregator\n\nA very simple log aggregator that dumps json strings received via udp into a MongoDB.\n\nHas several configurations options like global log retention, field validation and also supports indices with specific settings.\n\n## Configuration\n\n### Mongo\n\nLog Aggregator requires a mongoDB to work.\nSet your credentials and stuff in the mongo section.\n\n```json\n{\n  \"mongo\": {\n    \"user\": \"root\",\n    \"password\": \"example\",\n    \"host\": \"mongodb\",\n    \"port\": 27017,\n    \"database\": \"log-aggregator\"\n  }\n}\n```\n\n### Socket\n\nThe socket is defined here. You can set the host and port.\n\n```json\n{\n  \"socket\": {\n    \"host\": \"0.0.0.0\",\n    \"port\": 8888\n  }\n}\n```\n\n### Format\n\nThe format section allows to set format requirements for the incoming json objects.\n\n```json\n{\n  \"format\": {\n    \"required_keys\": [],\n    \"restricted_values\": {\n      \"index\": [\n        \"example\"\n      ]\n    }\n  }\n}\n```\n\n#### Required keys\n\nRequired keys is an array of keys that are expected at the toplevel of the json object.\n\n#### Restricted Values\n\nRestricted values is a dictionary which contains a list of valid values for each key.\n\n### Retention\n\nRetention modifies the time after which a log is deleted. Retention is optional and turned off by default.\n\n```json\n{\n  \"retention\": {\n    \"active\": false,\n    \"years\": 0,\n    \"months\": 0,\n    \"days\": 30,\n    \"hours\": 0,\n    \"seconds\": 0\n  }\n}\n```\n\n### Indices\n\nIndices allows to enforce retention and formating rules on a per index level.\n\n```json\n{\n  \"indices\": {\n    \"block_unkown\": false,\n    \"index_field\": \"_index\",\n    \"indices\": {\n      \"example\": {\n        \"retention\": {\n          \"active\": false,\n          \"years\": 0,\n          \"months\": 0,\n          \"days\": 30,\n          \"hours\": 0,\n          \"seconds\": 0\n        },\n        \"format\": {\n          \"required_keys\": [],\n          \"restricted_values\": {}\n        }\n      }\n    }\n  }\n}\n```\n\nIndices are identified by value of the `_index` field at the toplevel of the object.\nThe field can be changed by changing `index_field`.\n**Please be aware that changing that value afterward will break the application**\n\nThe `block_unkown` will block every record that is missing the index field or is not registered under indices.\n\nThe `indices` section contains a dict which an entry for each index with retention and format settings.\n\nIf an index is registered here, global rules for retention and format do no longer apply.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frainbowdashlabs%2Flog-aggregator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frainbowdashlabs%2Flog-aggregator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frainbowdashlabs%2Flog-aggregator/lists"}