{"id":18456225,"url":"https://github.com/khezen/bulklog","last_synced_at":"2025-09-02T13:40:16.406Z","repository":{"id":57523146,"uuid":"99691042","full_name":"khezen/bulklog","owner":"khezen","description":"collects, buffers, and outputs logs across multiple sources and destinations","archived":false,"fork":false,"pushed_at":"2020-04-17T10:30:18.000Z","size":671,"stargazers_count":5,"open_issues_count":3,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-23T06:25:13.840Z","etag":null,"topics":["elasticsearch","go","golang","log","logging"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/khezen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-08T12:37:05.000Z","updated_at":"2020-11-04T02:43:12.000Z","dependencies_parsed_at":"2022-08-28T11:20:58.688Z","dependency_job_id":null,"html_url":"https://github.com/khezen/bulklog","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khezen%2Fbulklog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khezen%2Fbulklog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khezen%2Fbulklog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khezen%2Fbulklog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/khezen","download_url":"https://codeload.github.com/khezen/bulklog/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247779796,"owners_count":20994569,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","go","golang","log","logging"],"created_at":"2024-11-06T08:10:47.521Z","updated_at":"2025-04-08T04:34:30.401Z","avatar_url":"https://github.com/khezen.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# *bulklog*\n\nCollects, buffers, and outputs logs across multiple sources and destinations.\n\n![icon.png](https://github.com/khezen/bulklog/raw/master/.doc/icon.png)\n\n*bulklog* is written in go and requires little resource.\n\n*bulklog* supports memory and [redis](https://redis.io/) buffering.\n*bulklog* also supports failover and can be set up for high availability.\n\n---\n\n## Concepts\n\n*bulklog* tries to structure data as JSON since it has enough structure to be accessible while providing felxibility.\n\n### Collection\n\nA collection is a set of declarative informations about how *bulklog* should process data.\n\n### Output\n\n*bulklog* outputs JSON docuemts to destinations such as Elasticsearch, MongoDB, etc...\n\n---\n\n## Install\n\n### Docker\n\n[docker.pkg.github.com/khezen/bulklog/bulklog](https://github.com/khezen/bulklog/packages)\n\n```bash\ndocker run -p 5017:5017 -v /etc/bulklog:/etc/bulklog docker.pkg.github.com/khezen/bulklog/bulklog:stable\n```\n\n#### Supported tags\n\n* `latest`\n* `2.0.1`, `2.0`, `2`, `stable`\n\n\n#### ENV\n\n| key | Description | Default Value|\n|---|---|---|\n|CONFIG_PATH|path to the configuration folder|/etc/bulklog|\n\n### Kubernetes\n\n#### Helm\n\nDeploy bulklog to a kubernetes cluster using Helm.\n\n```bash\nhelm repo add khezen https://khezen.github.com/charts\nhelm install khezen/bulklog --name bulklog\n```\n\n---\n\n## Config\n\nDefault [config.yaml](https://github.com/khezen/bulklog/raw/master/config.yaml).\n\n### Persistence\n\nPeristence is disabled by default in which case data is buffered in memory.\nIf enabled, it uses Redis(\u003e= 2.4) to persist documents buffer. \n[Learn how to tune Redis persistence](https://redis.io/topics/persistence) for your requirements. \n\n```yaml\npersistence:\n  enabled: true\n  redis:\n    endpoint: localhost:6379\n    password: changeme #(optional)\n    db: 0 #(optional, default:0)\n    idle_conn: 2 #(optional, default: 0)\n    max_conn: 10 #(optional, defaut: no limit)\n```\n\n### Output\n\nprovides declarative information about *bulklog* output.\n\n```yaml\noutput:\n  elasticsearch:\n    enabled: true\n    endpoint: localhost:9200\n    scheme: http\n#   aws_auth:\n#     access_key_id: changeme\n#     secret_access_key: changeme\n#     region: eu-west-1\n#   basic_auth:\n#     username: elastic\n#     password: changeme\n```\n\n*from version 2.0.0 bulklog supports Elasticsearch 7.0.0 and above*\n\n### Collections\n\nexamples:\n\n```yaml\ncollections:\n  - name: logs\n    flush_period: 5 seconds # hours|minutes|seconds|milliseconds\n    retention_period: 45 minutes\n    schema: {}\n```\n\n*bulklog* is schema free but we encourage you to provide some base structure since it might enbale output destination to process data more efficiently.\n\n```yaml\ncollections:\n  - name: logs\n    flush_period: 5 seconds # hours|minutes|seconds|milliseconds\n    retention_period: 45 minutes\n    shards: 5\n    replicas: 1\n    schema:\n      source: \n        type: string\n        max_length: 64\n      stream: \n        type: string\n        length: 6\n      event: \n        type: string\n      time:\n        type: datetime\n        date_format: 2006-01-02T15:04:05.999999999Z07:00\n```\n\nEven in the case above, *bulklog* remains schema free enabling log decoration with additional field.\n\n#### collection\n\n* **name**: `{collection name}`\n* **flush_period**: `{duration}`\n  * flush buffer to output every `{duration}`\n* **retention_period**: `{duration}`\n  * if an output is unavailable, **retention_period** set how long *bulklog* tries to output data to this output\n  * if the output is unavailable for too long, **retention_period** ensure that *bulklog* will not accumulate too much data and will be able to serve other outputs.\n* **shards**: the number of shards to allocate this collection to\n* **replicas**: the number of replicas to allocate this collection to\n* **schema**: `{map of fields by field name}`\n\n*[Learn more about Elasticsearch sharding and replication](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/scale.html)*.\n#### field\n\n* **type**: `{field type}`\n  * see [supported types](#supported-types)\n* **length**: `{field exact length}` (optional,string only)\n* **max_length**: `{field maximum length}` (optional, string only)\n* **date_format**: `{date time formatting}` (optional, datetime only)\n\n---\n\n## API\n\n### push document\n\n```http\nPOST bulklog/v1/{collectionName} HTTP/1.1\nContent-Type: application/json\n{\n  ...\n}\n\nHTTP/1.1 200 OK\n```\n\nexample:\n\n```http\nPOST bulklog/v1/logs HTTP/1.1\nContent-Type: application/json\n{\n  \"source\":\"service1\",\n  \"stream\": \"stderr\",\n  \"event\": \"divizion by zero\",\n  \"time\": \"2018-11-15T14:12:12Z\"\n}\n\n### push documents in batches\n\n```http\nPOST /v1/{collectionName}/batch HTTP/1.1\nContent-Type: application/json\n{...}\n{...}\n\nHTTP/1.1 200 OK\n```\n\nexample:\n\n```http\nPOST bulklog/v1/logs/batch HTTP/1.1\nContent-Type: application/json\n{\"source\":\"service1\",\"stream\": \"stderr\",\"event\": \"divizion by zero\",\"time\" : \"2019-01-13T19:30:12\"}\n{\"source\":\"service1\",\"stream\": \"stdout\",\"event\": \"successfully processed\",\"time\" : \"2019-01-13T19:35:12\"}\n\nHTTP/1.1 200 OK\n```\n\n### health\n\n```http\nGET bulklog/liveness HTTP/1.1\n\nHTTP/1.1 200 OK\n```\n\n```http\nGET bulklog/readiness HTTP/1.1\n\nHTTP/1.1 200 OK\n```\n\n---\n\n## supported types\n\n* **bool** : `True` or `False`\n\n* **unint8** : `0` to `255`\n* **uint16** : `0` to `65535`\n* **uint32** : `0` to `4294967295`\n* **unit64** : `0` to `18446744073709551615`\n\n* **int8** : `-128` to `127`\n* **int16** : `-32768` to `32767`\n* **int32** : `-2147483648` to `2147483647`\n* **int64** : `-9223372036854775808` to `9223372036854775807`\n\n* **float32** : `-3.40282346638528859811704183484516925440e+38` to `3.40282346638528859811704183484516925440e+38`\n* **float64** : `-1.797693134862315708145274237317043567981e+308` to `1.797693134862315708145274237317043567981e+308`\n\n* **string** : sequence of characters\n  * **lenght**: string exact length\n  * **max_length**: string maximum length\n\n* **datetime** : `1970-01-01T00:00:00.000000000Z` (example)\n  * *bulklog* doesn't check the date format. Most outputs accept any even if it defers from the configured one\n  * **date_format**: date format string\n    * `Mon Jan _2 15:04:05 2006`\n    * `Mon Jan _2 15:04:05 MST 2006`\n    * `Mon Jan 02 15:04:05 -0700 2006`\n    * `02 Jan 06 15:04 MST`\n    * `02 Jan 06 15:04 -0700`\n    * `Monday, 02-Jan-06 15:04:05 MST`\n    * `Mon, 02 Jan 2006 15:04:05 MST`\n    * `Mon, 02 Jan 2006 15:04:05 -0700`\n    * `2006-01-02T15:04:05Z07:00`\n    * `2006-01-02T15:04:05.999999999Z07:00` (**default**)\n    * `3:04PM`\n    * `Jan _2 15:04:05`\n    * `Jan _2 15:04:05.000`\n    * `Jan _2 15:04:05.000000`\n    * `Jan _2 15:04:05.000000000`\n    * `2006-01-02 15:04:05 MST`\n    * `2006-01-02 15:04:05.999999999 MST`\n\n* **object** : inner document\n\n---\n\n## Issues\n\nIf you have any problems or questions, please ask for help through a [GitHub issue](https://github.com/khezen/bulklog/issues).\n\n## Contributions\n\nHelp is always welcome! For example, documentation (like the text you are reading now) can always use improvement. There's always code that can be improved. If you ever see something you think should be fixed, you should own it. If you have no idea what to start on, you can browse the issues labeled with [help wanted](https://github.com/khezen/bulklog/labels/help%20wanted).\n\nAs a potential contributor, your changes and ideas are welcome at any hour of the day or night, weekdays, weekends, and holidays. Please do not ever hesitate to ask a question or send a pull request.\n\n[Code of conduct](https://github.com/khezen/bulklog/blob/master/CODE_OF_CONDUCT.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkhezen%2Fbulklog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkhezen%2Fbulklog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkhezen%2Fbulklog/lists"}