{"id":23074497,"url":"https://github.com/iamazeem/fluent-plugin-http-cwm","last_synced_at":"2025-06-19T08:03:55.413Z","repository":{"id":46056563,"uuid":"323902948","full_name":"iamazeem/fluent-plugin-http-cwm","owner":"iamazeem","description":"fluentd HTTP Input Plugin for CloudWebManage Logging Component (https://github.com/CloudWebManage)","archived":false,"fork":false,"pushed_at":"2024-09-24T05:47:56.000Z","size":49,"stargazers_count":1,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-12-01T00:29:21.148Z","etag":null,"topics":["fluentd","fluentd-input-plugin","fluentd-plugin","http","logging","logging-and-metrics","minio"],"latest_commit_sha":null,"homepage":"https://github.com/iamAzeem/fluent-plugin-http-cwm","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iamazeem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-23T12:59:32.000Z","updated_at":"2023-08-28T11:30:18.000Z","dependencies_parsed_at":"2022-09-12T11:13:49.180Z","dependency_job_id":null,"html_url":"https://github.com/iamazeem/fluent-plugin-http-cwm","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamazeem%2Ffluent-plugin-http-cwm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamazeem%2Ffluent-plugin-http-cwm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamazeem%2Ffluent-plugin-http-cwm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iamazeem%2Ffluent-plugin-http-cwm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iamazeem","download_url":"https://codeload.github.com/iamazeem/fluent-plugin-http-cwm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229929978,"owners_count":18146426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd","fluentd-input-plugin","fluentd-plugin","http","logging","logging-and-metrics","minio"],"created_at":"2024-12-16T08:28:40.809Z","updated_at":"2024-12-16T08:28:41.443Z","avatar_url":"https://github.com/iamazeem.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fluent-plugin-http-cwm\n\n![GitHub Workflow Status (branch)](https://img.shields.io/github/workflow/status/iamAzeem/fluent-plugin-http-cwm/ci/main?label=build\u0026style=flat-square)\n![GitHub Workflow Status (branch)](https://img.shields.io/github/workflow/status/iamAzeem/fluent-plugin-http-cwm/publish/main?label=publish\u0026style=flat-square)\n![GitHub release (latest by date)](https://img.shields.io/github/v/release/iamAzeem/fluent-plugin-http-cwm?style=flat-square)\n[![License: Apache-2.0](https://img.shields.io/badge/license-Apache-blue.svg?style=flat-square)](https://github.com/iamAzeem/fluent-plugin-http-cwm/blob/master/LICENSE)\n\n[![RubyGems Downloads](https://img.shields.io/gem/dt/fluent-plugin-http-cwm?color=blue\u0026style=flat-square)](https://rubygems.org/gems/fluent-plugin-http-cwm)\n![Lines of code](https://img.shields.io/tokei/lines/github/iamAzeem/fluent-plugin-http-cwm?label=LOC\u0026style=flat-square)\n![GitHub code size in bytes](https://img.shields.io/github/languages/code-size/iamAzeem/fluent-plugin-http-cwm?style=flat-square)\n![GitHub repo size](https://img.shields.io/github/repo-size/iamAzeem/fluent-plugin-http-cwm?style=flat-square)\n\n- [Overview](#overview)\n- [Installation](#installation)\n  - [RubyGems](#rubygems)\n  - [Bundler](#bundler)\n- [Configuration](#configuration)\n  - [`\u003credis\u003e` section (optional) (single)](#redis-section-optional-single)\n  - [Sample Configuration](#sample-configuration)\n- [Contribute](#contribute)\n- [Publish the gem](#publish-the-gem)\n- [License](#license)\n\n## Overview\n\n[Fluentd](https://fluentd.org/) HTTP input plugin for\n[CloudWebManage](https://github.com/CloudWebManage) Logging Component.\n\nThis plugin:\n\n1. receives the incoming JSON logs from MinIO using an HTTP endpoint,\n2. validates the required JSON fields,\n3. aggregates the logging metrics,\n4. flushes the aggregated metrics to the configured Redis instance; and,\n5. routes logs to the configured log targets e.g. S3, ElasticSearch, etc.\n\n```text\n  +------------------+\n  |       MinIO      |\n  +------------------+\n            |\n            | JSON\n            | logs\n            v\n  +------------------+\n  |     fluentd      |\n  |                  |\n  | +--------------+ |                   +-----------------+\n  | |   http_cwm   | |     [metrics]     |      Redis      |\n  | |   (input)    |--------------------\u003e|      Server     |\n  | +--------------+ |                   +-----------------+\n  |                  |\n  | +--------------+ |                   +-----------------+\n  | |      s3      | |     [raw logs]    |       S3        |\n  | |   (output)   |--------------------\u003e|   (log target)  |\n  | +--------------+ |                   +-----------------+\n  |                  |\n  | +--------------+ |                   +-----------------+\n  | |elasticsearch | |     [raw logs]    |  ElasticSearch  |\n  | |   (output)   |--------------------\u003e|  (log target)   |\n  | +--------------+ |                   +-----------------+\n  |                  |\n  +------------------+\n```\n\nThe following metrics are aggregated:\n\n| metric              | description                                        |\n| :------------------ | :------------------------------------------------- |\n| `bytes_in`          | size of Request header and its Content-Length      |\n| `bytes_out`         | size of Response header and its Content-Length     |\n| `num_requests_in`   | count of APIs [WebUpload, PutObject, DeleteObject] |\n| `num_requests_out`  | count of APIs [WebDownload, GetObject]             |\n| `num_requests_misc` | count of APIs other than `in` and `out`            |\n\n## Installation\n\n### RubyGems\n\n```shell\ngem install fluent-plugin-http-cwm\n```\n\n### Bundler\n\nAdd the following line to your Gemfile:\n\n```ruby\ngem 'fluent-plugin-http-cwm'\n```\n\nAnd then execute:\n\n```shell\nbundle\n```\n\n## Configuration\n\n- `host` (string) (optional): The address to bind to.\n  - Default value: `localhost`.\n- `port` (integer) (optional): The port to listen to.\n  - Default value: `8080`.\n- `tag` (string) (required): The tag for the event.\n\n### `\u003credis\u003e` section (optional) (single)\n\n- `host` (string) (optional): The address of Redis server.\n  - Default value: `localhost`.\n- `port` (integer) (optional): The port of Redis server.\n  - Default value: `6379`.\n- `db` (integer) (optional): The db to use.\n  - Default value: `0`.\n- `grace_period` (time) (optional): The grace period for last action update.\n  - Default value: `300s`.\n- `flush_interval` (time) (optional): The flush interval to send metrics.\n  - Default value: `300s`.\n- `last_update_prefix` (string) (optional): The prefix for last update key.\n  - Default value: `deploymentid:last_action`.\n- `metrics_prefix` (string) (optional): The prefix for metrics key.\n  - Default value: `deploymentid:minio-metrics`.\n\n### Sample Configuration\n\n```text\n# Endpoint for incoming logs: http://host:port/\u003ctag\u003e\n\n# HTTP Input\n\u003csource\u003e\n  @type                   http_cwm\n  @id                     http_cwm_logs\n\n  host                    localhost\n  port                    8080\n\n  tag                     logs\n\n  \u003credis\u003e\n    host                  localhost\n    port                  6379\n    db                    0\n    grace_period          10s\n    flush_interval        10s\n    last_update_prefix    deploymentid:last_action\n    metrics_prefix        deploymentid:minio-metrics\n  \u003c/redis\u003e\n\u003c/source\u003e\n\n# Output e.g. ElasticSearch, S3, etc.\n\u003cmatch logs\u003e\n  @type                   elasticsearch\n  # ...\n\u003c/match\u003e\n```\n\nThe environment variables may also be used for the configuration.\n\nExample:\n\n```text\n\u003csource\u003e\n  @type                   http_cwm\n  @id                     http_cwm_logs\n\n  host                    \"#{ENV['HTTP_HOST']}\"\n  port                    \"#{ENV['HTTP_PORT']}\"\n\n  # ...\n\u003c/source\u003e\n```\n\n## Contribute\n\n- Fork the project.\n- Check out the latest `main` branch.\n- Create a feature or bugfix branch from `main`.\n- Commit and push your changes.\n- Make sure to add and run tests locally: `bundle exec rake test`.\n- Run Rubocop locally and fix all the lint warnings.\n- Make sure to update [Gemfile.lock](Gemfile.lock): `sudo bundle update`.\n- Submit the PR.\n\n## Publish the gem\n\nThe gem is published via the [publish.yml](.github/workflows/publish.yml)\nWorkflow on tagging. The tag must be of the format `v0.3.0`. This workflow\ndepends on the successful completion of the [ci.yml](.github/workflows/ci.yml)\nworkflow and then it looks for the tag. So, make sure that all the CI issues are\nresolved before creating a new tag. If there are issues while publishing the gem\ni.e. publish workflow doesn't work properly, you can delete and then recreate\nthe tag to retrigger this workflow.\n\n## License\n\n[Apache 2.0](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiamazeem%2Ffluent-plugin-http-cwm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiamazeem%2Ffluent-plugin-http-cwm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiamazeem%2Ffluent-plugin-http-cwm/lists"}