{"id":18010836,"url":"https://github.com/yokawasa/fluent-plugin-documentdb","last_synced_at":"2025-03-26T14:32:15.456Z","repository":{"id":56846920,"uuid":"49806000","full_name":"yokawasa/fluent-plugin-documentdb","owner":"yokawasa","description":"Azure DocumentDB output plugin for Fluentd","archived":false,"fork":false,"pushed_at":"2020-03-18T17:44:42.000Z","size":633,"stargazers_count":5,"open_issues_count":2,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-21T23:33:47.153Z","etag":null,"topics":["azure","azure-documentdb","documentdb","fluent-plugin","fluentd","ruby"],"latest_commit_sha":null,"homepage":"https://rubygems.org/gems/fluent-plugin-documentdb","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yokawasa.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-01-17T06:30:11.000Z","updated_at":"2021-02-08T20:41:28.000Z","dependencies_parsed_at":"2022-09-09T01:00:55.729Z","dependency_job_id":null,"html_url":"https://github.com/yokawasa/fluent-plugin-documentdb","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Ffluent-plugin-documentdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Ffluent-plugin-documentdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Ffluent-plugin-documentdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Ffluent-plugin-documentdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yokawasa","download_url":"https://codeload.github.com/yokawasa/fluent-plugin-documentdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245670896,"owners_count":20653447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","azure-documentdb","documentdb","fluent-plugin","fluentd","ruby"],"created_at":"2024-10-30T02:15:29.556Z","updated_at":"2025-03-26T14:32:15.042Z","avatar_url":"https://github.com/yokawasa.png","language":"Ruby","readme":"# Azure DocumentDB output plugin for Fluentd\n\nfluent-plugin-documentdb is a fluent plugin to output to Azure DocumentDB\n\n![fluent-plugin-documentdb overview](https://github.com/yokawasa/fluent-plugin-documentdb/raw/master/img/fluentd-azure-documentdb-collection.png)\n\n[NEWS] From fluent-plugin-documentdb-0.2.0, it supports partitioned collections, not only single-partition collections (See [Partitioning and scaling in Azure DocumentDB](https://azure.microsoft.com/en-us/documentation/articles/documentdb-partition-data/#single-partition-and-partitioned-collections) for partitioned collections and single-partition collection ).\n\n\n## Requirements\n\n| fluent-plugin-documentdb | fluentd | ruby |\n|------------------------|---------|------|\n| \u003e= 0.3.0 | \u003e= v0.14.15 | \u003e= 2.1 |\n|  \u003c 0.3.0 | \u003e= v0.12.0 | \u003e= 1.9 |\n\n## Installation\n\n    $ gem install fluent-plugin-documentdb\n\n## Configuration\n\n### DocumentDB\n\nTo use Microsoft Azure DocumentDB, you must create a DocumentDB database account using either the Azure portal, Azure Resource Manager templates, or Azure command-line interface (CLI). In addition, you must have a database and a collection to which fluent-plugin-documentdb writes event-stream out. Here are instructions:\n\n * Create a DocumentDB database account using [the Azure portal](https://azure.microsoft.com/en-us/documentation/articles/documentdb-create-account/), or [Azure Resource Manager templates and Azure CLI](https://azure.microsoft.com/en-us/documentation/articles/documentdb-automation-resource-manager-cli/)\n * [How to create a database for DocumentDB](https://azure.microsoft.com/en-us/documentation/articles/documentdb-create-database/)\n * [Create a DocumentDB collection](https://azure.microsoft.com/en-us/documentation/articles/documentdb-create-collection/)\n * [Partitioning and scaling in Azure DocumentDB](https://azure.microsoft.com/en-us/documentation/articles/documentdb-partition-data/)\n\n\n### Fluentd - fluent.conf\n  \n    \u003cmatch documentdb.*\u003e\n        @type documentdb\n        @log_level info\n        docdb_endpoint  DOCUMENTDB_ACCOUNT_ENDPOINT\n        docdb_account_key DOCUMENTDB_ACCOUNT_KEY\n        docdb_database  mydb\n        docdb_collection mycollection\n        auto_create_database true\n        auto_create_collection true\n        partitioned_collection true \n        partition_key PARTITION_EKY\n        offer_throughput 10100\n        time_format %s\n        localtime false\n        add_time_field true\n        time_field_name time\n        add_tag_field true\n        tag_field_name time\n    \u003c/match\u003e\n\n * **docdb\\_endpoint (required)** - Azure DocumentDB Account endpoint URI\n * **docdb\\_account\\_key (required)** - Azure DocumentDB Account key (master key). You must NOT set a read-only key\n * **docdb\\_database (required)** - DocumentDB database nameb\n * **docdb\\_collection (required)** - DocumentDB collection name\n * **auto\\_create\\_database (optional)** - Default:true. By default, DocumentDB database named **docdb\\_database** will be automatically created if it does not exist\n * **auto\\_create\\_collection (optional)** - Default:true. By default, DocumentDB collection named **docdb\\_collection** will be automatically created if it does not exist\n * **partitioned\\_collection (optional)** - Default:false. Set true if you want to create and/or store records to partitioned collection. Set false for single-partition collection\n * **partition\\_key (optional)** - Default:nil. Partition key must be specified for paritioned collection (partitioned\\_collection set to be true)\n * **offer\\_throughput (optional)** - Default:10100. Throughput for the collection expressed in units of 100 request units per second. This is only effective when you newly create a partitioned collection (ie. Both auto\\_create\\_collection and partitioned\\_collection are set to be true )\n * **localtime (optional)** - Default:false. By default, time record is inserted with UTC (Coordinated Universal Time). This option allows to use local time if you set localtime true\n * **time\\_format (optional)** -  Default:%s. Time format for a time field to be inserted. Default format is %s, that is unix epoch time. If you want it to be more human readable, set this %Y%m%d-%H:%M:%S, for example.\n * **add\\_time\\_field (optional)** - Default:true. This option allows to insert a time field to record\n * **time\\_field\\_name (optional)** - Default:time. Time field name to be inserted\n * **add\\_tag\\_field (optional)** - Default:true. This option allows to insert a tag field to record\n * **tag\\_field\\_name (optional)** - Default:tag. Tag field name to be inserted\n\n[note] @log_level is a fluentd built-in parameter (optional) that controls verbosity of logging: fatal|error|warn|info|debug|trace (See also [Logging of Fluentd](http://docs.fluentd.org/articles/logging#log-level))\n\n## Configuration examples\n\nfluent-plugin-documentdb will add **id** attribute which is UUID format and any other attributes of record automatically. In addition, it will add **time** and **tag** attributes if **add_time_field** and **add_tag_field** are true respectively. Please see 2 types of the plugin configurations example below - single-parition collection and partitioned collection. Source for fluentd to read is apache access log.\n\n### (1) Single-Partition Collection Case\n\n\u003cu\u003efluent.conf\u003c/u\u003e\n\n    \u003csource\u003e\n        @type tail                          # input plugin\n        path /var/log/apache2/access.log   # monitoring file\n        pos_file /tmp/fluentd_pos_file     # position file\n        format apache                      # format\n        tag documentdb.access              # tag\n    \u003c/source\u003e\n    \n    \u003cmatch documentdb.*\u003e\n        @type documentdb\n        docdb_endpoint https://yoichikademo.documents.azure.com:443/\n        docdb_account_key Tl1xykQxnExUisJ+BXwbbaC8NtUqYVE9kUDXCNust5aYBduhui29Xtxz3DLP88PayjtgtnARc1PW+2wlA6jCJw==\n        docdb_database mydb\n        docdb_collection my-single-partition-collection\n        auto_create_database true\n        auto_create_collection true\n        partitioned_collection true \n        localtime true\n        time_format %Y%m%d-%H:%M:%S\n        add_time_field true\n        time_field_name time\n        add_tag_field true\n        tag_field_name tag\n    \u003c/match\u003e\n\n### (2) Partitioned Collection Case\n\n\u003cu\u003efluent.conf\u003c/u\u003e\n\n    \u003csource\u003e\n        @type tail                          # input plugin\n        path /var/log/apache2/access.log   # monitoring file\n        pos_file /tmp/fluentd_pos_file     # position file\n        format apache                      # format\n        tag documentdb.access              # tag\n    \u003c/source\u003e\n    \n    \u003cmatch documentdb.*\u003e\n        @type documentdb\n        docdb_endpoint https://yoichikademo.documents.azure.com:443/\n        docdb_account_key Tl1xykQxnExUisJ+BXwbbaC8NtUqYVE9kUDXCNust5aYBduhui29Xtxz3DLP88PayjtgtnARc1PW+2wlA6jCJw==\n        docdb_database mydb\n        docdb_collection my-partitioned-collection\n        auto_create_database true\n        auto_create_collection true\n        partitioned_collection true \n        partition_key host\n        offer_throughput 10100\n        localtime true\n        time_format %Y%m%d-%H:%M:%S\n        add_time_field true\n        time_field_name time\n        add_tag_field true\n        tag_field_name tag\n    \u003c/match\u003e\n\n\n## Sample inputs and expected records\n\nAn expected output record for sample input will be like this:\n\n\u003cu\u003eSample Input (apache access log)\u003c/u\u003e\n\n    125.212.152.166 - - [17/Jan/2016:05:03:25 +0000] \"GET /foo/bar/test.html HTTP/1.1\" 304 179 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36\"\n\n\n\u003cu\u003eOutput Record\u003c/u\u003e\n\n    {\n        id :  d2b2ece8-b948-41ae-a894-0ed1266e242a,\n        host :  125.211.152.166,\n        user :  -,\n        method :  GET,\n        path :  /foo/bar/test.html,\n        code :  304,\n        size :  179,\n        referer :  -,\n        agent :  Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36,\n        time :  20160117-05:03:25,\n        tag :  documentdb.access\n    }  \n\n## Tests\n### Running test code\n    $ git clone https://github.com/yokawasa/fluent-plugin-documentdb.git\n    $ cd fluent-plugin-documentdb\n    \n    # edit CONFIG params of test/plugin/test_documentdb.rb \n    $ vi test/plugin/test_documentdb.rb\n    \n    # run test \n    $ rake test\n\n### Creating package, running and testing locally \n    $ rake build\n    $ rake install:local\n     \n    # running fluentd with your fluent.conf\n    $ fluentd -c fluent.conf -vv \u0026\n     \n    # send test apache requests for testing plugin ( only in the case that input source is apache access log )\n    $ ab -n 5 -c 2 http://localhost/foo/bar/test.html\n\n## TODOs\n * Support automatic data expiration with TTL (Time-to-Live ). See [Expire data in DocumentDB collections automatically with time to live](https://azure.microsoft.com/en-us/documentation/articles/documentdb-time-to-live/)\n\n\n## Change log\n* [Changelog](ChangeLog.md)\n\n## Links\n\n* http://yokawasa.github.io/fluent-plugin-documentdb\n* https://rubygems.org/gems/fluent-plugin-documentdb\n* [Collecting logs into Azure DocumentDB using fluent-plugin-documentdb](http://unofficialism.info/posts/collecting-logs-into-azure-documentdb-using-fluent-plugin-documentdb/)\n* [fluent-plugin-documentdb supports Partitioned collections](http://unofficialism.info/posts/fluent-plugin-documentdb-supports-partitioned-collections/)\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/yokawasa/fluent-plugin-documentdb.\n\n## Copyright\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eCopyright\u003c/td\u003e\u003ctd\u003eCopyright (c) 2016- Yoichi Kawasaki\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eLicense\u003c/td\u003e\u003ctd\u003eApache License, Version 2.0\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyokawasa%2Ffluent-plugin-documentdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyokawasa%2Ffluent-plugin-documentdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyokawasa%2Ffluent-plugin-documentdb/lists"}