{"id":18010827,"url":"https://github.com/yokawasa/fluent-plugin-azuresearch","last_synced_at":"2025-06-23T01:35:39.724Z","repository":{"id":56846873,"uuid":"50757710","full_name":"yokawasa/fluent-plugin-azuresearch","owner":"yokawasa","description":"Azure Search Output Plugin for Fluentd","archived":false,"fork":false,"pushed_at":"2020-03-18T18:28:11.000Z","size":59,"stargazers_count":2,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-24T22:38:44.674Z","etag":null,"topics":["azure","azure-search","fluent-plugin","fluentd","ruby"],"latest_commit_sha":null,"homepage":"https://rubygems.org/gems/fluent-plugin-azuresearch","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yokawasa.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-01-31T03:51:13.000Z","updated_at":"2020-03-18T18:29:27.000Z","dependencies_parsed_at":"2022-09-09T01:00:28.864Z","dependency_job_id":null,"html_url":"https://github.com/yokawasa/fluent-plugin-azuresearch","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/yokawasa/fluent-plugin-azuresearch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Ffluent-plugin-azuresearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Ffluent-plugin-azuresearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Ffluent-plugin-azuresearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Ffluent-plugin-azuresearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yokawasa","download_url":"https://codeload.github.com/yokawasa/fluent-plugin-azuresearch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yokawasa%2Ffluent-plugin-azuresearch/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261396229,"owners_count":23152442,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","azure-search","fluent-plugin","fluentd","ruby"],"created_at":"2024-10-30T02:15:27.385Z","updated_at":"2025-06-23T01:35:34.711Z","avatar_url":"https://github.com/yokawasa.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Azure Search output plugin for Fluentd\n\nfluent-plugin-azuresearch is a fluent plugin to output to Azure Search\n\n## Requirements\n\n| fluent-plugin-azuresearch | fluentd | ruby |\n|------------------------|---------|------|\n| \u003e= 0.2.0 | \u003e= v0.14.15 | \u003e= 2.1 |\n|  \u003c 0.2.0 | \u003e= v0.12.0 | \u003e= 1.9 |\n\n## Installation\n\n    $ gem install fluent-plugin-azuresearch\n\n## Configuration\n\n### Azure Search\n\nTo use Microsoft Azure Search, you must create an Azure Search service in the Azure Portal. Also you must have an index, persisted storage of documents to which fluent-plugin-azuresearch writes event stream out. Here are instructions:\n\n * [Create a service](https://azure.microsoft.com/en-us/documentation/articles/search-create-service-portal/)\n * [Create an index](https://azure.microsoft.com/en-us/documentation/articles/search-what-is-an-index/)\n\n\n### Fluentd - fluent.conf\n\n    \u003cmatch azuresearch.*\u003e\n        @type azuresearch\n        @log_level info\n        endpoint   https://AZURE_SEARCH_ACCOUNT.search.windows.net\n        api_key    AZURE_SEARCH_API_KEY\n        search_index  messages\n        column_names id,user_name,message,tag,created_at\n        key_names postid,user,content,tag,posttime\n    \u003c/match\u003e\n  \n * **endpoint (required)** - Azure Search service endpoint URI\n * **api\\_key (required)** - Azure Search API key\n * **search\\_index (required)** - Azure Search Index name to insert records\n * **column\\_names (required)** - Column names in a target Azure search index. Each column needs to be separated by a comma.\n * **key\\_names (optional)** - Default:nil. Key names in incomming record to insert. Each key needs to be separated by a comma. ${time} is placeholder for Time.at(time).strftime(\"%Y-%m-%dT%H:%M:%SZ\"), and ${tag} is placeholder for tag. By default, **key\\_names** is as same as **column\\_names**\n\n[note] @log_level is a fluentd built-in parameter (optional) that controls verbosity of logging: fatal|error|warn|info|debug|trace (See also [Logging of Fluentd](http://docs.fluentd.org/articles/logging#log-level)) \n\n## Sample Configurations\n### Case1 - column_names is as same as key_names \n\nSuppose you have the following fluent.conf and azure search index schema:\n\n\u003cu\u003efluent.conf\u003c/u\u003e\n\n    \u003cmatch azuresearch.*\u003e\n        @type azuresearch\n        endpoint   https://yoichidemo.search.windows.net\n        api_key    2XX3D2456052A9AD21E54CB03C3ABF6A(dummy)\n        search_index  messages\n        column_names id,user_name,message,created_at\n    \u003c/match\u003e\n\n\u003cu\u003eAzure Search Schema: messages\u003c/u\u003e\n\n    {\n        \"name\": \"messages\",\n        \"fields\": [\n            { \"name\":\"id\", \"type\":\"Edm.String\", \"key\": true, \"searchable\": false },\n            { \"name\":\"user_name\", \"type\":\"Edm.String\" },\n            { \"name\":\"message\", \"type\":\"Edm.String\", \"filterable\":false, \"sortable\":false, \"facetable\":false, \"analyzer\":\"en.lucene\" },\n            { \"name\":\"created_at\", \"type\":\"Edm.DateTimeOffset\", \"facetable\":false}\n        ]\n    }\n\nThe plugin will write event stream out to Azure Ssearch like this:\n\n\u003cu\u003eInput event stream\u003c/u\u003e\n\n    { \"id\": \"1\", \"user_name\": \"taylorswift13\", \"message\":\"post by taylorswift13\", \"created_at\":\"2016-01-29T00:00:00Z\" },\n    { \"id\": \"2\", \"user_name\": \"katyperry\", \"message\":\"post by katyperry\", \"created_at\":\"2016-01-30T00:00:00Z\" },\n    { \"id\": \"3\", \"user_name\": \"ladygaga\", \"message\":\"post by ladygaga\", \"created_at\":\"2016-01-31T00:00:00Z\" }\n\n\n\u003cu\u003eSearch results\u003c/u\u003e\n\n    \"value\": [\n        { \"@search.score\": 1, \"id\": \"1\", \"user_name\": \"taylorswift13\", \"message\": \"post by taylorswift13\", \"created_at\": \"2016-01-29T00:00:00Z\" },\n        { \"@search.score\": 1, \"id\": \"2\", \"user_name\": \"katyperry\", \"message\": \"post by katyperry\", \"created_at\": \"2016-01-30T00:00:00Z\" },\n        { \"@search.score\": 1, \"id\": \"3\", \"user_name\": \"ladygaga\", \"message\": \"post by ladygaga\", \"created_at\": \"2016-01-31T00:00:00Z\" }\n    ]\n\n\n### Case2 - column_names is NOT as same as key_names\n\nSuppose you have the following fluent.conf and azure search index schema:\n\n\u003cu\u003efluent.conf\u003c/u\u003e\n\n    \u003cmatch azuresearch.*\u003e\n        @type azuresearch\n        endpoint   https://yoichidemo.search.windows.net\n        api_key    2XX3D2456052A9AD21E54CB03C3ABF6A(dummy)\n        search_index  messages\n        column_names id,user_name,message,created_at\n        key_names postid,user,content,posttime\n    \u003c/match\u003e\n\n\u003cu\u003eAzure Search Schema: messages\u003c/u\u003e\n\n    {\n        \"name\": \"messages\",\n        \"fields\": [\n            { \"name\":\"id\", \"type\":\"Edm.String\", \"key\": true, \"searchable\": false },\n            { \"name\":\"user_name\", \"type\":\"Edm.String\" },\n            { \"name\":\"message\", \"type\":\"Edm.String\", \"filterable\":false, \"sortable\":false, \"facetable\":false, \"analyzer\":\"en.lucene\" },\n            { \"name\":\"created_at\", \"type\":\"Edm.DateTimeOffset\", \"facetable\":false}\n        ]\n    }\n\nThe plugin will write event stream out to Azure Ssearch like this:\n\n\u003cu\u003eInput event stream\u003c/u\u003e\n\n    { \"postid\": \"1\", \"user\": \"taylorswift13\", \"content\":\"post by taylorswift13\", \"posttime\":\"2016-01-29T00:00:00Z\" },\n    { \"postid\": \"2\", \"user\": \"katyperry\", \"content\":\"post by katyperry\", \"posttime\":\"2016-01-30T00:00:00Z\" },\n    { \"postid\": \"3\", \"user\": \"ladygaga\", \"content\":\"post by ladygaga\", \"posttime\":\"2016-01-31T00:00:00Z\" }\n\n\n\u003cu\u003eSearch results\u003c/u\u003e\n\n    \"value\": [\n        { \"@search.score\": 1, \"id\": \"1\", \"user_name\": \"taylorswift13\", \"message\": \"post by taylorswift13\", \"created_at\": \"2016-01-29T00:00:00Z\" },\n        { \"@search.score\": 1, \"id\": \"2\", \"user_name\": \"katyperry\", \"message\": \"post by katyperry\", \"created_at\": \"2016-01-30T00:00:00Z\" },\n        { \"@search.score\": 1, \"id\": \"3\", \"user_name\": \"ladygaga\", \"message\": \"post by ladygaga\", \"created_at\": \"2016-01-31T00:00:00Z\" }\n    ]\n\n\n### Case3 - column_names is NOT as same as key_names, Plus, key_names includes ${time} and ${tag}\n\n\u003cu\u003efluent.conf\u003c/u\u003e\n\n    \u003cmatch azuresearch.*\u003e\n        @type azuresearch\n        endpoint   https://yoichidemo.search.windows.net\n        api_key    2XX3D2456052A9AD21E54CB03C3ABF6A(dummy)\n        search_index  messages\n        column_names id,user_name,message,tag,created_at\n        key_names postid,user,content,${tag},${time}\n    \u003c/match\u003e\n\n\u003cu\u003eAzure Search Schema: messages\u003c/u\u003e\n\n    {\n        \"name\": \"messages\",\n        \"fields\": [\n            { \"name\":\"id\", \"type\":\"Edm.String\", \"key\": true, \"searchable\": false },\n            { \"name\":\"user_name\", \"type\":\"Edm.String\" },\n            { \"name\":\"message\", \"type\":\"Edm.String\", \"filterable\":false, \"sortable\":false, \"facetable\":false, \"analyzer\":\"en.lucene\" },\n            { \"name\":\"created_at\", \"type\":\"Edm.DateTimeOffset\", \"facetable\":false}\n        ]\n    }\n\nThe plugin will write event stream out to Azure Ssearch like this:\n\n\u003cu\u003eInput event stream\u003c/u\u003e\n\n    { \"id\": \"1\", \"user_name\": \"taylorswift13\", \"message\":\"post by taylorswift13\" },\n    { \"id\": \"2\", \"user_name\": \"katyperry\", \"message\":\"post by katyperry\" },\n    { \"id\": \"3\", \"user_name\": \"ladygaga\", \"message\":\"post by ladygaga\" }\n\n\u003cu\u003eSearch results\u003c/u\u003e\n\n    \"value\": [\n        { \"@search.score\": 1, \"id\": \"1\", \"user_name\": \"taylorswift13\", \"message\": \"post by taylorswift13\", \"tag\": \"azuresearch.msg\", \"created_at\": \"2016-01-31T21:03:41Z\" },\n        { \"@search.score\": 1, \"id\": \"2\", \"user_name\": \"katyperry\", \"message\": \"post by katyperry\", \"tag\": \"azuresearch.msg\", \"created_at\": \"2016-01-31T21:03:41Z\" },\n        { \"@search.score\": 1, \"id\": \"3\", \"user_name\": \"ladygaga\", \"message\": \"post by ladygaga\", \"tag\": \"azuresearch.msg\", \"created_at\": \"2016-01-31T21:03:41Z\" }\n    ]\n[note] the value of created_at above is the time when fluentd actually recieves its corresponding input event.\n\n\n## Tests\n### Running test code\n    $ git clone https://github.com/yokawasa/fluent-plugin-azuresearch.git\n    $ cd fluent-plugin-azuresearch\n    \n    # edit CONFIG params of test/plugin/test_azuresearch.rb \n    $ vi test/plugin/test_azuresearch.rb\n    \n    # run test \n    $ rake test\n\n### Creating package, running and testing locally \n    $ rake build\n    $ rake install:local\n     \n    # running fluentd with your fluent.conf\n    $ fluentd -c fluent.conf -vv \u0026\n     \n    # send test input event to test plugin using fluent-cat\n    $ echo ' { \"postid\": \"100\", \"user\": \"ladygaga\", \"content\":\"post by ladygaga\"}' | fluent-cat azuresearch.msg\n\nPlease don't forget that you need forward input configuration to receive the message from fluent-cat\n\n    \u003csource\u003e\n        @type forward\n    \u003c/source\u003e\n\n\n## TODOs\n * Input validation for Azure Search - check total size of columns to add \n\n## Change log\n* [Changelog](ChangeLog.md)\n\n## Links\n\n* http://yokawasa.github.io/fluent-plugin-azuresearch\n* https://rubygems.org/gems/fluent-plugin-azuresearch\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/yokawasa/fluent-plugin-azuresearch.\n\n## Copyright\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eCopyright\u003c/td\u003e\u003ctd\u003eCopyright (c) 2016- Yoichi Kawasaki\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eLicense\u003c/td\u003e\u003ctd\u003eApache License, Version 2.0\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyokawasa%2Ffluent-plugin-azuresearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyokawasa%2Ffluent-plugin-azuresearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyokawasa%2Ffluent-plugin-azuresearch/lists"}