{"id":18115059,"url":"https://github.com/fluent/fluent-plugin-mongo","last_synced_at":"2025-05-16T04:05:37.991Z","repository":{"id":44767553,"uuid":"2472975","full_name":"fluent/fluent-plugin-mongo","owner":"fluent","description":"MongoDB input and output plugin for Fluentd","archived":false,"fork":false,"pushed_at":"2023-02-13T21:16:20.000Z","size":339,"stargazers_count":175,"open_issues_count":36,"forks_count":59,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-05-13T01:39:30.591Z","etag":null,"topics":["fluentd","fluentd-plugin","mongodb"],"latest_commit_sha":null,"homepage":"https://docs.fluentd.org/output/mongo","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fluent.png","metadata":{"files":{"readme":"README.rdoc","changelog":"ChangeLog","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2011-09-28T05:21:21.000Z","updated_at":"2025-05-04T09:24:16.000Z","dependencies_parsed_at":"2024-06-18T13:51:14.394Z","dependency_job_id":"6292bdc1-1ca5-43a9-816b-c4e098b4d12b","html_url":"https://github.com/fluent/fluent-plugin-mongo","commit_stats":{"total_commits":321,"total_committers":28,"mean_commits":"11.464285714285714","dds":"0.44548286604361376","last_synced_commit":"13e4575dc1816d1d3295f5929be1b0874e82a96f"},"previous_names":[],"tags_count":58,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-mongo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-mongo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-mongo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-mongo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fluent","download_url":"https://codeload.github.com/fluent/fluent-plugin-mongo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254464895,"owners_count":22075570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd","fluentd-plugin","mongodb"],"created_at":"2024-11-01T03:08:19.052Z","updated_at":"2025-05-16T04:05:32.979Z","avatar_url":"https://github.com/fluent.png","language":"Ruby","readme":"= MongoDB plugin for {Fluentd}[http://github.com/fluent/fluentd]\n\nfluent-plugin-mongo provides input and output plugins for {Fluentd}[http://fluentd.org] ({GitHub}[http://github.com/fluent/fluentd])\n\n= Requirements\n\n    |fluent-plugin-mongo|   fluentd  |  ruby  |\n    |-------------------|------------|--------|\n    |     \u003e= 1.0.0      | \u003e= 0.14.12 | \u003e= 2.1 |\n    |     \u003c  1.0.0      | \u003e= 0.12.0  | \u003e= 1.9 |\n\n= Installation\n\n== Gems\n\nThe gem is hosted at {Rubygems.org}[http://rubygems.org]. You can install the gem as follows:\n\n    $ fluent-gem install fluent-plugin-mongo\n\n= Plugins\n\n== Output plugin\n\n=== mongo\n\nStore Fluentd event to MongoDB database.\n\n==== Configuration\n\nUse _mongo_ type in match.\n\n    \u003cmatch mongo.**\u003e\n      @type mongo\n\n      # You can choose two approaches, connection_string or each parameter\n      # 1. connection_string for MongoDB URI\n      connection_string mongodb://fluenter:10000/fluent\n\n      # 2. specify each parameter\n      database fluent\n      host fluenter\n      port 10000\n\n      # collection name to insert\n      collection test\n\n      # Set 'user' and 'password' for authentication.\n      # These options are not used when use connection_string parameter.\n      user handa\n      password shinobu\n\n      # Set 'capped' if you want to use capped collection\n      capped\n      capped_size 100m\n\n      # Specify date fields in record to use MongoDB's Date object (Optional) default: nil\n      # Supported data types are String/Integer/Float/Fuentd EventTime.\n      # For Integer type, milliseconds epoch and seconds epoch are supported.\n      # eg: updated_at: \"2020-02-01T08:22:23.780Z\" or updated_at: 1580546457010\n      date_keys updated_at\n\n      # Specify id fields in record to use MongoDB's BSON ObjectID (Optional) default: nil\n      # eg: my_id: \"507f1f77bcf86cd799439011\"\n      object_id_keys my_id\n\n      # Other buffer configurations here\n    \u003c/match\u003e\n\nFor _connection_string_ parameter, see https://docs.mongodb.com/manual/reference/connection-string/ article for more detail.\n\n===== built-in placeholders\n\nfluent-plugin-mongo support built-in placeholders.\n_database_ and _collection_ parameters can handle them.\n\nHere is an example to use built-in placeholders:\n\n    \u003cmatch mongo.**\u003e\n      @type mongo\n\n      database ${tag[0]}\n\n      # collection name to insert\n      collection ${tag[1]}-%Y%m%d\n\n      # Other buffer configurations here\n      \u003cbuffer tag, time\u003e\n        @type memory\n        timekey 3600\n      \u003c/buffer\u003e\n    \u003c/match\u003e\n\nIn more detail, please refer to the officilal document for built-in placeholders: https://docs.fluentd.org/v1.0/articles/buffer-section#placeholders\n\n=== mongo(tag mapped mode)\n\nTag mapped to MongoDB collection automatically.\n\n==== Configuration\n\nUse _tag_mapped_ parameter in match of _mongo_ type.\n\nIf tag name is \"foo.bar\", auto create collection \"foo.bar\" and insert data.\n\n    \u003cmatch forward.*\u003e\n      @type mongo\n      database fluent\n\n      # Set 'tag_mapped' if you want to use tag mapped mode.\n      tag_mapped\n\n      # If tag is \"forward.foo.bar\", then prefix \"forward.\" is removed.\n      # Collection name to insert is \"foo.bar\".\n      remove_tag_prefix forward.\n\n      # This configuration is used if tag not found. Default is 'untagged'.\n      collection misc\n\n      # Other configurations here\n    \u003c/match\u003e\n\n=== mongo_replset\n\nReplica Set version of mongo.\n\n==== Configuration\n\n===== v0.8 or later\n\n    \u003cmatch mongo.**\u003e\n      @type mongo_replset\n      database fluent\n      collection logs\n\n      nodes localhost:27017,localhost:27018\n\n      # The replica set name\n      replica_set myapp\n\n      # num_retries is threshold at failover, default is 60.\n      # If retry count reached this threshold, mongo plugin raises an exception.\n      num_retries 30\n\n      # following optional parameters passed to mongo-ruby-driver.\n      # See mongo-ruby-driver docs for more detail: https://docs.mongodb.com/ruby-driver/master/tutorials/ruby-driver-create-client/\n      # Specifies the read preference mode\n      #read secondary\n    \u003c/match\u003e\n\n===== v0.7 or ealier\n\nUse _mongo_replset_ type in match.\n\n    \u003cmatch mongo.**\u003e\n      @type mongo_replset\n      database fluent\n      collection logs\n\n      # each node separated by ','\n      nodes localhost:27017,localhost:27018,localhost:27019\n\n      # following optional parameters passed to mongo-ruby-driver.\n      #name replset_name\n      #read secondary\n      #refresh_mode sync\n      #refresh_interval 60\n      #num_retries 60\n    \u003c/match\u003e\n\n== Input plugin\n\n=== mongo_tail\n\nTail capped collection to input data.\n\n==== Configuration\n\nUse _mongo_tail_ type in source.\n\n    \u003csource\u003e\n      @type mongo_tail\n      database fluent\n      collection capped_log\n\n      tag app.mongo_log\n\n      # waiting time when there is no next document. default is 1s.\n      wait_time 5\n\n      # Convert 'time'(BSON's time) to fluent time(Unix time).\n      time_key time\n\n      # Convert ObjectId to string\n      object_id_keys [\"id_key\"]\n    \u003c/source\u003e\n\nYou can also use _url_ to specify the database to connect.\n\n    \u003csource\u003e\n      @type mongo_tail\n      url mongodb://user:password@192.168.0.13:10249,192.168.0.14:10249/database\n      collection capped_log\n      ...\n    \u003c/source\u003e\n\nThis allows the plugin to read data from a replica set.\n\nYou can save last ObjectId to tail over server's shutdown to file.\n\n    \u003csource\u003e\n      ...\n\n      id_store_file /Users/repeatedly/devel/fluent-plugin-mongo/last_id\n    \u003c/source\u003e\n\nOr Mongo collection can be used to keep last ObjectID.\n\n    \u003csource\u003e\n      ...\n\n      id_store_collection last_id\n    \u003c/source\u003e\n\nMake sure the collection is capped. The plugin inserts records but does not remove at all.\n\n= NOTE\n\n== replace_dot_in_key_with and replace_dollar_in_key_with\n\nBSON records which include '.' or start with '$' are invalid and they will be stored as broken data to MongoDB. If you want to sanitize keys, you can use _replace_dot_in_key_with_ and _replace_dollar_in_key_with_.\n\n    \u003cmatch forward.*\u003e\n      ...\n      # replace '.' in keys with '__dot__'\n      replace_dot_in_key_with __dot__\n\n      # replace '$' in keys with '__dollar__'\n      # Note: This replaces '$' only on first character\n      replace_dollar_in_key_with __dollar__\n      ...\n    \u003c/match\u003e\n\n== Broken data as a BSON\n\nNOTE: This feature will be removed since v0.8\n\nFluentd event sometimes has an invalid record as a BSON.\nIn such case, Mongo plugin marshals an invalid record using Marshal.dump\nand re-inserts its to same collection as a binary.\n\nIf passed following invalid record:\n\n    {\"key1\": \"invalid value\", \"key2\": \"valid value\", \"time\": ISODate(\"2012-01-15T21:09:53Z\") }\n\nthen Mongo plugin converts this record to following format:\n\n    {\"__broken_data\": BinData(0, Marshal.dump result of {\"key1\": \"invalid value\", \"key2\": \"valid value\"}), \"time\": ISODate(\"2012-01-15T21:09:53Z\") }\n\nMongo-Ruby-Driver cannot detect an invalid attribute,\nso Mongo plugin marshals all attributes excluding Fluentd keys(\"tag_key\" and \"time_key\").\n\nYou can deserialize broken data using Mongo and Marshal.load. Sample code is below:\n\n    # _collection_ is an instance of Mongo::Collection\n    collection.find({'__broken_data' =\u003e {'$exists' =\u003e true}}).each do |doc|\n      p Marshal.load(doc['__broken_data'].to_s) #=\u003e {\"key1\": \"invalid value\", \"key2\": \"valid value\"}\n    end\n\n=== ignore_invalid_record\n\nIf you want to ignore an invalid record, set _true_ to _ignore_invalid_record_ parameter in match.\n\n    \u003cmatch forward.*\u003e\n      ...\n\n      # ignore invalid documents at write operation\n      ignore_invalid_record true\n\n      ...\n    \u003c/match\u003e\n\n=== exclude_broken_fields\n\nIf you want to exclude some fields from broken data marshaling, use _exclude_broken_fields_ to specfiy the keys.\n\n    \u003cmatch forward.*\u003e\n      ...\n\n      # key2 is excluded from __broken_data.\n      # e.g. {\"__broken_data\": BinData(0, Marshal.dump result of {\"key1\": \"invalid value\"}), \"key2\": \"valid value\", \"time\": ISODate(\"2012-01-15T21:09:53Z\")\n      exclude_broken_fields key2\n\n      ...\n    \u003c/match\u003e\n\nSpecified value is a comma separated keys(e.g. key1,key2,key3).\nThis parameter is useful for excluding shard keys in shard environment.\n\n== Buffer size limitation\n\nMongo plugin has the limitation of buffer size.\nBecause MongoDB and mongo-ruby-driver checks the total object size at each insertion.\nIf total object size gets over the size limitation, then\nMongoDB returns error or mongo-ruby-driver raises an exception.\n\nSo, Mongo plugin resets _buffer_chunk_limit_ if configurated value is larger than above limitation:\n- Before v1.8, max of _buffer_chunk_limit_ is 2MB\n- After  v1.8, max of _buffer_chunk_limit_ is 8MB\n\n= Tool\n\nYou can tail mongo capped collection.\n\n    $ mongo-tail -f\n\n= Test\n\nRun following command:\n\n    $ bundle exec rake test\n\nYou can use 'mongod' environment variable for specified mongod:\n\n    $ mongod=/path/to/mongod bundle exec rake test\n\nNote that source code in test/tools are from mongo-ruby-driver.\n\n= Copyright\n\nCopyright:: Copyright (c) 2011- Masahiro Nakagawa\nLicense::   Apache License, Version 2.0\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent%2Ffluent-plugin-mongo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffluent%2Ffluent-plugin-mongo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent%2Ffluent-plugin-mongo/lists"}