{"id":15524122,"url":"https://github.com/fmjsjx/fileminer","last_synced_at":"2025-04-23T07:29:58.508Z","repository":{"id":56846372,"uuid":"160171283","full_name":"fmjsjx/fileminer","owner":"fmjsjx","description":"A simple file/log transfer tool coding by ruby.","archived":false,"fork":false,"pushed_at":"2020-04-15T03:18:43.000Z","size":61,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-26T21:40:01.071Z","etag":null,"topics":["file","log","ruby","transfer"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fmjsjx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-03T10:09:58.000Z","updated_at":"2020-04-09T04:22:02.000Z","dependencies_parsed_at":"2022-09-09T00:52:41.771Z","dependency_job_id":null,"html_url":"https://github.com/fmjsjx/fileminer","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fmjsjx%2Ffileminer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fmjsjx%2Ffileminer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fmjsjx%2Ffileminer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fmjsjx%2Ffileminer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fmjsjx","download_url":"https://codeload.github.com/fmjsjx/fileminer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250390604,"owners_count":21422731,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["file","log","ruby","transfer"],"created_at":"2024-10-02T10:49:14.603Z","updated_at":"2025-04-23T07:29:58.463Z","avatar_url":"https://github.com/fmjsjx.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fileminer\n[![Gem Version](https://d25lcipzij17d.cloudfront.net/badge.svg?id=rb\u0026type=6\u0026v=1.2.1\u0026x2=0)](https://rubygems.org/gems/fileminer)\n[![MIT licensed](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/fmjsjx/fileminer/blob/master/LICENSE)\n\n\nA simple line based file/log transfer tool coding by ruby.\n\n# quick start\n1. installation\n\nInstall fileminer from RubyGems:\n```\n$ gem install fileminer\nFetching fileminer-1.2.1.gem\nSuccessfully installed fileminer-1.2.1\nParsing documentation for fileminer-1.2.1\nInstalling ri documentation for fileminer-1.2.1\nDone installing documentation for fileminer after 0 seconds\n1 gem installed\n$ _\n```\n\n*Note: This tool now only support cruby.*\n\n\n2. setting up fileminer\n\n* generate configuration file\n\nAt first, we should generate the fileminer configuration file.\n\nWe provide a command tool 'fileminer-genconf' to generate configurations:\n```\n$ fileminer-genconf -h\nUsage:\n    fileminer-genconf [options]\n\nSamples:\n    fileminer-genconf -t fileminer -o      Generate config on /etc/fileminer/fileminer.yml\n    fileminer-genconf -t supervisor -o -l  Generate ./fileminer.ini with logfile on\n                                           /var/log/fileminer/stderr.log\n    fileminer-genconf -t systemd -o        Generate systemd config on\n                                           /usr/lib/systemd/system/fileminer.service\n\nOptions:\n    -t fileminer|supervisor|systemd, Type of the config file to be generated\n        --type                       Default is fileminer\n    -o, --out [path]                 Output content to a file\n                                     For type fileminer, default is /etc/fileminer/fileminer.yml\n                                     For type supervisor, default is ./fileminer.ini\n    -l, --logfile [path]             Logfile configured on supervisor config file\n                                     Default is /var/log/fileminer/stderr.log\n    -h, --help                       Print help\n\n$ _\n```\n\nGeneration with default options:\n```\n$ fileminer-genconf -o\ngenerated config file: /etc/fileminer/fileminer.yml\n$ _\n```\n\n* config input on fileminer.yml\n\nEdit fileminer.inputs on fileminer.yml:\n```yaml\nfileminer.inputs:\n  paths:\n    - /path/to/*.log\n```\n\n* configure output on fileminer.yml\n\nIn current version, fileminer provides three prefab output plugins: redis, kafka \u0026 mysql. One process can only choose one of them.\n\n**Output to redis:**\n\nRedis output plugin now using LPUSH to send messages for each line.\n\nInstall redis client, fileminer use hiredis by default:\n```\n$ gem install redis hiredis\nFetching redis-4.1.0.gem\nSuccessfully installed redis-4.1.0\nParsing documentation for redis-4.1.0\nInstalling ri documentation for redis-4.1.0\nDone installing documentation for redis after 0 seconds\nFetching hiredis-0.6.3.gem\nBuilding native extensions. This could take a while...\nSuccessfully installed hiredis-0.6.3\nParsing documentation for hiredis-0.6.3\nInstalling ri documentation for hiredis-0.6.3\nDone installing documentation for hiredis after 0 seconds\n2 gems installed\n$ _\n```\n\nEdit fileminer.yml:\n```yaml\noutput.redis:\n  # target redis server URI\n  uri: redis://localhost:6379/0\n  # target redis key, type must be LIST\n  key: fileminer\n```\n\n\n**Output to kafka:**\n\nInstall kafka ruby client:\n```\n$ gem install ruby-kafka\nSuccessfully installed ruby-kafka-0.7.5\nParsing documentation for ruby-kafka-0.7.5\nDone installing documentation for ruby-kafka after 0 seconds\n1 gem installed\n$ _\n```\n\nEdit fileminer.yml\n```yaml\noutput.kafka:\n  brokers: ['host1:9092','host2:9092','host3:9092']\n  client_id: fileminer\n  topic: fileminer\n```\n\n\n**Output to MySQL:**\n\nInstall mysql2:\n```\n$ gem install mysql2\nBuilding native extensions. This could take a while...\nSuccessfully installed mysql2-0.5.2\nParsing documentation for mysql2-0.5.2\nDone installing documentation for mysql2 after 0 seconds\n1 gem installed\n$ _\n```\n\nEdit fileminer.yml\n```yaml\noutput.mysql:\n  host: hostname\n  port: 3306\n  username: someuser\n  password: somepwd\n  database: somedb\n  table: sometable\n```\n\n\n*Since 1.1.0, you can also use customized output plugin.*\n\n**Output using customized output plugin:**\n\nGet your script ready:\n```ruby\nrequire 'fileminer/plugins'\n\nclass YourPlugin \u003c Output::OutputPlugin\n  def initialize options\n    ...\n  end\n  ...\n  public\n  def send_all(lines, \u0026listener)\n    ...\n  end\n  ...\nend\n```\n\nEdit fileminer.yml\n```yaml\noutput.script:\n  script: /path/to/your_script.rb\n  plugin_class: YourPlugin\n  init_options:\n    some_field: some value\n```\n\n3. runnning fileminer\n\nRun in command line:\n```\n$ fileminer\n\n```\nor\n```\n$ fileminer /path/to/fileminer.yml\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffmjsjx%2Ffileminer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffmjsjx%2Ffileminer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffmjsjx%2Ffileminer/lists"}