{"id":19878655,"url":"https://github.com/ironsource/tailf2kafka","last_synced_at":"2025-03-01T02:33:17.170Z","repository":{"id":59157201,"uuid":"43002322","full_name":"ironSource/tailf2kafka","owner":"ironSource","description":"Watch and tail files with specified time based patterns and push them to kafka","archived":false,"fork":false,"pushed_at":"2023-03-23T19:10:41.000Z","size":34,"stargazers_count":0,"open_issues_count":1,"forks_count":1,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-02-02T10:17:58.881Z","etag":null,"topics":["devops","ruby"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ironSource.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-09-23T13:23:19.000Z","updated_at":"2022-07-05T09:30:02.000Z","dependencies_parsed_at":"2022-09-13T20:03:30.755Z","dependency_job_id":null,"html_url":"https://github.com/ironSource/tailf2kafka","commit_stats":null,"previous_names":["supersonicads/tailf2kafka"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironSource%2Ftailf2kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironSource%2Ftailf2kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironSource%2Ftailf2kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ironSource%2Ftailf2kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ironSource","download_url":"https://codeload.github.com/ironSource/tailf2kafka/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241308910,"owners_count":19941719,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["devops","ruby"],"created_at":"2024-11-12T17:06:12.549Z","updated_at":"2025-03-01T02:33:17.140Z","avatar_url":"https://github.com/ironSource.png","language":"Ruby","readme":"# Tailf2Kafka\n\nWatch and tail files in dirs with specified filename time based patterns and push them to kafka.\n\n\n## Installation\n\nInstall libsnappy dev libs if you want to take advantage of compression\n\n    apt-get install libsnappy-dev\n\nAdd this line to your application's Gemfile:\n\n    gem 'tailf2kafka'\n\nAnd then execute:\n\n    $ bundle install\n\nOr install it yourself as:\n\n    $ gem install tailf2kafka\n\n## Usage\n\n    $ tailf2kafka -h\n    Usage: tailf2kafka [options]\n            --config PATH                Path to settings config\n        -h, --help                       Display this screen\n    $\n\n## Config\n\n    tailf:\n      files:\n        - topic: haproxy\n          prefix: /var/log/haproxy/haproxy\n          suffix: ''\n          time_pattern: \".%Y-%m-%d.%H\"\n      position_file: \"/var/lib/haproxy/tail2kafka.offsets\"\n      flush_interval: 1\n      max_batch_lines: 1024\n      from_begining: true\n      delete_old_tailed_files: true\n    kafka:\n      brokers: [\"broker1:9092\", \"broker2:9092\", \"broker3:9092\"]\n      producer_type: sync\n      produce: true\n\n* kafka.brokers - Array of kafka brokers to connect to\n* kafka.producer_type - type of producer sync or async\n* kafka.produce - if false will not conect to kafka and will not produce any messages to it\n* tailf.position_file - file where to save tailed files offsets which were pushed to kafka\n* tailf.flush_interval - how often in seconds to save the offsets to a file\n* tailf.max_batch_lines - max number of lines to batch in each send request\n* tailf.from_beggining - in case of a new file added to tailing , if to start tailing from beggining or end of the file\n* tailf.delete_old_tailed_files - if to delete files once their time_pattern does not match the current time window and if they have been fully produced to kafka\n* tailf.files - array of file configs for tail, each tailed file configs consists of:\n  * topic - which kafka topic to produce the messages to\n  * prefix - the files prefix to watch for\n  * time_pattern - ruby time pattern of files to tail\n  * suffix - optional suffix of files to watch for\nso the tool will watch for files that match - prefix + time_pattern + suffix\n\n## Features/Facts\n\n* The config is validated by [schash](https://github.com/ryotarai/schash) gem\n* Tailed files are watched for changes by [rb-notify](https://github.com/nex3/rb-inotify) gem\n* Dirnames of all files prefixes are watched for new files creation or files moved to the dir and are automaticaly\nadded to tailing.\n* As well dirnames are watched for deletion or files being moved out of directory, and they are removed from the  list of files watched for changing.\n* Based time_pattern, files are periodicaly autodeleted , thus avoiding need for log rotation tools.\n* Files are matched by converting time_pattern to a regexp\n\n## Contributing\n\n1. Fork it\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create new Pull Request\n6. Go to 1\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fironsource%2Ftailf2kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fironsource%2Ftailf2kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fironsource%2Ftailf2kafka/lists"}