{"id":29293377,"url":"https://github.com/bmedici/rest-ftp-daemon","last_synced_at":"2025-08-23T06:34:43.360Z","repository":{"id":19242464,"uuid":"22477595","full_name":"bmedici/rest-ftp-daemon","owner":"bmedici","description":"A pretty simple but configurable and efficient FTP-client daemon, driven through a RESTful API, used by France Télévisions in production","archived":false,"fork":false,"pushed_at":"2019-06-04T09:54:45.000Z","size":2798,"stargazers_count":24,"open_issues_count":1,"forks_count":9,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-08-17T08:24:36.349Z","etag":null,"topics":["daemon","francetelevision","ftp-client","microservice","robust","ruby","video","vod"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bmedici.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-07-31T17:04:57.000Z","updated_at":"2022-07-26T10:37:11.000Z","dependencies_parsed_at":"2022-07-27T00:16:10.144Z","dependency_job_id":null,"html_url":"https://github.com/bmedici/rest-ftp-daemon","commit_stats":null,"previous_names":[],"tags_count":192,"template":false,"template_full_name":null,"purl":"pkg:github/bmedici/rest-ftp-daemon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bmedici%2Frest-ftp-daemon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bmedici%2Frest-ftp-daemon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bmedici%2Frest-ftp-daemon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bmedici%2Frest-ftp-daemon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bmedici","download_url":"https://codeload.github.com/bmedici/rest-ftp-daemon/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bmedici%2Frest-ftp-daemon/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271745678,"owners_count":24813521,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["daemon","francetelevision","ftp-client","microservice","robust","ruby","video","vod"],"created_at":"2025-07-06T11:02:07.937Z","updated_at":"2025-08-23T06:34:43.335Z","avatar_url":"https://github.com/bmedici.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"rest-ftp-daemon\n====================================================================================\n\n[![Gem Version](https://badge.fury.io/rb/rest-ftp-daemon.svg)](http://badge.fury.io/rb/rest-ftp-daemon)\n[![Code Climate](https://codeclimate.com/github/bmedici/rest-ftp-daemon/badges/gpa.svg)](https://codeclimate.com/github/bmedici/rest-ftp-daemon)\n[![Test Coverage](https://codeclimate.com/github/bmedici/rest-ftp-daemon/badges/coverage.svg)](https://codeclimate.com/github/bmedici/rest-ftp-daemon/coverage)\n[ ![Codeship](https://codeship.com/projects/46492fd0-00a4-0134-6639-66759d382560/status?branch=master)](https://codeship.com/projects/153245)\n\nA pretty simple but configurable and efficient FTP-client daemon, driven\nthrough a RESTful API. Create transfer jobs by POSTing a simple JSON structure,\nbe notified of their completion, watch their status on a dedicated dashboard.\n\n\n![Dashboard](dashboard-201607.png)\n\n\nFeatures\n------------------------------------------------------------------------------------\n\n* System and process features\n  * environment-aware configuration in a YAML file\n  * daemon process is tagged with its name and environment in process lists\n  * global dashboard directly served within the daemon HTTP interface\n  * support pooling of worker to dedicate workers to groups of jobs\n\n* File management ans transferts\n  * allow authentication in FTP target in a standard URI-format\n  * static path pointers in configuration to abstract local mounts or remote FTPs (endpoint tokens)\n  * local source path and local/remote target path can use patterns to match multiple files (`/dir/file*.jpg`)\n  * several file transfer protocols supported: FTPs, FTPes, sFTP\n  * display bitrate to any pool or any FTP destination currently transferring (API and dashboard)\n\n* Job management\n  * highly parrallel job processing using dedicated worker threads with their own context\n  * jobs are taken into account as soon as they are submitted\n  * each job carry its own attributes: build subdirectories (mkdir), overwrite target file, priority weight\n  * dynamic evaluation of priorities, honoring any change on context until the job is picked\n  * automatically clean-up jobs after a configurable amount of time (failed, finished)\n\n* Realtime status reporting\n  * realtime transfer status reporting, with progress and errors\n  * periodic update notifications sent along with transfer status and progress to an arbitrary URL (JSON resource POSTed)\n  * metrics about pools, throughtput, and queues output to NewRelic\n\n\nProject status and quick installation\n------------------------------------------------------------------------------------\n\n#### Stability\n\nThough it may need more robust tests, this gem has been used successfully in production for\na while without any glitches at France Télévisions.\n\n#### API Documentation\n\nAPI documentation is self-hosted on ```/swagger.html```\n\n#### Expected features in a short-time range\n\n* Provide swagger-style API documentation\n* Authenticate API clients\n* Allow more transfer protocols (HTTP POST etc)\n* Expose JSON status of workers on `GET /jobs/` for automated monitoring\n\n#### Installation\n\nWith Ruby (version 2.3 or higher) and rubygems properly installed, you only need :\n\n```\ngem install rest-ftp-daemon\n```\n\nIf that is not the case yet, see section [Debian install preparation](#debian-install-preparation).\n\n\nSubsystems\n------------------------------------------------------------------------------------\n\n#### Conchita: jobs queues cleanup\n\nJob queue can be set to automatically cleanup after a certain delay. Entries are removed from the queue when they have been idle (updated_at) for more than X seconds, and in any of the following statuses:\n\n- failed (conchita.clean_failed)\n- finished (conchita.clean_finished)\n- queued, (conchita.clean_queued)\n\nCleanup is done on a regular basis, every (conchita.timer) seconds.\n\n#### Reporter: metrics collection\n\n[TODO]\n\n\nUsage and examples\n------------------------------------------------------------------------------------\n\n#### Launching rest-ftp-daemon\n\nYou must provide a configuration file for the daemon to start, either explicitly using\noption `--config` or implicitly at `/etc/rest-ftp-daemon.yml`. A sample file is provided, issue\n`--help` to get more info.\n\nYou then simply start the daemon on its standard port, or on a specific port using `-p`\n\n```\n$ rest-ftp-daemon -p 3000 start\n```\n\nCheck that the daemon is running and exposes a JSON status structure at `http://localhost:3000/status`.\n\nThe dashboard will provide an overview at `http://localhost:3000/`\n\nIf the daemon appears to exit quickly when launched, it may be caused by logfiles that can't be written (check files permissions or owner).\n\n#### Launcher options :\n\n| Param   | Short         | Default       | Description                                                 |\n|-------  |-------------- |-------------  |-----------------------------------------------------------  |\n| -p      | --port        | (automatic)   | Port to listen for API requests                             |\n| -e      |               | production    | Environment name                                            |\n|         | --dev         |               | Equivalent to -e development                                |\n| -d      | --daemonize   | false         | Wether to send the daemon to background                     |\n| -f      | --foreground  | false         | Wether to keep the daemon running in the shell              |\n| -P      | --pid         | (automatic)   | Path of the file containing the PID                         |\n| -u      | --user        | (none)        | User to run the daemon as                                   |\n| -g      | --group       | (none)        | Group of the user to run the daemon as                      |\n| -h      | --help        |               | Show info about the current version and available options   |\n| -v      | --version     |               | Show the current version                                    |\n\n\n#### Start a job to transfer a file named \"file.iso\" to a local FTP server\n\n```\ncurl -H \"Content-Type: application/json\" -X POST -D /dev/stdout -d \\\n'{\"source\":\"~/file.iso\",\"target\":\"ftp://anonymous@localhost/incoming/dest2.iso\"}' \"http://localhost:3000/jobs\"\n```\n\n#### Start a job using endpoint tokens\n\nFirst define ``nas`` ans ``ftp1`` in the configuration file :\n\n```\ndefaults: \u0026defaults\n\ndevelopment:\n  \u003c\u003c: *defaults\n\n  endpoints:\n    nas: \"~/\"\n    ftp1: \"ftp://anonymous@localhost/incoming/\"\n```\n\nThose tokens will be expanded when the job is run:\n\n```\ncurl -H \"Content-Type: application/json\" -X POST -D /dev/stdout -d \\\n'{\"source\":\"~/file.dmg\",\"priority\":\"3\",\"target\":\"ftp://anonymous@localhost/incoming/dest4.dmg\",\"notify\":\"http://requestb.in/1321axg1\"}' \"http://localhost:3000/jobs\"\n```\n\n\n#### Start a job with a specific pool name\n\nThe daemon spawns groups of workers (worker pools) to work on groups of jobs (job pools). Any ```pool``` attribute not declared in configuration will land into the ```\"default\"``` pool.\n\n```\ncurl -H \"Content-Type: application/json\" -X POST -D /dev/stdout -d \\\n'{\"pool\": \"maxxxxx\",source\":\"~/file.iso\",target\":\"ftp://anonymous@localhost/incoming/dest2.iso\"}' \"http://localhost:3000/jobs\"\n```\nThis job will be handled by the \"maxxxxx\" workers only, or by the ```\"default\"``` worker is this pool is not declared.\n\n\n#### Get info about a job with ID=\"q89j.1\"\n\nBoth parameters `q89j.1` and `1` will be accepted as ID in the API. Requests below are equivalent:\n\n```\nGET http://localhost:3000/jobs/q89j.1\nGET http://localhost:3000/jobs/1\n```\n\n\nConfiguration\n------------------------------------------------------------------------------------\n\nMost of the configuration options live in a YAML configuration file, containing two main sections:\n\n* `defaults` section should be left as-is and will be used is no other environment-specific value is provided.\n* `production` section can receive personalized settings according to your environment-specific setup and paths.\n\nConfiguration priority is defined as follows (from most important to last resort):\n\n* command-line parameters\n* config file defaults section\n* config file environment section\n* application internal defaults\n\nAs a starting point, `rest-ftp-daemon.yml.sample` is an example config file that can be  copied into the expected location ``/etc/rest-ftp-daemon.yml``.\n\nDefault administrator credentials are `admin/admin`. Please change the password in this configuration file before starting any kind of production.\n\n\nHere is the contents of the default configuration (oeverride by passing -c local.yml at startup)\n\n```yaml\ndaemonize: true\nport: 3000\nuser: rftpd\n# group: rftpd\n# host: \"myhost\"\nallow_reload: false\n\npools:                      # number of workers decidated to each pool value\n  default: 2\n  urgent: 1\n\nreporter:                   # the subsytem in charge of reporting metrics, mainly to NewRelic\n  debug: false\n  timer: 10                 # report every X seconds\n\nconchita:\n  debug: false\n  timer: 60                 # do the cleaning up every X seconds\n  garbage_collector: true   # force a garbage collector cleanup when cleaning things up\n  clean_failed: 3600        # after X seconds, clean jobs with status=\"failed\"\n  clean_finished: 3600      # //              //              //       finished\n  clean_queued: 86400       # //              //              //       queued\n\ntransfer:\n  debug: false\n  mkdir: true               # build directory tree if missing\n  tempfile: true            # transfer to temporary file, rename after sucessful transfer\n  overwrite: false          # overwrite any target file with the same name\n  timeout: 1800             # jobs running for longer than X seconds will be killed\n  notify_after: 5           # wait at least X seconds between HTTP notifications\n\n  debug_ftp: false\n  debug_ftps: false\n  debug_sftp: false\n\n  retry_on:                 # job error values that will allow a retry\n    - ftp_perm_error\n    - net_temp_error\n    - conn_reset_by_peer\n    - conn_timed_out\n    - conn_refused\n    - sftp_auth_failed\n    - conn_host_is_down\n    - conn_unreachable\n    - conn_failed\n    - conn_openssl_error\n  retry_max: 5              # maximum number of retries before giving up on that job\n  retry_for: 1800           # maximum time window to retry failed jobs\n  retry_after: 10           # delay to wait before tries\n\nnewrelic:\n  debug: false\n  # license: \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n  # app_name: \"rftpd-bigbusiness-dev\"     # app_name used for naming app (used as-is if provided)\n  prefix: \"rftpd\"                         # app prefix to build app_name\n  # platform: \"bigbusiness\"               # app platform to build app_name\n\nlogs:\n  path:     \"/var/log/\"\n  thin:     \"rftpd-environment-thin.log\"\n  newrelic: \"rftpd-environment-newrelic.log\"\n  queue:    \"rftpd-environment-core.log\"\n  api:      \"rftpd-environment-core.log\"\n  workers:  \"rftpd-environment-core.log\"\n  transfer: \"rftpd-environment-workers.log\"\n  conchita: \"rftpd-environment-workers.log\"\n  reporter: \"rftpd-environment-workers.log\"\n  notify:   \"rftpd-environment-workers.log\"\n\n```\n\n\n\n\nTODO for this document\n------------------------------------------------------------------------------------\n\n* Document /status\n* Document /routes\n* Document mkdir and overwrite options\n* Document stats\n\n\n\nDebian install preparation\n------------------------------------------------------------------------------------\n\nThis project is available as a rubygem, requires Ruby 2.3.0 and RubyGems installed.\n\n#### Using rbenv and ruby-build\n\nYou may use `rbenv` and `ruby-build` to get the right Ruby version. If this is your case, ensure that ruby-build definitions are up-to-date and include the right Ruby version.\nYou may have to install some extra packages for the compilations to complete.\n\n```\n# apt-get install libffi-dev zlib1g-dev bison libreadline-dev\n# git clone https://github.com/rbenv/rbenv.git ~/.rbenv\n# git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build\n# echo 'export PATH=\"$HOME/.rbenv/bin:$PATH\"' \u003e\u003e ~/.bashrc\n# echo 'eval \"$(rbenv init -)\"' \u003e\u003e ~/.bashrc\n# rbenv install --list | grep '2.3'\n```\n\n```\n# curl -fsSL https://gist.github.com/mislav/055441129184a1512bb5.txt | rbenv install --patch 2.2.3\n```\n\nOtherwise, you way have to update ruby-build to include Ruby 2.3 definitions.\nOn Debian, 2.3 is not included in Wheezy and appears in Jessie's version of the package.\n\n#### Dedicated user\n\nUse a dedicated user for the daemon, switch to this user and enable rbenv\n\n```\n# adduser --disabled-password --gecos \"\" rftpd\n# su rftpd -l\n```\n\n#### Ruby version\n\nInstall the right ruby version and activate it\n\n```\n# rbenv install 2.3.0\n# rbenv local 2.3.0\n# rbenv rehash\n```\n\n#### Daemon installation\n\nUpdate RubyGems and install the gem from rubygems.org\n\n```\n# gem update --system\n# gem install rest-ftp-daemon --no-ri --no-rdoc\n# rbenv rehash\n# rest-ftp-daemon start\n```\n\nKnown bugs\n------------------------------------------------------------------------------------\n\n* As this project is based on the Psyck YAML parser, configuration merge from \"defaults\" section and environment-specific section are broken. A sub-tree defined for a specific environment, will overwrite the corresponding subtree from \"defaults\". Please repeat whole sections from \"defaults\".\n\n* As this project is based on Chamber, and it considers hyphens in filename as namespaces, the global /etc/rest-ftp-daemon.yml config file is not parsed (and thus, ignored). Until this is worked around, please specify a config filename on the commandline.\n\n* If you get ```fatal error: 'openssl/ssl.h' file not found when installing eventmachine``` on OSX El Capitan, you can try with:\n```\ngem install eventmachine -v '1.0.8' -- --with-cppflags=-I/usr/local/opt/openssl/include\nbundle install\n```\n\n* If you get ```uncommon.mk:189: recipe for target 'build-ext' failed``` on Debian, you can try with:\n```\ncurl -fsSL https://gist.github.com/mislav/055441129184a1512bb5.txt | rbenv install --patch 2.2.3\n```\n\n\nContributing\n------------------------------------------------------------------------------------\n\nContributions are more than welcome, be it for documentation, features, tests,\nrefactoring, you name it. If you are unsure of where to start, the [Code\nClimate](https://codeclimate.com/github/bmedici/rest-ftp-daemon) report will\nprovide you with improvement directions. And of course, if in doubt, do not\nhesitate to open an issue. (Please note that this project has adopted a [code\nof conduct](CODE_OF_CONDUCT.md).)\n\nIf you want your contribution to adopted in the smoothest and fastest way, don't\nforget to:\n\n* provide sufficient documentation in you commit and pull request\n* add proper testing (we know full grown solid test coverage is still lacking and\n  need to up the game)\n* use the [RuboCop](https://github.com/bbatsov/rubocop) guidelines provided\n  (there are all sorts of editor integration plugins available)\n\nSo,\n\n1. Fork the project\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Code\n  * add proper tests if adding a feature\n  * run the tests using `rake`\n  * check for RuboCop style guide violations\n\n4. Commit your changes\n5. Push to the branch (`git push origin my-new-feature`)\n\n6. Create new Pull Request\n\n\nAbout\n------------------------------------------------------------------------------------\n\nThanks to https://github.com/berkshelf/berkshelf-api for parts and ideas used in this project\n\nThis project has been initiated and originally written by\nBruno MEDICI Consultant (http://bmconseil.com/)\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbmedici%2Frest-ftp-daemon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbmedici%2Frest-ftp-daemon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbmedici%2Frest-ftp-daemon/lists"}