{"id":18726914,"url":"https://github.com/pytroll/trollmoves","last_synced_at":"2025-04-12T16:32:02.027Z","repository":{"id":37733056,"uuid":"41292869","full_name":"pytroll/trollmoves","owner":"pytroll","description":"Server/Client -based file transfer in Python","archived":false,"fork":false,"pushed_at":"2024-05-08T08:30:35.000Z","size":902,"stargazers_count":6,"open_issues_count":38,"forks_count":11,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-05-11T11:21:17.328Z","etag":null,"topics":["file-transfer","hacktoberfest","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pytroll.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-08-24T08:56:27.000Z","updated_at":"2024-05-30T16:09:05.410Z","dependencies_parsed_at":"2024-05-30T16:09:03.213Z","dependency_job_id":"50d97ed4-2e05-4fd0-8097-5881c3394886","html_url":"https://github.com/pytroll/trollmoves","commit_stats":{"total_commits":751,"total_committers":12,"mean_commits":"62.583333333333336","dds":0.6058588548601864,"last_synced_commit":"6e77ed1709ddf4502b1833ecc644b9d9a9e76277"},"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytroll%2Ftrollmoves","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytroll%2Ftrollmoves/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytroll%2Ftrollmoves/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pytroll%2Ftrollmoves/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pytroll","download_url":"https://codeload.github.com/pytroll/trollmoves/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248596582,"owners_count":21130725,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["file-transfer","hacktoberfest","python"],"created_at":"2024-11-07T14:15:53.178Z","updated_at":"2025-04-12T16:32:01.324Z","avatar_url":"https://github.com/pytroll.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Trollmoves\n\nTrollmoves is a package providing software for file transfers.\n\nThe common nominator is the use of Posttroll messaging to make requests, and to\npublish the completed transfers. These messages can be used to trigger further\nprocessing.\n\nThe required libraries are listed for each of the parts below.\n\n## Server/Client\n\nSetup where one or more Server processes announce new files, and one or two\nClient processes make transfer requests for the configured files.\n\n### Trollmoves Server\n\nTrollmoves Server is a process that follows a directory for new files, and publishes\na message when a matching file appears. If a Client makes a request for a file,\nthe file is transferred using one of the built-in movers (see below) based on the\ndestination given in the request.\n\nRequired libraries:\n- ``netifaces``\n- ``posttroll``\n- ``pyzmq``\n- ``trollsift``\n- ``watchdog``\n\nIn addition, the required packages for the transfer protocol(s) to be used need to be\ninstalled. See the mover documentation below for more details.\n\n#### Running the server without a client\nIn some situations, it might be difficult to use the server/client architecture,\nand thus there is a possibility to run the server in stand alone mode. To do\nthis, the only thing to do is omit the `request_port` configuration item in the\nserver configuration. From that point on, the server will send full uris in\nthe messages it publishes, along with a json representation of a `fsspec`\nfilesystem. From there, processes accepting these (eg `trollflow2`) will be able\nto use `fsspec` to read and process the remote files.\n\n### Trollmoves Client\n\nTrollmoves Client is configured to subscribe to a specific topic, and to make requests\nfor matching files published by a Server. The destination of the file is given in\nthe request message. The Server handles the actual transfer.\n\nClient can be configured to listen to multiple sources for the same files. The request\nis made to the Server where the first announcement were received from.\n\nTwo clients can be configured to handle requests for a given data. This makes it possible\nto make updates without outages, and in general add redundancy. One of the Client\nprocesses is considered the primary, and the secondary will process the leftover\nmessages after a small (for example 0.2 s - 1 s) delay. The Clients communicate\nwhich files are already handled, so duplicate transfers should not happen.\n\nRequired libraries:\n- ``netifaces``\n- ``posttroll``\n- ``pyzmq``\n- ``trollsift``\n\n### Trollmoves Mirror\n\nTrollmoves Mirror is a setup of back-to-back Server and Client that is used for\nexample to handle transfers from internal network to external Client processes. The\nMirror receives announcements from the internal network, publishes the file on\nexternal network, and upon receiving a request handles the transfer from internal\nServer to temporary directory and further on to the external destination.\n\nRequired libraries:\n- ``netifaces``\n- ``posttroll``\n- ``pyzmq``\n- ``trollsift``\n- ``watchdog``\n\nIn addition, the required packages for the transfer protocol(s) to be used. See the\nmover documentation below for more details.\n\n## Trollmoves Dispatcher\n\nTrollmoves Dispatcher can push files from local file system to any destination supported\nby the built-in movers. The dispatching is triggered by Posttroll messages published\nby a process creating the files, or otherwise following the arrival/creation of\nfiles.\n\nRequired libraries:\n- ``netifaces``\n- ``posttroll``\n- ``pyinotify``\n- ``pyzmq``\n- ``trollsift``\n\nIn addition, the required packages for the transfer protocol(s) to be used. See the\nmover documentation below for more details.\n\n## Individual movers\n\nThe individual movers can be used via the above listed processes, or used directly\nin other applications. The movers can be imported from the ``trollmoves.movers``\nmodule.\n\n### ``FileMover``\n\n``FileMover`` copies or moves a file between local filesystems.\n\nAdditional required packages: none.\n\n### ``FtpMover``\n\n``FtpMover`` transfers a local file to a FTP server.\n\nAdditional required packages: none.\n\n### ``ScpMover``\n\n``ScpMover`` uses SSH to transfer a local file to another (or the same) server.\n\nAdditional required packages:\n- ``scp``\n- ``paramiko``\n\n### ``SftpMover``\n\n``SftpMover`` uses SFTP protocol to transfer a local file to an SFTP server.\n\nAdditional required packages:\n- ``paramiko``\n\n### ``S3Mover``\n\n``S3Mover`` uploads a file to an S3 object storage.\n\nAdditional required packages:\n- ``s3fs``\n\nSpecial behaviour on destination filepath when using the S3Mover class:\n\nIf the destination prefix (~filepath) has a trailing slash ('/') the original\nfilename will be appended (analogous to moving a file from one directory to\nanother keeping the same filename).\n\nIf the destination prefix does not have a trailing slash the operation will be\nanalogous to moving a file from one directory to a new destination changing the\nfilename. The new destination filename will be the last part of the provided\ndestination following the last slash ('/').\n\n\n## s3downloader\n\nThis module is able to download files from a s3 endpoint.\n\nThe s3downloader module need to get posttroll messages from eg. s3stalker from the pytroll-collectors module which announces new available files in a configured s3 bucket. The s3downloader then downloads these files.\n\nExample config for the module is given in `examples/s3downloader-config.yaml`. If you use a `nameserver` with muliticast (nameserver default) you don't need to include the nameserver option nor the service name in the config. Opposite if you use nameserver without multicast.\n\nAdditional required packages:\n- ``boto3``\n- ``botocore``\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytroll%2Ftrollmoves","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpytroll%2Ftrollmoves","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpytroll%2Ftrollmoves/lists"}