{"id":31027087,"url":"https://github.com/willhackett/azure-mft","last_synced_at":"2025-09-13T18:51:21.224Z","repository":{"id":94676215,"uuid":"389502481","full_name":"willhackett/azure-mft","owner":"willhackett","description":"Managed File Transfer proof-of-concept using Azure Blob \u0026 ServiceBus as a back-end","archived":false,"fork":false,"pushed_at":"2023-12-15T14:54:18.000Z","size":150,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-06-20T23:31:02.388Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/willhackett.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-26T03:59:05.000Z","updated_at":"2021-07-28T07:04:22.000Z","dependencies_parsed_at":"2024-06-20T22:37:54.867Z","dependency_job_id":"96a8664d-b087-44fc-b762-2b28129d24a6","html_url":"https://github.com/willhackett/azure-mft","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/willhackett/azure-mft","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willhackett%2Fazure-mft","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willhackett%2Fazure-mft/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willhackett%2Fazure-mft/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willhackett%2Fazure-mft/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/willhackett","download_url":"https://codeload.github.com/willhackett/azure-mft/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/willhackett%2Fazure-mft/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275013282,"owners_count":25390481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-13T02:00:10.085Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-13T18:51:19.547Z","updated_at":"2025-09-13T18:51:21.203Z","avatar_url":"https://github.com/willhackett.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# azure-mft\n\nManaged File Transfer for distributed systems using only Azure Blob Storage \u0026 Queue Storage.\n\nThis concept relies on a distributed-trust model — each transaction is verified by responding to the initiating agent to verify the integrity of the request. Requests are signed by the initiating agent and validated by requesting the public key. The agents don't directly communicate with eachother ensuring that they remain free from the restrictions of firewalls \u0026 network paths.\n\nThis model works well with a VWAN or private link, but can also work effectively via the public web.\n\n## CLI\n\n```\n  Start the MFT service, this should be done by systemctl:\n  $ mft init\n\n  Copy a file to a specific agent:\n  $ mft copy \u003cfile\u003e --destAgent=\u003cagentName\u003e --destPath=\u003cpath\u003e --overwrite=true|false\n\n  Copy a file to multiple agents:\n  $ mft copy \u003cfile\u003e --config=\u003cdestinations.yaml\u003e\n\n  Request a file from a specific agent:\n  $ mft req --sourceAgent=\u003cagentName\u003e --sourcePath=\u003cpath\u003e \u003cdestinationPath\u003e\n\n  Check for updates \u0026 update the service as needed:\n  $ mft update\n```\n\n### Configuration\n\nThe MFT configuration file is located by default in `/var/mft2/config.yaml` on unix systems and in `%PROGRAMDATA%\\mft2\\config.yaml` on Windows systems.\n\nThe user that executes the MFT service must have permission to write or read from a destination in order for a file transfer to be successful.\n\n```yaml\nversion: 1\nconfig:\n  app_insights_key: ...\n  agent:\n    name: '...'\n  mft:\n    storage_account: '...'\n  allow_files_from:\n    - 'allowed_agent_name'\n  allow_requests_from:\n    - 'allowed_agent_name'\n  exits:\n    - source_agent: 'source_agent_name'\n      file_match: '*\\\\.txt$'\n      command: 'cat {fullFilePath}'\n```\n\n### Security\n\n#### Service Account\n\nThe IAM rules lock down this solution and allow it to be operated in a distributed manner.\nThe agent name is used to determine access to resources.\n\n- Queue Storage Restrictions\n  - `{agent_name}` queue is read/write\n  - Other agent names are write-only\n- Blob Storage Restrictions\n  - `{agent_name}` container is read/write/generate SAS token\n  - `public_keys/{agent_name}` is write-only\n  - `public_keys/*` is read-only\n- AppInsights\n  - Write to app insights\n\n#### Allow Lists\n\nEach agent must specify which agents it wishes to receive files and file requests from. Requests from an agent that is not specified in one of these lists will be rejected.\n\n#### Handshake \u0026 Signed Messages\n\nWhen a file transfer is initiated, to verify the identity of the sender a handshake is performed where a signed message of intent is verified by asking for the sending agent to send its public key. This ensures that the requesting agent and signature match before a file transfer is performed.\n\nThe requester agent name is checked against a KeyID file stored within the `public_keys/*` container of the blob storage account. This allows for highly available agents to store their keys and for key rotation to perform effectively.\n\n#### SAS Tokens\n\nNo agent shoule be able to access another agent's blob storage container. They can only use the SAS token to perform a download.\n\n#### Transfer ID\n\nEach transfer is accompanied with a transfer ID which is consistent across the file transfer. This is used to ensure that all the events can be correlated in Application Insights.\n\n### Exits\n\nExits are a mechanism for executing arbitrary comands after a successful file transfer. When a transfer is complete and an exit exists that matches the source agent and file regex, the contents of command will be executed by the `azmft` process.\n\nYou can have as many exits as you want.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwillhackett%2Fazure-mft","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwillhackett%2Fazure-mft","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwillhackett%2Fazure-mft/lists"}