{"id":20140641,"url":"https://github.com/goffinet/mftlab","last_synced_at":"2025-09-03T06:36:31.883Z","repository":{"id":116034310,"uuid":"406125251","full_name":"goffinet/mftlab","owner":"goffinet","description":null,"archived":false,"fork":false,"pushed_at":"2021-09-14T19:17:25.000Z","size":35,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-02T23:28:29.925Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/goffinet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-13T20:50:51.000Z","updated_at":"2021-09-14T19:16:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"aa95b7cd-e9b7-45ca-a280-f7bed519e13a","html_url":"https://github.com/goffinet/mftlab","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/goffinet/mftlab","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goffinet%2Fmftlab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goffinet%2Fmftlab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goffinet%2Fmftlab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goffinet%2Fmftlab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/goffinet","download_url":"https://codeload.github.com/goffinet/mftlab/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goffinet%2Fmftlab/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273403950,"owners_count":25099299,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-03T02:00:09.631Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T21:52:57.816Z","updated_at":"2025-09-03T06:36:31.855Z","avatar_url":"https://github.com/goffinet.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LAB: MQMFT Agent in Container\r\n\r\n## Quick start\r\n\r\nThis lab requires only Docker and docker-compose, here on a Scaleway instance.\r\n\r\n```bash\r\nscw instance server create type=DEV1-M zone=nl-ams-1 image=centos_8 root-volume=l:40G ip=new name=mqftlab\r\nscw instance server list zone=nl-ams-1\r\n```\r\n\r\n```bash\r\ndnf -y install git\r\ngit clone https://github.com/goffinet/mftlab\r\ncd mftlab\r\nbash -x startup.sh\r\n```\r\n\r\nOr launch it with Terraform:\r\n\r\n```bash\r\ngit clone https://github.com/goffinet/mftlab\r\ncd mftlab\r\nterraform init\r\nterraform apply -auto-approve\r\n```\r\n\r\n## 1. Introduction\r\n\r\nThis lab introduces how to use MQ Managed File Transfer Agent container.\r\n\r\nIBM MQ Managed File Transfer transfers files between systems in a managed and auditable way, regardless of file size or the operating systems used. You can use Managed File Transfer to build a customized, scalable, and automated solution that enables you to manage, trust, and secure file transfers. Managed File Transfer eliminates costly redundancies, lowers maintenance costs, and maximizes your existing IT investments.\r\n\r\n- IBM MQ Knowledge Centre ([https://www.ibm.com/docs/en/ibm-mq/9.2?topic=overview-managed-file-transfer](https://www.ibm.com/docs/en/ibm-mq/9.2?topic=overview-managed-file-transfer))\r\n- IBM MQ Container ([https://hub.docker.com/r/ibmcom/mq](https://hub.docker.com/r/ibmcom/mq))\r\n- IBM MQ Managed File Transfer Container - ([https://hub.docker.com/r/ibmcom/mqmft](https://hub.docker.com/r/ibmcom/mqmft))\r\n\r\n## 2.Basic Requirements\r\n\r\n- Linux Operation System. Preferably RHEL 8.1 or Ubuntu 18.x.\r\n- As this lab involves usage of containers, this lab requires Docker Container Runtime and docker-compose to be installed on your machine.\r\n\r\n```bash\r\nsetup_docker_ce() {\r\necho $(date -Is) Setup Docker\r\ndnf -y install epel-release\r\ndnf -y install git htop\r\ndnf -y config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo\r\ndnf install docker-ce --nobest --allowerasing -y\r\nsystemctl enable --now docker\r\ndnf -y install python3\r\npip3 install docker-compose\r\necho $(date -Is) Setup Docker OK\r\n}\r\nsetup_docker_ce\r\n```\r\n\r\n## 3. Analyze the docker-compose file\r\n\r\nIn this lab, you will create from the [`docker-compose.yml`](https://github.com/goffinet/mftlab/blob/main/docker-compose.yml) file:\r\n\r\n- A queue manager `MQMFT` in a Docker container using the Docker image from DockerHub.([https://hub.docker.com/r/ibmcom/mq](https://hub.docker.com/r/ibmcom/mq)) This queue manager will be used as Coordination, Command and Agent queue manager. [We use here custom build](https://github.com/goffinet/mftlab/tree/main/images/qm).\r\n- Two MQ MFT agents, `AGENT10` and `AGENT20` running in separate containers ([https://hub.docker.com/r/ibmcom/mqmft](https://hub.docker.com/r/ibmcom/mqmft)).\r\n- A [DNS server for `mqmftlab.com`](https://github.com/goffinet/mftlab/blob/main/images/bind/lib/mqmftlab.com.hosts) local domain and a [data generator](https://github.com/goffinet/mftlab/tree/main/images/datagen).\r\n\r\nOn the command line:\r\n\r\n- You will create a resource monitor to automatically transfer files from source file system to destination file system.\r\n\r\n[`agentconfig.json`](https://github.com/goffinet/mftlab/blob/main/images/mftagent/agentconfig.json) - A JSON file containing the configuration information required for creating agent containers. [Here](https://github.com/ibm-messaging/mft-cloud/tree/master/docs/agentconfig.md) are the details of each attribute in the JSON file.\r\n\r\nA datagenerator container will generate fake data to `/mountpath/agent20/flow2/input/new` on AGENT10 agent.\r\n\r\n`/mountpath` – Path on the host file system mounted into container as `/mountpath`. This will be the directory where the agent will pick files to transfer in the local `./data/\u003cagent\u003e/\u003cpeer\u003e/\u003cflow\u003e` folder with this subfolder structure:\r\n\r\n```\r\n.\r\n├── input\r\n│   ├── new\r\n│   ├── old\r\n│   └── tmp\r\n└── output\r\n    ├── new\r\n    ├── old\r\n    └── tmp\r\n```\r\n\r\n## 4. Launch the lab\r\n\r\n```bash\r\ngit clone https://github.com/goffinet/mftlab.git\r\ncd mftlab\r\ndocker-compose up --build -d\r\ndocker-compose ps\r\ndocker-compose logs qm\r\ndocker-compose logs agent10\r\n```\r\n\r\nRun `dspmq` command inside the container and verify queue manager is running:\r\n\r\n```bash\r\ndocker exec -it mftlab_qm_1 dspmq\r\n```\r\n\r\n## 5. Check if transfer is working\r\n\r\nNow that you have configured queue manager and agents, it's time to automate transfers using a resource monitor. Resource monitor is major feature of MQ Managed File Transfer Agent. It helps to automatically trigger transfers at the occurrence of an event, for example arrival of a file with certain pattern name in a directory or MQ Queue. You can read more about [Resource Monitors](https://www.ibm.com/docs/en/ibm-mq/9.2?topic=resources-mft-resource-monitoring-concepts) in Knowledge Center.\r\n\r\nRemember you mounted `./data/agent10/agent20/flow1/input/new` of the host file system into `agent10` container as `/mountpath/agent20/flow1/input/new` directory. Similarly, `./data/agent20/agent10/flow1/output/new` was mounted as `/mountpath/agent10/flow1/output/new` of `agent20` container.\r\n\r\nPlease give all permissions on the data transfer folder on the Docker host before to start any transfer:\r\n\r\n```bash\r\nchmod -R 777 data/\r\n```\r\n\r\nRun the following command into the source agent container:\r\n\r\n```bash\r\ndocker exec -it mftlab_agent10_1 ls -lh /mountpath/agent20/flow1/input/new/\r\nls -lh data/agent10/agent20/flow1/input/new/\r\n```\r\n\r\nFor your convenience the `/mountpath/agent20/flow1/input/new/` directory already has some \".csv\" files.\r\n\r\nRun the following command to status of available agents:\r\n\r\n```bash\r\ndocker exec -it mftlab_agent10_1 fteListAgents\r\n```\r\n\r\nThe output would list the agents and their status.\r\n\r\nBefore setting up resource monitor, let's verify transfer works. Submit a transfer request using `fteCreateTransfer` command.\r\n\r\n```bash\r\ndocker exec -it mftlab_agent10_1 fteCreateTransfer -p MQMFT -t binary -sa AGENT10 -sm MQMFT -da AGENT20 -dm MQMFT -de overwrite -dd \"/mountpath/agent10/flow1/output/new/\" \"/mountpath/agent20/flow1/input/new/*\"\r\n```\r\n\r\nView the status of transfer by running the `mqfts` utility. This utility displays transfer status by parsing `capture0.log` file located in source agent's log directory.\r\n\r\nTo view more details of the transfer, run `mqfts -–id=\u003ctransfer id\u003e`. For example:\r\n\r\n```bash\r\ndocker exec -it mftlab_agent10_1 mqfts\r\ndocker exec -it mftlab_agent10_1 mqfts --id=414d51204d514d46542020202020202044bfbd60019b0040\r\nls -lh data/agent20/agent10/flow1/output/new/\r\n```\r\n\r\n## 6. Automate transfer with Resource monitor\r\n\r\nNow it's time to automate transfers using a resource monitor. You will create a Directory type resource monitor that monitors a directory for certain pattern of files. It will transfer file from that directory when files of matching pattern are placed in the directory.\r\n\r\nIn the below example you will create a resource monitor on AGENT10 that monitors `/mountpath/agent20/flow2/input/new` directory every 15 seconds for \".csv\" files and transfers them to `/mountpath/agent10/flow2/output/new/` folder on the destination agent AGENT20.\r\n\r\nThe `fteCreateTransfer -gt` option creates a file in the current directory. You may not have access to current directory. Hence task.xml file will be created in /mountpath directory.\r\n\r\nNow run the following commands to create transfer definition for the monitor AGENT10_AGENT20_FLOW2.\r\n**Important note: The `$` must be prefixed with escape character `\\` on bash shell, otherwise it will be ignored when the command is run.**\r\n\r\n```bash\r\ndocker exec -it mftlab_agent10_1 fteCreateTransfer -gt /mountpath/task.xml -sa AGENT10 -sm MQMFT -da AGENT20 -dm MQMFT -sd delete -de overwrite -dd \"/mountpath/agent10/flow2/output/new/\" \"\\${FilePath}\"\r\n```\r\n\r\nThen run the following command to create resource monitor:\r\n\r\n```bash\r\ndocker exec -it mftlab_agent10_1 fteCreateMonitor -p MQMFT -ma AGENT10 -mn AGENT10_AGENT20_FLOW2 -md \"/mountpath/agent20/flow2/input/new/\" -pi 15 -pu SECONDS -pt wildcard -tr \"noSizeChange=2,*.csv\" -f -mt /mountpath/task.xml\r\n```\r\n\r\nVerify the resource monitor creation by running the following command:\r\n\r\n```bash\r\ndocker exec -it mftlab_agent10_1 fteListMonitors -v -ma AGENT10 -mn AGENT10_AGENT20_FLOW2\r\n```\r\n\r\nNow that resource monitor has been created and started, exit the shell of `agent10` container to come back to host systems shell.\r\n\r\n1. For your convenience the `/mountpath/agent20/flow2/input/new/` get fake data generated randomly.\r\n\r\nAfter few seconds, verify that transfer has completed, and files are indeed available in `/mountpath/agent10/flow2/output/new/` output directory on AGENT20 or in the local directory `data/agent20/agent10/flow2/output/new/`.\r\n\r\nYou can also verify the transfer status by logging into `agent10` container and running mqfts command\r\n\r\n```bash\r\ndocker exec -it mftlab_agent10_1 mqfts\r\n```\r\n\r\nThis completes the setting up of automated transfers using resource monitors.\r\n\r\nLogout of `agent10` container shell, if you had logged in.\r\n\r\nResource monitor triggers a transfer only if any new files arrive in the monitored directory or any existing files are modified. To verify this, create a new .csv file by running the following command:\r\n\r\n```bash\r\ntouch data/agent10/agent20/flow2/input/new/newsalary.csv\r\n```\r\n\r\nA transfer will be trigged when resource monitor starts the next poll. The polling interval of resource monitor is set to 15 seconds, a transfer will be triggered within 15 seconds. Verify the contents of `data/agent20/agent10/flow2/output/new/` after 15 seconds.\r\n\r\nYou can also verify the resource monitor transfer triggers transfers only when files of a matching pattern arrive in the monitored folder. Create a file by running the following command\r\n\r\n```bash\r\ntouch data/agent10/agent20/flow2/input/new/oldperks.xls\r\n```\r\n\r\nVerify the contents of `data/agent10/agent20/flow2/input/new/` after 15 seconds, `oldperks.xls` file should not be present.\r\n\r\n## 7. Other commands of Managed File Transfer\r\n\r\nIt's now time to explore other commands of Managed File Transfer\r\n\r\nStop resource monitor using the command:\r\n\r\n```bash\r\ndocker exec -it mftlab_agent10_1 fteStopMonitor -ma AGENT10 -mn AGENT10_AGENT20_FLOW2\r\n```\r\n\r\nPlace new files with .csv extension in `data/agent10/agent20/flow2/input/new/` on the host file system and see if the new files are transferred.\r\n\r\nStart monitor using the command:\r\n\r\n```bash\r\ndocker exec -it mftlab_agent10_1 fteStartMonitor -ma AGENT10 -mn AGENT10_AGENT20_FLOW2\r\n```\r\n\r\nVerify new csv files you placed are transferred.\r\n\r\nStop agent using the command:\r\n\r\n```bash\r\ndocker exec -it mftlab_agent10_1 fteStopAgent AGENT10\r\n```\r\n\r\nVerify the container is still running.\r\n\r\nRestart the agent using the command:\r\n\r\n```bash\r\ndocker exec -it mftlab_agent10_1 fteStartAgent AGENT10\r\n```\r\n\r\n## 8. Backup a monitor\r\n\r\n## 9. Delete a monitor\r\n\r\n## 10. Import a monitor\r\n\r\n## 11. Shutdown the lab\r\n\r\nOnce you have explored other commands of Managed File Transfer, stop all containers with the commands below\r\n\r\n```bash\r\ndocker-compose down\r\n```\r\n\r\n## 12. Add an agent\r\n\r\nIf you want add an agent, you must adapt :\r\n\r\n1. the `docker-compose.yml` file,\r\n2. the qm image build,\r\n3. the `agentconfig.json` file\r\n4. and the dns zone.\r\n\r\nProject: add a bridge agent and a SFTP server\r\n\r\n## License\r\n\r\nIssued from [IBM MQ Managed File Transfer Container](https://github.com/ibm-messaging/mft-cloud)\r\n\r\nThe Dockerfiles and associated code and scripts are licensed under the [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0.html).\r\nLicenses for the products installed within the images are as follows:\r\n\r\n- [IBM MQ Advanced for Developers](http://www14.software.ibm.com/cgi-bin/weblap/lap.pl?la_formnum=Z125-3301-14\u0026li_formnum=L-APIG-BMKG5H) (International License Agreement for Non-Warranted Programs). This license may be viewed from an image using the `LICENSE=view` environment variable as described above or by following the link above.\r\n- [IBM MQ Advanced](http://www14.software.ibm.com/cgi-bin/weblap/lap.pl?la_formnum=Z125-3301-14\u0026li_formnum=L-APIG-BMJJBM) (International Program License Agreement). This license may be viewed from an image using the `LICENSE=view` environment variable as described above or by following the link above.\r\n\r\nNote: The IBM MQ Advanced for Developers license does not permit further distribution and the terms restrict usage to a developer machine.\r\n\r\n## Copyright\r\n\r\n© Copyright IBM Corporation 2020, 2021\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoffinet%2Fmftlab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoffinet%2Fmftlab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoffinet%2Fmftlab/lists"}