https://github.com/ukhomeoffice/dq_sftp-nats-dmz
https://github.com/ukhomeoffice/dq_sftp-nats-dmz
hodq
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/ukhomeoffice/dq_sftp-nats-dmz
- Owner: UKHomeOffice
- License: mit
- Created: 2018-02-02T16:43:41.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2018-11-16T15:33:46.000Z (over 6 years ago)
- Last Synced: 2025-02-16T21:26:02.973Z (4 months ago)
- Topics: hodq
- Language: Python
- Size: 11.7 KB
- Stars: 0
- Watchers: 17
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
### Background
This application originated from code written to collect OAG flight schedules via a 3rd party SFTP provider, a regex change was made and the code re-purposed to process NATS data.
Due to only one source of data and a business requirement to service all environments with a live feed, the code `sftp_nats_prod1.py` firstly downloads and virus scans the files from the remote landing directory and places them on the server. At this point the data is now availible for use. The same script then re-uploads the files it downloaded back to the third party SFTP site for use by other environments. eg. dev / pre-prod
`sftp_nats_prod2.py` takes all the files that `sftp_nats_prod1.py` re-uploaded, downloads them again re-uploads them again to an arbitary amount of extra destinations on the remote SFTP provider, in this case 2, Dev / Preprod.
`sftp_nats_client_maytech.py` is placed on any client server which needs to collect the data uploaded by `sftp_nats_prod2.py`
A number of limiting factors gave birth to the code above.
These include:
- No ability to simply sync the directories on the upstream SFTP site
(which would have eliminated almost all code entirely)
- Suppliers still operating via FTP and SFTP
#### Legacy Code
Streaming / MQ systems are planned to process this type of feed in future, further the business requirement of servicing multiple environments no longer exists for this code base.