{"id":13942596,"url":"https://github.com/openbridge/s3-sftp-gateway","last_synced_at":"2025-04-10T11:53:26.154Z","repository":{"id":72465316,"uuid":"191176970","full_name":"openbridge/s3-sftp-gateway","owner":"openbridge","description":"Openbridge SFTP S3 Gateway: A broad feature set you would expect from mature, enterprise-focused SFTP service","archived":false,"fork":false,"pushed_at":"2019-06-11T00:53:30.000Z","size":135,"stargazers_count":27,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-24T10:47:53.138Z","etag":null,"topics":["aws-ec2","aws-s3","mariadb","rds","s3","scp","sftp","sftp-server","sql"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openbridge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-06-10T13:49:40.000Z","updated_at":"2025-01-19T19:09:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"788802b4-b49f-428c-b299-e697f9d198c9","html_url":"https://github.com/openbridge/s3-sftp-gateway","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openbridge%2Fs3-sftp-gateway","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openbridge%2Fs3-sftp-gateway/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openbridge%2Fs3-sftp-gateway/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openbridge%2Fs3-sftp-gateway/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openbridge","download_url":"https://codeload.github.com/openbridge/s3-sftp-gateway/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248215193,"owners_count":21066622,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-ec2","aws-s3","mariadb","rds","s3","scp","sftp","sftp-server","sql"],"created_at":"2024-08-08T02:01:56.870Z","updated_at":"2025-04-10T11:53:26.137Z","avatar_url":"https://github.com/openbridge.png","language":null,"funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"\n\u003cimg src=\"./s3.png\" alt=\"drawing\" width=\"200\"/\u003e\n\n# Openbridge SFTP S3 Transfer Gateway\nThis is a Secure File Transfer Protocol (`SFTP`) service for the transfer of data to AWS S3. The SFTP S3 Transfer Gateway supports basic file transfers as well as creating data pipelines that allow you to deliver, process and route data sources to a target warehouse system like Amazon Redshift, Amazon Redshift Spectrum, Amazon Athena or even Google BigQuery. Consolidating your data to a warehouse allows you to easily use your favorite analytics tools like Tableau, QlikView, Mode or Looker.\n\nSFTP is based on the `SSH2` protocol, which encodes activity over a secure channel. Unlike `FTP`, `SSH2`  uses a single TCP connection and sends multiple transfers, or \"channels\", over that single connection. Use of SSH allows the Openbridge SFTP S3 Transfer Gateway to been setup in HA configurations via HaProxy or AWS ELB.\n\n\n# Use Cases\n\nWhat are some use cases for Openbridge SFTP S3 Transfer Gateway?\n\n* File sharing between teams\n* Perfect for automated exports from internal systems like an ERP, ETL, MySQL, SQL Server, Oracle or other enterprise systems\n* Process exports from 3rd party systems like Salesforce (see [How to export data from ExactTarget](https://blog.openbridge.com/export-tracking-data-from-salesforce-marketing-cloud-8a0a4c1f37dc) and Adobe Analytics ([see How to export data from Adobe Analytics](https://docs.openbridge.com/data-pipelines/setting-up-adobe-clickstream-feeds)).\n* Supports secure file transfer protocol with a variety of SFTP client. The SFTP protocol is secure and commonly use for transferring files via command line or graphical interfaces.\n* Lastly, support use cases where you have an ad hoc CSV file (e.g., sales reports, media plans, lookup files or any other CSV file) that you want to get loaded into your data warehouse.\n\n# Features\n\nThe core modules being used in install are as follows;\n\n- SFTP support\n- SSL/TLS support\n- SCP support\n- SQL support\n- Fixed IP for external partner/IT whitelisting\n- Quota support\n- Dynamic blacklist support\n- Traffic shaping support\n- TCP wrappers support\n- Monitoring\n- High availability support\n- Transaction Logging\n- User/Password and Public Key Authentication\n- MySQL (MariaDB) backend\n- Amazon S3 filesystem\n- Customized User Directories\n- Throttled transfers\n- Strong encryption\n- Large file support\n- Drag and drop support\n- Activity notifications via Slack\n\n# Getting Started\n\n## Prerequisites\n * **Docker**: You will need to make sure Docker is installed on your host.\n * **MariaDB**: This is where your users, keys, logs, quotas.. are stored. You can run this locally, use Amazon RDS or an existing install within your infrastructure.\n\n\n## Step 1: Building\nIf you are interested in the Openbridge SFTP S3 Transfer Gateway, we provided a hosted version here: https://www.openbridge.com/data-pipeline/batch\n\nIf you would like a self-service, self-hosted option, we do offer a license for Openbridge SFTP S3 Transfer Gateway. Contact us here for more details: https://www.openbridge.com/contact\n\n## Step 2: Setting Up Your Environment\n\nThere are a few options when running on AWS. If you are using an IAM Role, then you can forgo the need for a key and secret. This is likely the primary use case for those deploying to AWS.\n\n```bash\nMODE=\nCRONFILE=\nMOUNT_POINT=\nAWS_ACCESS_KEY_ID=\nAWS_SECRET_ACCESS_KEY=\nAWS_DEFAULT_REGION=\nAWS_IAMROLE=\nAWS_S3BUCKET=\nMYSQL_HOST=\nMYSQL_READHOST=\nAPP_DATABASE=\nAPP_PORT=\nAPP_SYSTEM_USER=\nAPP_SYSTEM_PASSWORD=\nAPP_TEST_USER=\nAPP_TEST_PASSWORD=\nAPP_VERSION=\nCLAMD_STATUS=off\nCLAMD_HOST=\nMEMCACHE_STATUS=off\nMEMCACHE_HOST=\n```\n\n\n### 1. Mode\nIf you are deploying to AWS, then you should set `MODE` to `aws`:\n```bash\nMODE=aws\n```\nHowever, if you are running this locally or another could service, say for development or testing, then you should set `MODE` to something that makes sense for you:\n```bash\nMODE=oracle\n```\nIf you do not set `MODE`, it will use a default of `dev`.\n\n**NOTE**: Please note that when running in non AWS settings (`MODE=oracle`), the server still **requires a connection to AWS S3**. The reason for this dependency is that the underlying filesystem is based on S3 so without that testing and development becomes difficult. So can run this anywhere, you will still need to set your S3 connection details (see #4)\n\n\n### 2. IP Address\n\nThere is a local IP and a public IP for a container. The address of your instance is determined automatically based on the context of your `MODE`.\n\n* Using non AWS mode will get the primary route for your Docker service. For example: `route -n | awk '$2 ~/[1-9]+/ {print $2;}'`\n* Using `MODE=aws` will check AWS for the IP. For example: `curl http://169.254.169.254/latest/meta-data/public-ipv4 2\u003e/dev/null`\n\nIf running on `aws`, it will use:\n```bash\nPUBLICIPV4=\"$(curl http://169.254.169.254/latest/meta-data/public-ipv4 2\u003e/dev/null)\"\nLOCALIPV4=\"$(curl http://169.254.169.254/latest/meta-data/local-ipv4 2\u003e/dev/null)\"\n```\n\n```bash\nPUBLICIPV4=$(route -n | awk '$2 ~/[1-9]+/ {print $2;}')\nLOCALIPV4=$(route -n | awk '$2 ~/[1-9]+/ {print $2;}')\n```\nYou can check `/docker-entrypoint.sh` for more details on the setup process.\n\n### 3. Cron File\nIf you want to persist your container you can set it up to always be running with crond as a background process. While most everything is automated there are a few configuration items you need to set.\n\n```bash\nSHELL=/bin/bash\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\nCACHE_PREFIX={{CACHE_PREFIX}}\nAWS_ACCESS_KEY_ID={{AWS_ACCESS_KEY_ID}}\nAWS_SECRET_ACCESS_KEY={{AWS_SECRET_ACCESS_KEY}}\nAWS_S3BUCKET={{AWS_S3BUCKET}}\n*/5 * * * * /usr/bin/env bash -c /tests/test_mount.sh 2\u003e\u00261\n*/5 * * * * /usr/bin/env bash -c /tests/test_proftpd.sh 2\u003e\u00261\n59 1 * * * /usr/bin/env bash -c /tests/test_cache.sh 1000000000 --silent\n*/5 * * * * /usr/bin/env bash -c /tests/test_mysql.sh 2\u003e\u00261\n*/45 * * * * /usr/bin/env bash -c /usr/bin/ban 2\u003e\u00261\n*/30 * * * * /usr/bin/env bash -c /usr/bin/verify 2\u003e\u00261\n*/5 * * * * /usr/bin/env bash -c /usr/bin/pubkeysync 2\u003e\u00261\n```\n\nThere is a sample of this in `cron/crontab.conf`. You can use this as a starting point for your own config. Once you have your config ready, we can move to Step 2.\n\n#### Mount your `crontab.conf` config\nNext, you will want to mount your config file. The config is on your host, not in the container. To get it into your container you need to mount it from your host into the container.\n\nThe basic template is this:\n\n```docker\n-v /path/to/the/file/on/your/host/crontab.conf:/where/we/mount/in/container/crontab.conf\n```\n\nThis example shows the mount your config in docker compose format:\n```docker\nvolumes:\n  - /Github/local-path/cron/crontab.conf:/crontab.conf\n```\nIt will look the same if you are doing it via Docker run:\n```docker\n-v /Github/local-path/cron/crontab.conf:/crontab.conf\n```\nIn those examples, the `crontab.conf` located in my local GitHub folder will get mounted inside the container at `/crontab.conf`\n\nMounting your config makes it available to the startup service within your container. If you are unfamiliar with `-v` or `volumes`, check the docs from Docker.\n\n\n### 4. Mount Point For S3\nThe default mount location for a container is `/mnt`. We suggest **NOT** changing this behavior unless you know what you are doing.\n```\nMOUNT_POINT=/mnt\n```\n\n#### AWS S3 Information\nThere a couple of different paths to mount your S3 location. One path requires that you are running the service on AWS. The other path can work locally or on AWS.\n\n##### AWS Only: Using IAM Roles\n```\nAWS_DEFAULT_REGION=us-east-1\nAWS_IAMROLE=my-role\nAWS_S3BUCKET=my-sftp-test\n```\n\n##### AWS or Local: Using KEY and Secret\nAn alternate approach, one you need to use for local development, is the use of the key and secret:\n\n```\nAWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXX\nAWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXX\nAWS_DEFAULT_REGION=us-east-1\nAWS_S3BUCKET=my-sftp-test\n```\nIn all cases, you need to set your S3 bucket name and region.\n\n### 5. MariaDB (MySQL)\nThe service supports using a read replica, hence the `MYSQL_READHOST` variable. However, if you are not using a read replica simply copy the IP or hostname into both locations. This is an example:\n```bash\nMYSQL_HOST=172.17.0.1\nMYSQL_READHOST=172.17.0.1\n```\n\nThe system assumes the default `3306` port is used. If you are using a different port, included it in the variables. This is an example:\n\n```bash\nMYSQL_HOST=172.17.0.1:3308\nMYSQL_READHOST=172.17.0.1:3308\n```\nIf you are using a remote/external MySQL instance, make sure it can be reached by the container. This usually will require firewall access for the instance/host running the container. Also, please see the section on setting up user accounts for the service below.\n\n### 6. SFTP Server System Settings\nThere are a few settings you need to configure for the S3 SFTP Gateway. The first is the database name. The default name is `ftpd`:\n\n```bash\nAPP_DATABASE=ftpd\n```\nNext, you need to set up a user that is authorized to connect to the database:\n```bash\nAPP_SYSTEM_USER=foobar\nAPP_SYSTEM_PASSWORD=changeme\n```\n\n```bash\nAPP_TEST_USER=tester\nAPP_TEST_PASSWORD=changeme\n```\nThis references your configuration directory. For example, let us say you have a local configuration on your host located here: `/path/to/config/v2`.\nYou should set your `APP_VERSION` to your config. In this case `v2`:\n```bash\nAPP_VERSION=v2\n```\nLastly, you will want to make sure your config is mounted **IN** the container at runtime. Make sure your local path is mounted to `/etc/proftpd`\ninside the container.\n```bash\n-v /path/to/config:/etc/proftpd\n```\nIf you ran a build and included the config within the build, there is no need to do the volume mount. Simply set the `APP_VERSION` to whatever your config is.\n\n### 7. Optional Settings\nSee below for virus and malware scanning options.\n\nBy default these are off. Memcache **only** applies to FTPES, which is used to cache TLS sessions. \n\nClam AV can be resource intensive, so make sure it the environment you run it in is sized correctly. We have a pre-built `clamd` Docker service here: https://github.com/openbridge/clamav\n\nWhile these services are supported they require external services to be configured, deployed and managed.  They are **not** included in this project.\n\n```bash\nCLAMD_STATUS=off\nCLAMD_HOST=\nMEMCACHE_STATUS=off\nMEMCACHE_HOST=\n```\n\n# Setup And Configure User Accounts In Database\nOpenbridge currently supports public key and password authentication for both SFTP and SCP file transfer protocols. The server does not support any shell access. Also, not all SFTP commands are accepted.\n\nUsers are created and stored outside of the Openbridge application. Openbridge uses MariaDB (MySQL) to store user account information. This ensures each Openbridge container can scale horizontally. Spin up 30 Openbridge container, each will connect securely to a backend database for authorizations, whitelists, blacklists, quotas and logs.\n\n## Step 1: Creating Accounts\n\nIn this example we will have a user called `peter`. Peter needs a password! How did you generate a password? You can create a random password string like this `openssl rand -hex 16 | tr -d '\\n'`. This will result in a `USER_PASSWORD` like:\n```bash\nUSER_PASSWORD=\"00cd555ee93dec9c999a0622b00e890e\"\n```\n## Step 2: Generate Your Salt\n\nIn addition to user and password, we need to create a **salt**. Salts are used to safeguard passwords in the database. Historically a password was stored in plaintext on a system, but over time additional safeguards developed to protect a user's password against being read from the system. A salt is one of those methods.\n\nThe salt uses a cryptographic hash function, and the resulting output (but not the original password) is stored with the salt in a database. Hashing allows for later authentication without keeping and therefore risking the plaintext password in the event that the authentication data store is compromised.\n\n\nWe generate the salt like we do the password: `openssl rand -hex 32 | tr -d '\\n'`. This gave us our `USER_SALT`:\n\n```bash\nUSER_SALT=\"245f772c6e7eda0a3d09ab4a2ee55eb4a31e97c2df2fa8da0a7cf8baf2d45db\"\n```\n\n## Step 3: Hashing Your Password\nYou need to hash your password in the database. The process is simple enough and here is an example.\n\nFirst, you need to take your password `00cdCCCee93dec9c549a06b5000e890e` and salt `245f772c6e7eda0a3d99ab4a2eeVVeb4a31e97c2df2fa8da0a7cf8baf2d45db`  to generate hashed password. Here is the an example CLI using openssl:\n\n```bash\nprintf \"%s\" \"{{USER_SALT}}{{USER_PASSWORD}}\" | openssl dgst -sha256 | tr -d '\\n'\n```\nInputing the values into the CLI will look something like this:\n```bash\nprintf \"%s\" \"245f772c6e7eda0a3d09ab4a2ee55eb4a31e97c2df2fa8da0a7cf8baf2d45db00cd555ee93dec9c549a06b5000e890e\" | openssl dgst -sha256 | tr -d '\\n'\n```\n\nThis would create a hashed password for us to use:\n```bash\n564f573debf1920535b5d7cb686a474b5cbe98819fbd98e34478c3070e4948f7\n```\nLastly, you can then insert this into the `ftpd_user` table for the user this password applies:\n```sql\nINSERT IGNORE INTO ftpd_user (id, userid, passwd, subscriptionid, uid, gid, homedir, status, shell, count, accessed, modified) VALUES ('', 'peter', '564f573debf1920535b5d7cb686a474b5cbe98819fbd98e34478c3070e4948f7','test001', 2001, 2001, '/tmp/tester', 'active', '/sbin/nologin', 0, '', '');\n```\nLastly, you will need to update the `ftpd_salt` table with your userid and salt value\n\n```sql\nINSERT INTO ftpd_salt (userid,salt) VALUES ('peter','245f772c6e7eda0a3d09ab4a2ee55eb4a31e97c2df2fa8da0a7cf8baf2d45db') ON DUPLICATE KEY UPDATE salt='245f772c6e7eda0a3d09ab4a2ee55eb4a31e97c2df2fa8da0a7cf8baf2d45db';\n```\n\n## Step 4: Public Key\nSSH2 uses public keys for authentication and message protection. The very popular OpenSSH server uses a particular file format for storing public keys in files. However, that format is not portable; it uses a format very specific to OpenSSH. To ensure portable keys, we leverage the [RFC4716](http://www.faqs.org/rfcs/rfc4716.html) format for public keys.\n\n**The RFC4716 format is required**. If you do not use this format, it is unlikely to work. This means that if you wish to use your public keys with the you will need to convert them from to the [RFC4716](http://www.faqs.org/rfcs/rfc4716.html) format. Fortunately, this is supported by OpenSSH's ssh-keygen utility, e.g.:\n\n```bash\nsudo ssh-keygen -e -f /path/to/your/key.pub\n```\n\nThis will generate a RFC key which you would then create a record for in the `ftpd_userkey` table;\n\n```sql\nINSERT INTO ftpd_userkey (userid,publickey) VALUES ('peter','---- BEGIN SSH2 PUBLIC KEY ----\nComment: \"2048-bit RSA, converted by root@dude-MacBook-Pro.local fr\"\nAAAAB3NzaC1yc2EAAAADAQABAAABAQC+4ejAXI7PEa1i9J50LRQMOlHtEQ0+nRK91uBH77\ngmZR5OxuaI092ErqspYpOa4DkOPxaoU5qYUxiUbbkKwtutCgTHXuS5Wt8IZMVtKsuMGZ3j\ngUkNWcSLfyRnXK0XejNnMCSdOuauyHzD8ogRyDHznzS4kt+Ikaxr0n5rZ9e+zES5vOFML6\nSeI3zq9ROHOjxNx4cbmbMJo7aG93xPsKG3kmtI7UndbQf+/Q68qPVoLqJJh5HRDt7CExHQ\n2BAStaDpy2alMhZ1b+Ie9HRTDdZtixDrmkBsc09+cqyAATkKx5nTaHCw81SDAihGkAr309\nQXxJvRWo7TeRfZMnZR1eqp\n---- END SSH2 PUBLIC KEY ----');\n```\n\n\n# Logging\n\n\n\n## Event Logs\nThe service defaults to output extend logs to SYSLOG. This is to facilitate dispatching logs to Cloudwatch. Please note that while this **enables** the ability to send to Cloudwatch, you still need to configure your container to do so. You can check out the docs here: https://docs.docker.com/config/containers/logging/awslogs/\n\n\n```bash\nJan 21 23:27:50 ede0c5892520 local0.info proftpd[472]: {\"time\":\"2019-01-21 23:27:50,677\",\"cmd\":\"CHANNEL_OPEN\",\"full_path\":\"-\",\"remote_user\":\"tester\",\"remote_dns\":\"172.24.0.1\",\"remote_ip\":\"172.24.0.2\",\"local_ip\":\"172.24.0.2\",\"local_dns\":\"172.24.0.2\",\"protocol\":\"ssh2\",\"transfe\nJan 21 23:27:50 ede0c5892520 local0.info proftpd[472]: {\"time\":\"2019-01-21 23:27:50,679\",\"cmd\":\"CHANNEL_REQUEST\",\"full_path\":\"-\",\"remote_user\":\"tester\",\"remote_dns\":\"172.24.0.1\",\"remote_ip\":\"172.24.0.2\",\"local_ip\":\"172.24.0.2\",\"local_dns\":\"172.24.0.2\",\"protocol\":\"sftp\",\"tran\nJan 21 23:27:50 ede0c5892520 local0.info proftpd[472]: {\"time\":\"2019-01-21 23:27:50,679\",\"cmd\":\"INIT\",\"full_path\":\"-\",\"remote_user\":\"tester\",\"remote_dns\":\"172.24.0.1\",\"remote_ip\":\"172.24.0.2\",\"local_ip\":\"172.24.0.2\",\"local_dns\":\"172.24.0.2\",\"protocol\":\"sftp\",\"transfer_time\":\nJan 21 23:27:50 ede0c5892520 local0.info proftpd[472]: {\"time\":\"2019-01-21 23:27:50,681\",\"cmd\":\"REALPATH\",\"full_path\":\"-\",\"remote_user\":\"tester\",\"remote_dns\":\"172.24.0.1\",\"remote_ip\":\"172.24.0.2\",\"local_ip\":\"172.24.0.2\",\"local_dns\":\"172.24.0.2\",\"protocol\":\"sftp\",\"transfer_ti\nJan 21 23:27:50 ede0c5892520 local0.info proftpd[472]: {\"time\":\"2019-01-21 23:27:50,698\",\"cmd\":\"OPENDIR\",\"full_path\":\"-\",\"remote_user\":\"tester\",\"remote_dns\":\"172.24.0.1\",\"remote_ip\":\"172.24.0.2\",\"local_ip\":\"172.24.0.2\",\"local_dns\":\"172.24.0.2\",\"protocol\":\"sftp\",\"transfer_tim\nJan 21 23:27:50 ede0c5892520 local0.info proftpd[472]: {\"time\":\"2019-01-21 23:27:50,852\",\"cmd\":\"READDIR\",\"full_path\":\"-\",\"remote_user\":\"tester\",\"remote_dns\":\"172.24.0.1\",\"remote_ip\":\"172.24.0.2\",\"local_ip\":\"172.24.0.2\",\"local_dns\":\"172.24.0.2\",\"protocol\":\"sftp\",\"transfer_tim\nJan 21 23:27:50 ede0c5892520 local0.info proftpd[472]: {\"time\":\"2019-01-21 23:27:50,858\",\"cmd\":\"READDIR\",\"full_path\":\"-\",\"remote_user\":\"tester\",\"remote_dns\":\"172.24.0.1\",\"remote_ip\":\"172.24.0.2\",\"local_ip\":\"172.24.0.2\",\"local_dns\":\"172.24.0.2\",\"protocol\":\"sftp\",\"transfer_tim\nJan 21 23:27:50 ede0c5892520 local0.info proftpd[472]: {\"time\":\"2019-01-21 23:27:50,859\",\"cmd\":\"MLSD\",\"full_path\":\"/mnt/ebs/ftpd/tester/\",\"remote_user\":\"tester\",\"remote_dns\":\"172.24.0.1\",\"remote_ip\":\"172.24.0.2\",\"local_ip\":\"172.24.0.2\",\"local_dns\":\"172.24.0.2\",\"protocol\":\"sf\nJan 21 23:27:50 ede0c5892520 local0.info proftpd[472]: {\"time\":\"2019-01-21 23:27:50,859\",\"cmd\":\"CLOSE\",\"full_path\":\"-\",\"remote_user\":\"tester\",\"remote_dns\":\"172.24.0.1\",\"remote_ip\":\"172.24.0.2\",\"local_ip\":\"172.24.0.2\",\"local_dns\":\"172.24.0.2\",\"protocol\":\"sftp\",\"transfer_time\"\n```\n\n## Transaction Log To Database\n\nThe `ftpd_log` captures all the file transfer transactions that occur in the system. This is the primary log for events that are transacting at the SFTP application level. In an effort to minimize signal to noise in the logs, the following commands are logged: `ABOR`,`DELE`,`ERR_STOR`,`STOR` and `STOU`\n```sql\nCREATE TABLE IF NOT EXISTS `ftpd_log` (\n  `id` int(10) unsigned NOT NULL AUTO_INCREMENT DEFAULT NULL,\n  `userid` VARCHAR(50) NOT NULL,\n  `server_ip` VARCHAR(64) NOT NULL,\n  `transfer_date` DATETIME NOT NULL,\n  `operation` VARCHAR(1024) NOT NULL DEFAULT '',\n  `protocol` VARCHAR(6) NOT NULL DEFAULT '',\n  `client_ip` VARCHAR(64) NOT NULL DEFAULT '',\n  `transfer_time` FLOAT(16) unsigned NOT NULL DEFAULT '0',\n  `bytes_transfer` BIGINT(20) unsigned NOT NULL DEFAULT '0',\n  `path_hash` VARCHAR(512) NOT NULL,\n  `file_hash_type` VARCHAR(6) NOT NULL DEFAULT '',\n  `file_hash` VARCHAR(512) NOT NULL,\n  `bucket` VARCHAR(255) NOT NULL DEFAULT '',\n  `file_path` VARCHAR(1024) NOT NULL DEFAULT '',\n  `transfer_status` VARCHAR(12) NOT NULL DEFAULT '',\n  `process_status` TINYINT(1)NOT NULL DEFAULT '0',\n  `modified` TIMESTAMP NOT NULL DEFAULT NOW(),\n  `accessed` TIMESTAMP NOT NULL DEFAULT NOW() ON UPDATE now(),\n  PRIMARY KEY (`id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci COMMENT='ProFTP access log table';\n```\n\n```bash\n+----+--------+------------+---------------------+--------------------------+----------+------------+---------------+----------------+----------------------------------+----------------+-----------+--------------+-----------------------------------------+-----------------+----------------+---------------------+---------------------+\n| id | userid | server_ip  | transfer_date       | operation                | protocol | client_ip  | transfer_time | bytes_transfer | path_hash                        | file_hash_type | file_hash | bucket       | file_path                               | transfer_status | process_status | modified            | accessed            |\n+----+--------+------------+---------------------+--------------------------+----------+------------+---------------+----------------+----------------------------------+----------------+-----------+--------------+-----------------------------------------+-----------------+----------------+---------------------+---------------------+\n|  1 | tester | 172.24.0.1 | 2019-01-09 01:01:15 | STOR /summary.xls        | sftp     | 172.24.0.2 |             0 | -              | 687071838d02524f28e3ee8a63646b3f |                |           | s3sftp-test | /mnt/ebs/ftpd/tester/summary.xls        | failed          |              6 | 2019-01-09 01:01:15 | 2019-01-09 01:30:01 |\n|  2 | tester | 172.24.0.1 | 2019-01-09 01:01:16 | STOR /summary.xls        | sftp     | 172.24.0.2 |             0 | -              | 687071838d02524f28e3ee8a63646b3f |                |           | s3sftp-test | /mnt/ebs/ftpd/tester/summary.xls        | failed          |              6 | 2019-01-09 01:01:16 | 2019-01-09 01:30:01 |\n|  3 | tester | 172.24.0.1 | 2019-01-09 01:05:59 | STOR /select-s3.png      | sftp     | 172.24.0.2 |         1.002 | 79995          | e18fb25ed6434279ae96a879bada9d60 |                |           | s3sftp-test | /mnt/ebs/ftpd/tester/select-s3.png      | success         |              5 | 2019-01-09 01:05:59 | 2019-01-09 01:30:01\n```\n\n## Audit \u0026 Verification Log\n\nWhile `ftpd_log` captures overall transactions the `ftpd_process` table reflects an audit and verification of files resident on S3. Unlike `ftpd_log` this log only deals with successful transfers. The goal is to identify any file hash mismatches, corruptions or other problems that may impact the veracity of the files on S3. While there is some overlap with `ftpd_log`, this log is is meant to be authoritative for auditing a successful transfer.\n\n\n```sql\nCREATE TABLE IF NOT EXISTS `ftpd_process` (\n  `id` INT(10) unsigned NOT NULL AUTO_INCREMENT DEFAULT NULL,\n  `path_hash` VARCHAR(512) NOT NULL,\n  `file_hash` VARCHAR(512) NOT NULL,\n  `file_hash_type` VARCHAR(6) NOT NULL DEFAULT '',\n  `bucket` VARCHAR(255) NOT NULL DEFAULT '',\n  `file_path` VARCHAR(1024) NOT NULL DEFAULT '',\n  `file_linecount` VARCHAR(256) NOT NULL DEFAULT '',\n  `file_size` VARCHAR(256) NOT NULL DEFAULT '',\n  `process_status` TINYINT(1)NOT NULL,\n  `modified` TIMESTAMP NOT NULL DEFAULT NOW(),\n  `accessed` TIMESTAMP NOT NULL DEFAULT NOW() ON UPDATE now(),\n  PRIMARY KEY (`id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci COMMENT='ProFTP file processed table';\n```\n\n\n```bash\n+----+----------------------------------+----------------------------------+----------------+--------------+---------------------------------+----------------+-----------+----------------+---------------------+---------------------+\n| id | path_hash                        | file_hash                        | file_hash_type | bucket       | file_path                       | file_linecount | file_size | process_status | modified            | accessed            |\n+----+----------------------------------+----------------------------------+----------------+--------------+---------------------------------+----------------+-----------+----------------+---------------------+---------------------+\n|  1 | a2f65667aed12e1eb7184b7030f1efe1 | a7f41a2b49a84b4f6e89e59c723def89 | MD5            | my-s3-test | /mnt/ebs/ftpd/tester/664708.zip | 0              | 6733763   |              1 | 2019-01-09 01:30:02 | 2019-01-09 01:30:02 |\n+----+----------------------------------+----------------------------------+----------------+--------------+---------------------------------+----------------+-----------+----------------+---------------------+---------------------+\n```\n\nA background process is responsible for updating `ftpd_process` and `ftpd_log`\n\n\n### Processing Status Codes\nStatus codes for `ftpd_log` and `ftpd_process` denote the state of a file that has been delivered to the server.\n\n*  **0** = Unprocessed\n*  **1** = Transfer and checksum successfully processed. Success!\n*  **2** = There was an error in the file transfer to AWS. Could be a network or application failure\n*  **3** = The file checksum between the original file and the one on s3 does not match\n*  **4** = The file can not be found locally or on S3. Upload may have aborted or the file was deleted by the user\n*  **5** = The file successful transferred over the network, but is not a supported file type. This will occur if you place restrictions on file types. For example, a user trying to upload a `PNG` would get a status code file if the supported are: `(*.csv | *.txt | *.tsv | *.gz | *.tgz | *.zip)`\n*  **6** = The was a hard failure of the file transfer. This be be due to transfer, user, file or other error\n\n## Status Codes\n\nA `\"response_code\":\"226\"` in a log file indicates success.\n\nHowever, there are other status codes that may be present. See below:\n\n* accepts the STOR request with code `226` if the entire file was successfully received and stored\n* rejects the STOR request with code `425` if no TCP connection was established\n* rejects the STOR request with code `426` if the TCP connection was established but then broken by the client or by network failure or\n* rejects the STOR request with code `451`, `452`, or `552` if the server had trouble saving the file to disk.\n\n\nSee https://en.wikipedia.org/wiki/List_of_FTP_server_return_codes\n\n**NOTE**: If any \"reject\" codes are present, then you will want to have a workflow for further consideration.\n\n\n\n# Monitoring\nServices in the container are monitored via `Monit`. Monit is orchestrating the automated monitoring for various health and upkeep operations. For example, here are a few operations it cares for:\n\n* Server `PID` process\n* Server `PORT`\n* Server `DISK` size\n* `MYSQL` Database existence check\n* `MYSQL` Database write check\n* `Volume` mount check\n* `CACHE` check\n* CROND `PID` process\n\nOne thing to note is that if `Monit` detects a problem it will issue a STOP command. If you are using a Docker `--restart unless-stopped` in your docker run command this will trigger the server to automatically restart. While not activated, you can pair these checks and tests with Slack notifications (**not included**).\n\n## Examples\nThe following reflects a Monit config to monitor the server application and associated processes for `ports` , `disk` and others. You can find a collection of similar configs in `etc/monit.d/*`.\n\n```bash\n### proftpd\ncheck process s3sftp-server\n    matching \"proftpd\"\n    start program = \"/usr/sbin/proftpd -c /etc/proftpd/proftpd.conf\" with timeout 60 seconds\n    stop program = \"/bin/bash -c 'pkill syslogd'\"\n    if 3 restarts within 5 cycles then stop\n    if cpu \u003e 90% for 8 cycles then stop\n\ncheck host s3sftp-server-port with address {{LOCALIPV4}}\n     every 10 cycles\n     start program = \"/usr/sbin/proftpd -c /etc/proftpd/proftpd.conf\" with timeout 60 seconds\n     stop program = \"/bin/bash -c 'pkill syslogd'\"\n     if failed port {{APP_PORT}} for 10 cycles then stop\n\ncheck filesystem s3sftp-disk-size with path /\n     every 5 cycles\n     start program = \"/usr/sbin/proftpd -c /etc/proftpd/proftpd.conf\" with timeout 60 seconds\n     stop program = \"/bin/bash -c 'pkill syslogd'\"\n     if space usage \u003e 95% for 5 cycles then stop\n\ncheck program s3sftp-health-check with path \"/bin/bash -c '/tests/test_proftpd.sh'\"\n    every 5 cycles\n    start program = \"/usr/sbin/proftpd -c /etc/proftpd/proftpd.conf\" with timeout 60 seconds\n    stop program = \"/bin/bash -c 'pkill syslogd'\"\n    if status != 0 for 2 cycles then stop\n```\n\n# Internalization and Localization\n\nOpenbridge supports UTF-8 filename character sets. The system will perform any character set decoding checks as the file is being delivered and will automatically discover the encoding used.\n\n**Please note**: While our system tries to do the requested encoding/decoding for UTF-8, if operations fail it continues using the bytes as sent by the client. While the process fails gracefully and allows the delivery of the file in this situation there is no guarantee that downstream data processing pipelines will work properly. Please verify prior to delivery of data that it is properly encoded to UTF-8 standards.\n\n# Blocked Files\n\nTo help protect the integrity of the data sent to Openbridge, we do not allow you to deliver files of certain types (such as.exe files) because of their potential for introducing a unwanted or malicious software threats. By default, we block these files:\n\n```bash\n(ade|adp|app|ai|asa|ashx|asmx|asp|bas|bat|cdx|cer|cgi|chm|class|cmd|com|config|cpl|crt|csh|dmg|doc|docx|dll|eps|exe|fxp|ftaccess|hlp|hta|htr|htaccess|htw|html|htm|ida|idc|idq|ins|isp|its|jse|ksh|lnk|mad|maf|mag|mam|maq|mar|mas|mat|mau|mav|maw|mda|mdb|mde|mdt|mdw|mdz|msc|msh|msh1|msh1xml|msh2|msh2xml|mshxml|msi|msp|mst|ops|pdf|php|php3|php4|php5|pcd|pif|prf|prg|printer|pst|psd|rar|reg|rem|scf|scr|sct|shb|shs|shtm|shtml|soap|stm|tgz|taz|url|vb|vbe|vbs|ws|wsc|wsf|wsh|xls|xlsx|xvd)\n```\n\nHidden files are also not allowed. These files have a prefix of \"`.`\" or \"`..`\"\n\n```\n.file.txt\n.file.csv\n..file\n```\n\nAny files uploaded meeting the criteria listed in \"Blocked Files\" will result the transfer being rejected.\n\n# DNS-based Blackhole List (DNSBL) / Real-time Blackhole List (RBL)\nOpenbridge employs a DNSBL (commonly known as a \"Blocklist\"). This is a database that is queried in realtime for the purpose of obtaining an opinion on the origin of incoming hosts. The role of a DNSBL is to assess whether a particular IP Address meets acceptance policies of inbound connections. DNSBL is often used by email, web and other network services for determining and rejecting addresses known to be sources of spam, phishing and other unwanted behavior.\n\nMore information on DNS blacklists can be found here: http://en.wikipedia.org/wiki/DNSBL\n\n# Account Ban and Lockout\nOpenbridge employs a dynamic \"ban\" lists that prevents the banned user or host from logging in to the server. This will occur if our system detects 4 incorrect login attempts. The ban will last for approximately 30 minutes at which time you can attempt to login again. If you continue to have difficulties please contact support.\n\nOpenbridge can also use **[Autoban](https://github.com/openbridge/ob_APP_autoban)**, a package that leverages `mod_wrap2` to detect malicious login attempts and ban them.\n\n# IP Blacklist and Whitelist\nAccess can be controlled via `ftpd_allow` and `ftpd_deny` tables.\n\nBelow is an example of the **allow** table with allow IP addresses.\n\n```bash\n+----------------+---------+---------+---------------------+---------------------+\n| client_ip      | allowed | options | modified            | accessed            |\n+----------------+---------+---------+---------------------+---------------------+\n| 11.11.22.134  | ALL     | ALLOWED | 2019-01-09 01:45:04 | 2019-01-09 01:45:04 |\n| 2.4.96.175    | ALL     | ALLOWED | 2019-01-09 01:45:04 | 2019-01-09 01:45:04 |\n| 45.48.112.77  | ALL     | ALLOWED | 2019-01-09 01:45:05 | 2019-01-09 01:45:05 |\n| 34.32.175.2   | ALL     | ALLOWED | 2019-01-09 01:45:05 | 2019-01-09 01:45:05 |\n| 66.4.9.2      | ALL     | ALLOWED | 2019-01-09 01:45:05 | 2019-01-09 01:45:05 |\n+----------------+---------+---------+---------------------+---------------------+\n```\n# Idle Connection Time Limits\nThe app sets the maximum number of seconds a connection between the server and a client after the client has successfully authenticated. This is typically 10 minutes or 600 seconds in length. If you are idle for longer than this time allotment the server will think you are finished and disconnect your client. If this occurs a client will simply need to reconnect.\n\n# Encrypting Your Databases Connections\n\nWhen setting up the MYSQL or MariaDB instance, you may want the application level connection using SSL. The first step is to set the `GRANT` for the user and specify SSL\n\n```sql\nGRANT USAGE ON *.* TO 'name'@\u003chost REQUIRE SSL;\n```\nFlush when complete:\n\n```sql\nFLUSH PRIVILEGES;\n```\n\nTest the connection\n```bash\nmysql -h 'server'.rds.amazonaws.com --ssl_ca=/etc/pki/tls/rds-combined-ca-bundle.pem --ssl-verify-server-cert -u 'name' -p'password with no space between p and pass' -e \"show status like 'Ssl_cipher';\"\n```\nAnd you will see something like that:\n```bash\n+---------------+------------+\n| Variable_name | Value      |\n+---------------+------------+\n| Ssl_cipher    | AES256-SHA |\n+---------------+------------+\n```\n\nThe service has a `my.conf.d/client.cnf` file which is included in the build. This relies on AWS certs/keys. If you are deploying this outside AWS take note of this fact and adjust accordingly.\n\nDetails from AWS\nhttp://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html#MySQL.Concepts.SSLSupport\n\n# Hash Validation\nThe service will perform an MD5 checksum of a completed transfer and of the file at rest on Amazon S3. This test ensures that the received file at the server level matches what is resident within S3.\n\n## Considerations\nDepending on the file size and the hash function, it takes a fair amount of CPU and IO resources to calculate the result.\n\n* Only works for binary, not ASCII, uploads/downloads\n* Only works for uploads (STOR) and downloads (RETR), but not for appends (APPE) or resumed uploads/downloads (REST + RETR/STOR)\n\nSee the `Logging` section for details on how the hash validation process is logged\n\n# GeoIP Filters\nYou can the ability to block connections based on GeoIP. For example, in this case deny access to specific country IDs:\n\n```bash\n\u003cIfModule mod_geoip.c\u003e\n   GeoIPEngine      on\n   GeoIPTable       /etc/geoip/GeoIP.dat MemoryCache UTF8\n   GeoIPTable       /etc/geoip/GeoLiteCity.dat MemoryCache UTF8\n   GeoIPDenyFilter CountryCode (CN|IR|KN)\n\u003c/IfModule\u003e\n```\nIf you do not want to use this feature, simply set `GeoIPEngine` to off:\n\n```bash\nGeoIPEngine      off\n```\n\n# Anti-Virus, Malware and Trojans\nYou can employ an anti-virus toolkit to scan for viruses, trojans, and other questionable items to prevent them from ever being uploaded to our system. The process is designed to detect in real-time threats present in any incoming files.\n\nThis means any file sent to your system will be scanned prior to be fully written to the filesystem. Any files uploaded meeting your scan rejection criteria will be viewed as a threat and be rejected.\n\nA running ClamAV service is required for this to work. To activate ClamAV scanning, activate via the `env` file by setting the status to `CLAMD_STATUS` and `CLAMD_HOST`:\n\n```bash\nCLAMD_STATUS=on\nCLAMD_HOST=172.24.0.1\n```\n\n# FTP\n\nOpenbridge does support the use of FTP/FTPS as part of the provided configuration files. We recognize that there are some systems that can only deliver data via FTP. For example, many of the Adobe export processes occur via FTP. However, it should be noted that the use of FTP offers no encryption security for connection and data transport. We strongly recommend the use of SFTP or FTPES whenever possible.\n\n# Sample SQL Queries\n\nExample commands.\n```sql\nINSERT INTO ftpd_salt (userid, salt) VALUES ('tester', 'n74f624573q47ddaa4d4aaf6ff1c465d695e0a94ace0784zz801ffbd73d6dd9b') ON DUPLICATE KEY UPDATE salt='n74f624573q47ddaa4d4aaf6ff1c465d695e0a94ace0784zz801ffbd73d6dd9b';\n```\n\n```sql\nALTER TABLE ftpd_quotalimit rename column userid to name;\n```\n```sql\nCREATE INDEX users ON ftpd_user(userid) USING HASH;\n```\n```sql\nINSERT INTO `ftpd_quotalimit` (`name`, `quota_type`, `per_session`, `limit_type`, `bytes_in_avail`, `bytes_out_avail`, `bytes_xfer_avail`, `files_in_avail`, `files_out_avail`, `files_xfer_avail`) VALUES ('tester', 'user', 'false', 'soft', 214748364800, 214748364800, 214748364800, 1000, 1000, 1000);\n```\n```sql\nUPDATE ftpd_user SET passwd='c637ab6550b7ed1813f77aaaad836d9d2bb52e489697d699ae7406e3e64a7e7f' WHERE userid='tester';\n```\n```sql\nUPDATE ftpd_user SET status='inactive' WHERE userid='tester';\n```\n```sql\nINSERT IGNORE INTO ftpusers (id, userid, passwd, subscriptionid, uid, gid, homedir, status, shell, count, accessed, modified) VALUES ('', 'foobar', 'f4270936c0a84ef714a99999f2438fd421f95994e64b1165d77be989936cdf43','test', 2001, 2001, '/tmp/tester', 'active', '/sbin/nologin', 0, '', '');\n```\n```sql\nUPDATE ftpd_user SET homedir='/mnt/ebs/ftpd/foobar-00022' WHERE userid='foobar';\n```\n```sql\nDELETE from ftpd_user where userid = foobar;\n```\n```sql\nINSERT INTO `ftpd_quotalimit` (`userid`, `quota_type`, `per_session`, `limit_type`, `bytes_in_avail`, `bytes_out_avail`, `bytes_xfer_avail`, `files_in_avail`, `files_out_avail`, `files_xfer_avail`) VALUES ('tester', 'user', 'true', 'hard', 15728640, 15728640, 15728640, 100, 100, 100);\n```\n```sql\nINSERT INTO ftpd_quotatally (`userid`, bytes_in_used`, `bytes_out_used`, `bytes_xfer_used`, `files_in_used`, `files_out_used`, `files_xfer_used`) VALUES ('tester', 0, 0, 0, 0, 0, 0);\n```\n```sql\nmysql -h ${MYSQL_HOST} -u ${APP_SYSTEM_USER} -p${APP_SYSTEM_PASSWORD} -e \"use $APP_DATABASE; INSERT INTO ftpd_group (groupname, gid, members) VALUES ('ftpgroup', '2001', 'tester');\"\n```\n```sql\nUPDATE ftpd_user SET homedir='/mnt/ebs/share/foobar-00022/foobar/foobar-external' WHERE userid='foobar-external';\n```\n\n```sql\nUPDATE ftpd_log SET process_status='6' WHERE transfer_status='failed';\n```\n# Versioning\nHere are the latest releases:\n\n| Docker Tag | Git Hub Release |  Version | Alpine Version |\n|-----|-------|-----|--------|\n| latest | master  | 1.3.7 | 3.7 |\n\n\n# TODO\n\n\n# Issues\n\nIf you have any problems with or questions about this image, please contact us through a GitHub issue.\n\n# Contributing\n\nYou are invited to contribute new features, fixes, or updates, large or small; we are always thrilled to receive pull requests, and do our best to process them as fast as we can.\n\nBefore you start to code, we recommend discussing your plans through a GitHub issue, especially for more ambitious contributions. This gives other contributors a chance to point you in the right direction, give you feedback on your design, and help you find out if someone else is working on the same thing.\n\n# References\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenbridge%2Fs3-sftp-gateway","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenbridge%2Fs3-sftp-gateway","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenbridge%2Fs3-sftp-gateway/lists"}