{"id":25869449,"url":"https://github.com/wobondar/s3-sftp","last_synced_at":"2025-09-07T10:38:55.704Z","repository":{"id":279891596,"uuid":"940253119","full_name":"wobondar/s3-sftp","owner":"wobondar","description":"A high-performance CLI utility for transferring a large amount of files from AWS S3 to SFTP servers","archived":false,"fork":false,"pushed_at":"2025-09-01T23:56:39.000Z","size":154,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-06T21:50:53.417Z","etag":null,"topics":["aws","aws-s3","azure-sftp","file-migrations","file-transfer","s3","sftp"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wobondar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-27T21:31:32.000Z","updated_at":"2025-09-01T23:56:36.000Z","dependencies_parsed_at":"2025-02-28T10:29:13.345Z","dependency_job_id":"79d992c2-f55c-486f-a827-411e2feae1a8","html_url":"https://github.com/wobondar/s3-sftp","commit_stats":null,"previous_names":["wobondar/s3-sftp"],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/wobondar/s3-sftp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wobondar%2Fs3-sftp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wobondar%2Fs3-sftp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wobondar%2Fs3-sftp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wobondar%2Fs3-sftp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wobondar","download_url":"https://codeload.github.com/wobondar/s3-sftp/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wobondar%2Fs3-sftp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274026706,"owners_count":25209739,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-07T02:00:09.463Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-s3","azure-sftp","file-migrations","file-transfer","s3","sftp"],"created_at":"2025-03-02T05:17:28.957Z","updated_at":"2025-09-07T10:38:55.690Z","avatar_url":"https://github.com/wobondar.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS S3 to SFTP File Mover\n\n[![Release](https://github.com/wobondar/s3-sftp/actions/workflows/release.yml/badge.svg)](https://github.com/wobondar/s3-sftp/actions/workflows/release.yml)\n![Code style: gofmt](https://img.shields.io/badge/code_style-gofmt-00ADD8.svg)\n[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/wobondar/s3-sftp)\n\nA high-performance CLI utility for transferring a large number of files from AWS S3 to SFTP servers with concurrency, retry mechanisms, and robust logging.\n\n## Features\n\n- **Concurrent Processing**: Configurable concurrent downloads from S3 and uploads to SFTP\n- **Flexible Configuration**: JSON-based configuration for easy customization\n- **Robust Error Handling**: Automatic retries for failed transfers\n- **Comprehensive Logging**: Detailed logs with debug mode for troubleshooting\n- **CSV Input**: Process file transfers defined in CSV format\n\n### Prerequisites\n\n- Go 1.23 or higher\n- AWS credentials configured (via profile or environment variables)\n- Access to both S3 bucket and SFTP server\n\n## Installation\n\n```bash\ngo install github.com/wobondar/s3-sftp/cmd/s3-sftp@latest\n```\n\n### Building from Source\n\n```bash\n# Clone the repository\ngit clone https://github.com/wobondar/s3-sftp.git\ncd s3-sftp\n\n# Build the application\nmake all\n```\n\n## Usage\n\n```bash\n# Basic usage\ns3-sftp -csv input.csv\n\n# With debug mode\ns3-sftp -d -csv input.csv\n\n# With custom config file\ns3-sftp -csv input.csv -c custom_config.json\n```\n\n### Command Line Options\n\n- `-c \u003cpath\u003e`: Path to the configuration file (default: `config.json`)\n- `-csv \u003cpath\u003e`: Path to the CSV file containing file transfer definitions (required)\n- `-debug`: Enable debug mode for verbose logging\n- `-delim \u003cchar\u003e`: Delimiter for the CSV file (default: `;`)\n- `-version`: Show version information\n- `-help`: Show help information\n\n\n### CSV Format\n\nThe CSV file should be semicolon-separated with the following columns:\n\n```csv\nfolder;file;key;lastmodified;etag;size\n```\n\nExample:\n```csv\ntest_1;some_file.pdf;tests/test_1/some_file.pdf;2023-11-17T17:15:25.000Z;\"c2760c8065ee374bb92c36a6899f2016\";28500\ntest_1;another_file.pdf;tests/test_1/another_file.pdf;2023-11-17T17:15:25.000Z;\"8e985792cda4d1adba490c77a3cb76db\";22113\n```\n\n### Configuration\n\nThe application uses a JSON configuration file (`config.json` by default):\n\n```json\n{\n    \"s3_bucket_name\": \"your_s3_bucket_name\",\n    \"s3_region\": \"your_s3_region\",\n    \"aws_profile\": \"your_aws_profile\",\n    \"sftp_host\": \"your_sftp_host\",\n    \"sftp_port\": 22,\n    \"sftp_username\": \"your_sftp_username\",\n    \"sftp_password\": \"your_sftp_password\",\n    \"sftp_directory\": \"your_sftp_directory\",\n    \"sftp_connect_timeout\": 30,\n    \"concurrent_s3_downloads\": 8,\n    \"concurrent_sftp_uploads\": 16,\n    \"max_retries\": 5,\n    \"retry_delay\": 1\n}\n```\n\n## Development\n\n### Available Make Targets\n\n```bash\n$ make help\nAvailable targets:\n  all                - Check dependencies, clean, and build\n  check-deps         - Check for required dependencies\n  build              - Build the application\n  build-all          - Build for multiple platforms (Linux, macOS, Windows)\n  clean              - Remove build artifacts\n  vet                - Run go vet\n  release-check      - Check goreleaser release without publishing\n  fmt                - Format code\n  mod-tidy           - Tidy Go modules\n  run                - Build and run the application\n  help               - Show this help message\n\nExample usage:\n  make run ARGS=\"-csv input.csv -c config.json\"\n  make build-all ARGS=\"v0.0.1\"\n```\n\n## Performance Tuning\n\nFor optimal performance, adjust the following configuration parameters:\n\n- `concurrent_s3_downloads`: Number of concurrent S3 download operations\n- `concurrent_sftp_uploads`: Number of concurrent SFTP upload operations\n- `max_retries`: Maximum number of retry attempts for failed operations\n- `retry_delay`: Delay in seconds between retry attempts\n\nIn my personal experience, the optimal configuration for 4 million files was:\n\n```text\n\"concurrent_s3_downloads\": 80,\n\"concurrent_sftp_uploads\": 128,\n\"max_retries\": 5,\n\"retry_delay\": 2\n```\n\nAlso, to improve performance, you might consider increasing:\n1. maximum number of open file descriptors: `ulimit -n 30000`\n2. maximum number of pending connections (backlog)\n3. TCP SYN backlog\n\n## Troubleshooting\n\nIf you encounter issues:\n\n1. Enable debug mode with the `-d` flag\n2. Check AWS credentials and permissions\n3. Verify SFTP server connectivity and credentials\n4. Ensure the CSV file is properly formatted\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwobondar%2Fs3-sftp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwobondar%2Fs3-sftp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwobondar%2Fs3-sftp/lists"}