{"id":29784802,"url":"https://github.com/aaronlmathis/cloudawsync","last_synced_at":"2026-07-03T10:06:30.046Z","repository":{"id":304495214,"uuid":"1018943937","full_name":"aaronlmathis/CloudAWSync","owner":"aaronlmathis","description":"CloudAWSync is a cloud file synchronization agent written in Go. It provides real-time and scheduled synchronization between local directories and cloud storage (currently AWS S3), with support for multiple sync modes, comprehensive monitoring, and security features.","archived":false,"fork":false,"pushed_at":"2025-07-27T17:46:31.000Z","size":74,"stargazers_count":15,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-11T03:16:08.864Z","etag":null,"topics":["aws","aws-s3","cloud-backup","file-sync","file-synchronization"],"latest_commit_sha":null,"homepage":"https://deepthought.sh/projects/cloudawsync/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aaronlmathis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-13T11:56:39.000Z","updated_at":"2025-08-10T10:31:42.000Z","dependencies_parsed_at":"2025-07-13T14:19:06.210Z","dependency_job_id":"5a392620-5d1d-4d53-bc73-4e0edb06c3c9","html_url":"https://github.com/aaronlmathis/CloudAWSync","commit_stats":null,"previous_names":["aaronlmathis/cloudawsync"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/aaronlmathis/CloudAWSync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aaronlmathis%2FCloudAWSync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aaronlmathis%2FCloudAWSync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aaronlmathis%2FCloudAWSync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aaronlmathis%2FCloudAWSync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aaronlmathis","download_url":"https://codeload.github.com/aaronlmathis/CloudAWSync/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aaronlmathis%2FCloudAWSync/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35081285,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-03T02:00:05.635Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-s3","cloud-backup","file-sync","file-synchronization"],"created_at":"2025-07-27T16:14:26.004Z","updated_at":"2026-07-03T10:06:30.029Z","avatar_url":"https://github.com/aaronlmathis.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CloudAWSync - Cloud File Synchronization Agent\n\nCloudAWSync is a cloud file synchronization agent written in Go. It provides real-time and scheduled synchronization between local directories and cloud storage (currently AWS S3), with support for multiple sync modes, monitoring, and security features.\n\n## Features\n\n### Core Functionality\n- **Multi-mode Synchronization**: Real-time, scheduled, or hybrid sync modes\n- **AWS S3 Support**: Full S3 integration with support for S3-compatible services\n- **Modular Architecture**: Easy to extend for other cloud providers\n- **SystemD Integration**: Designed to run as a system service\n\n### Performance \u0026 Reliability\n- **High Concurrency**: Configurable concurrent upload/download workers\n- **Bandwidth Control**: Optional bandwidth limiting\n- **Retry Logic**: Automatic retry with exponential backoff\n- **Integrity Verification**: MD5 hash verification for all transfers\n- **Efficient Batching**: Event batching to reduce redundant operations\n\n### Monitoring \u0026 Metrics\n- **Prometheus Integration**: Comprehensive metrics collection\n- **System Monitoring**: CPU, memory, disk usage tracking\n- **Transfer Statistics**: Bandwidth, file counts, error rates\n- **Health Reporting**: Sync status and error reporting\n\n### Security \u0026 Safety\n- **Encryption Support**: Server-side encryption for S3\n- **File Filtering**: Configurable include/exclude patterns\n- **Path Validation**: Protection against path traversal attacks\n- **Permission Preservation**: Maintains file permissions when possible\n\n## Quick Start\n\n### Installation Methods\n\nCloudAWSync includes an installation script that handles system setup, user creation, and service configuration automatically:\n\n```bash\n# Clone the repository\ngit clone https://github.com/aaronlmathis/CloudAWSync.git\ncd CloudAWSync\n\n# Make the script executable\nchmod +x install.sh\n\n# Run automated installation\nsudo ./install.sh install\n```\n\n### Docker Deployment\n```bash\n# Build image\ndocker build -t cloudawsync:latest .\n\n# Run container\ndocker-compose up -d\n\n# View logs\ndocker-compose logs -f cloudawsync\n```\n\n### Installation Script Reference\n\nThe `install.sh` script provides complete installation and management capabilities for CloudAWSync. It automates system user creation, directory setup, service installation, and maintenance tasks.\n\n### Basic Usage\n\n```bash\nsudo ./install.sh \u003ccommand\u003e\n```\n\n### Available Commands\n\n#### Installation Commands\n\n**`install` (default)** - Complete system installation\n```bash\nsudo ./install.sh install\n# or simply\nsudo ./install.sh\n```\n\n**What it does:**\n- Checks for required dependencies (Go, systemctl)\n- Installs Go 1.21 if not present\n- Creates dedicated `cloudawsync` system user and group\n- Creates directory structure:\n  - `/opt/cloudawsync/` - Installation directory\n  - `/etc/cloudawsync/` - Configuration directory  \n  - `/var/log/cloudawsync/` - Log directory\n- Builds CloudAWSync binary with optimization flags\n- Generates and installs sample configuration\n- Creates systemd service with security hardening\n- Sets up log rotation via logrotate\n- Installs bash completion\n- Provides next-step instructions\n\n#### Service Management Commands\n\n**`status`** - Show comprehensive service status\n```bash\nsudo ./install.sh status\n```\n- Displays systemd service status\n- Shows recent log entries (last 20 lines)\n- No root privileges required for viewing\n\n**`restart`** - Restart the CloudAWSync service\n```bash\nsudo ./install.sh restart\n```\n- Cleanly restarts the service\n- Useful after configuration changes\n\n**`logs`** - Follow service logs in real-time\n```bash\nsudo ./install.sh logs\n```\n- Equivalent to `journalctl -u cloudawsync -f`\n- Press Ctrl+C to exit\n\n#### Maintenance Commands\n\n**`uninstall`** - Remove CloudAWSync\n```bash\nsudo ./install.sh uninstall\n```\n\n**What it removes:**\n- Stops and disables the systemd service\n- Removes systemd service file\n- Removes logrotate configuration\n- Removes bash completion\n- Removes installation directory (`/opt/cloudawsync/`)\n- Removes log directory (`/var/log/cloudawsync/`)\n- Removes system user and group\n- **Note**: Configuration files in `/etc/cloudawsync/` are preserved\n\n**`help`** - Show usage information\n```bash\n./install.sh help\n# or\n./install.sh --help\n./install.sh -h\n```\n\n### Installation Directory Structure\n\nAfter installation, CloudAWSync uses this directory structure:\n\n```\n/opt/cloudawsync/              # Installation directory (owned by cloudawsync:cloudawsync)\n└── cloudawsync                # Main binary (executable)\n\n/etc/cloudawsync/              # Configuration directory (owned by root:root)\n├── config.yaml                # Main configuration file\n└── config.yaml.example        # Example configuration (if available)\n\n/var/log/cloudawsync/          # Log directory (owned by cloudawsync:cloudawsync)\n└── cloudawsync.log            # Application logs (rotated daily)\n\n/etc/systemd/system/           # SystemD integration\n└── cloudawsync.service        # Service definition\n\n/etc/logrotate.d/              # Log rotation\n└── cloudawsync                # Log rotation configuration\n\n/etc/bash_completion.d/        # Shell completion\n└── cloudawsync                # Bash completion script\n```\n\n---\n\n### SystemD Service Configuration\n\nThe script creates a production-ready systemd service with security hardening:\n\n```ini\n[Unit]\nDescription=CloudAWSync - Cloud File Synchronization Agent\nAfter=network.target network-online.target\nWants=network-online.target\n\n[Service]\nType=simple\nUser=cloudawsync\nGroup=cloudawsync\nWorkingDirectory=/opt/cloudawsync\nExecStart=/opt/cloudawsync/cloudawsync -config=/etc/cloudawsync/config.yaml -daemon=true\nRestart=always\nRestartSec=5\nStandardOutput=journal\nStandardError=journal\nSyslogIdentifier=cloudawsync\n\n# Security settings\nNoNewPrivileges=true\nPrivateTmp=true\nProtectSystem=strict\nProtectHome=read-only\nReadWritePaths=/var/log/cloudawsync /etc/cloudawsync\n\n# Resource limits\nMemoryMax=512M\nCPUQuota=50%\n\n[Install]\nWantedBy=multi-user.target\n```\n\n---\n\n#### Post-Installation Configuration\n\nAfter installation, you must configure CloudAWSync before starting:\n\n```bash\n# Edit the main configuration file\nsudo nano /etc/cloudawsync/config.yaml\n```\n\n**Required configuration:**\n- AWS credentials (access_key_id, secret_access_key)\n- S3 bucket name\n- Local directories to sync\n- Sync modes and schedules\n\n**Example minimal configuration:**\n```yaml\naws:\n  region: \"us-east-1\"\n  s3_bucket: \"your-backup-bucket\"\n  access_key_id: \"YOUR_ACCESS_KEY_ID\"\n  secret_access_key: \"YOUR_SECRET_ACCESS_KEY\"\n\ndirectories:\n  - local_path: \"/home/user/Documents\"\n    remote_path: \"documents\"\n    sync_mode: \"realtime\"\n    recursive: true\n    enabled: true\n```\n### Advanced Configuration\n```yaml\nperformance:\n  max_concurrent_uploads: 10\n  max_concurrent_downloads: 5\n  upload_chunk_size: 10485760  # 10MB\n  retry_attempts: 5\n  bandwidth_limit: 10485760    # 10MB/s\n\nsecurity:\n  encryption_enabled: true\n  max_file_size: 1073741824    # 1GB\n  denied_extensions:\n    - \"*.tmp\"\n    - \"*.lock\"\n    - \".DS_Store\"\n\nmetrics:\n  enabled: true\n  port: 9090\n  collect_interval: \"30s\"\n```\n#### Service Management\n\n```bash\n# Start the service\nsudo systemctl start cloudawsync\n\n# Check status\nsudo ./install.sh status\n\n# Follow logs\nsudo ./install.sh logs\n\n# Restart after config changes\nsudo ./install.sh restart\n\n# Stop the service\nsudo systemctl stop cloudawsync\n```\n\n### Troubleshooting Installation\n\n#### Permission Issues\n```bash\n# Verify installation directories exist with correct ownership\nls -la /opt/cloudawsync/\nls -la /etc/cloudawsync/\nls -la /var/log/cloudawsync/\n\n# Fix ownership if needed\nsudo chown -R cloudawsync:cloudawsync /opt/cloudawsync/\nsudo chown -R cloudawsync:cloudawsync /var/log/cloudawsync/\n```\n\n#### Service Issues\n```bash\n# Check if service file exists\nls -la /etc/systemd/system/cloudawsync.service\n\n# Reload systemd configuration\nsudo systemctl daemon-reload\n\n# Check service status\nsudo systemctl status cloudawsync\n\n# View detailed logs\nsudo journalctl -u cloudawsync -n 50\n```\n\n#### Dependency Issues\n```bash\n# Verify Go installation\ngo version\n\n# Check if binary was built correctly\n/opt/cloudawsync/cloudawsync -version\n\n# Test configuration generation\nsudo -u cloudawsync /opt/cloudawsync/cloudawsync -generate-config\n```\n\n### Log Management\n\nThe script configures automatic log rotation:\n\n```bash\n# Log rotation configuration\ncat /etc/logrotate.d/cloudawsync\n\n# Manually rotate logs\nsudo logrotate -f /etc/logrotate.d/cloudawsync\n\n# View current log\nsudo tail -f /var/log/cloudawsync/cloudawsync.log\n```\n\n### Bash Completion\n\nAfter installation, bash completion is available for the CloudAWSync binary:\n\n```bash\n# Tab completion works for options\n/opt/cloudawsync/cloudawsync -\u003cTAB\u003e\n\n# Available completions: -config, -daemon, -generate-config, -help, -log-level, -version\n```\n\n### Uninstallation\n\nTo completely remove CloudAWSync:\n\n```bash\n# Stop and remove service\nsudo ./install.sh uninstall\n\n# Manual cleanup if needed (configuration files are preserved by default)\nsudo rm -rf /etc/cloudawsync/\n```\n\n**Note**: The uninstall command preserves configuration files in `/etc/cloudawsync/` by design. Remove them manually if you want a complete cleanup.\n\n## Configuration\n\nThe configuration file uses YAML format. Here's a minimal example:\n\n```yaml\naws:\n  region: \"us-east-1\"\n  s3_bucket: \"my-backup-bucket\"\n  s3_prefix: \"cloudawsync/\"\n  access_key_id: \"YOUR_ACCESS_KEY\"\n  secret_access_key: \"YOUR_SECRET_KEY\"\n\ndirectories:\n  - local_path: \"/home/user/Documents\"\n    remote_path: \"documents\"\n    sync_mode: \"realtime\"\n    recursive: true\n    enabled: true\n    filters:\n      - \"*.tmp\"\n      - \"*.lock\"\n      - \".DS_Store\"\n\n  - local_path: \"/home/user/Pictures\"\n    remote_path: \"pictures\"\n    sync_mode: \"scheduled\"\n    schedule: \"0 2 * * *\"  # Daily at 2 AM\n    recursive: true\n    enabled: true\n\nlogging:\n  level: \"info\"\n  format: \"json\"\n  output_path: \"/var/log/cloudawsync/cloudawsync.log\"\n\nmetrics:\n  enabled: true\n  port: 9090\n  path: \"/metrics\"\n\nperformance:\n  max_concurrent_uploads: 5\n  max_concurrent_downloads: 5\n  upload_chunk_size: 5242880  # 5MB\n  retry_attempts: 3\n  retry_delay: \"5s\"\n```\n\n### SystemD Service\n\n1. **Generate service file**:\n```bash\n./cloudawsync -generate-systemd\n```\n\n2. **Install service**:\n```bash\nsudo cp cloudawsync.service /etc/systemd/system/\nsudo systemctl daemon-reload\nsudo systemctl enable cloudawsync\nsudo systemctl start cloudawsync\n```\n\n3. **Check status**:\n```bash\nsudo systemctl status cloudawsync\njournalctl -u cloudawsync -f\n```\n\n---\n\n## Configuration Reference\n\n### AWS Configuration\n- `region`: AWS region (default: us-east-1)\n- `s3_bucket`: S3 bucket name (required)\n- `s3_prefix`: Prefix for all uploaded files\n- `access_key_id`: AWS access key ID\n- `secret_access_key`: AWS secret access key\n- `session_token`: AWS session token (optional)\n- `endpoint`: Custom S3 endpoint for S3-compatible services\n\n### Directory Configuration\n- `local_path`: Local directory to sync (absolute path required)\n- `remote_path`: Remote path in S3 bucket\n- `sync_mode`: \"realtime\", \"scheduled\", or \"both\"\n- `schedule`: Cron expression for scheduled sync\n- `recursive`: Sync subdirectories recursively\n- `enabled`: Enable/disable this directory\n- `filters`: File patterns to exclude\n\n### Performance Tuning\n- `max_concurrent_uploads`: Number of simultaneous uploads\n- `max_concurrent_downloads`: Number of simultaneous downloads\n- `upload_chunk_size`: Chunk size for multipart uploads\n- `retry_attempts`: Number of retry attempts on failure\n- `retry_delay`: Delay between retries\n- `bandwidth_limit`: Bandwidth limit in bytes/second (0 = unlimited)\n\n### Security Settings\n- `encryption_enabled`: Enable S3 server-side encryption\n- `max_file_size`: Maximum file size to sync\n- `allowed_extensions`: Whitelist of file extensions\n- `denied_extensions`: Blacklist of file extensions\n\n## Monitoring\n\n### Prometheus Metrics\n\nCloudAWSync exposes comprehensive metrics at `/metrics` endpoint (default port 9090):\n\n- **Bandwidth**: Upload/download bytes\n- **File Operations**: Counts and durations by operation type\n- **System Resources**: CPU, memory, disk usage\n- **Sync Statistics**: Files processed, errors, last sync time\n- **Performance**: Active goroutines, queue sizes\n\n### Logging\n\nStructured logging with configurable levels and outputs:\n- **Levels**: debug, info, warn, error\n- **Formats**: json, text\n- **Outputs**: file, stdout\n- **Rotation**: Configurable log rotation\n\n---\n\n## Architecture\n\n### Components\n\n1. **Cloud Provider Interface**: Abstraction for cloud storage services\n2. **File Watcher**: Real-time file system monitoring with batching\n3. **Sync Engine**: Core synchronization logic with worker pools\n4. **Metrics Collector**: Prometheus-based metrics collection\n5. **Configuration Manager**: YAML-based configuration with validation\n6. **Service Manager**: Main service orchestration\n\n### Data Flow\n\n```\nFile System Events → File Watcher → Sync Engine → Cloud Provider\n                                        ↓\n                                  Metrics Collector\n```\n\n### Concurrency Model\n\n- **Worker Pools**: Separate pools for uploads and downloads\n- **Event Batching**: Reduces redundant operations\n- **Rate Limiting**: Configurable concurrency limits\n- **Graceful Shutdown**: Proper resource cleanup\n\n---\n\n## Security Considerations\n\n### File System Security\n- Path validation prevents directory traversal\n- Configurable file filters\n- Permission preservation\n- Atomic file operations\n\n### Cloud Security\n- Server-side encryption support\n- Secure credential handling\n- TLS for all communications\n- IAM role support\n\n### Operational Security\n- Minimal required permissions\n- Secure defaults\n- Audit logging\n- Error handling without information leakage\n\n---\n\n## Development\n\n### Building from Source\n\n```bash\n# Clone repository\ngit clone https://github.com/aaronlmathis/CloudAWSync.git\ncd CloudAWSync\n\n# Install dependencies\ngo mod tidy\n\n# Build\ngo build -o cloudawsync\n\n# Run tests\ngo test ./...\n\n# Build for production\ngo build -ldflags=\"-s -w\" -o cloudawsync\n```\n\n### Project Structure\n\n```\nCloudAWSync/\n├── main.go                     # Main application entry point\n├── go.mod                      # Go module definition\n├── internal/\n│   ├── config/                 # Configuration management\n│   ├── interfaces/             # Core interfaces\n│   ├── providers/              # Cloud provider implementations\n│   │   └── s3.go               # AWS S3 provider\n│   ├── watcher/                # File system watching\n│   ├── engine/                 # Sync engine\n│   ├── metrics/                # Metrics collection\n│   ├── service/                # Main service\n│   └── utils/                  # Utility functions\n└── README.md\n```\n\n### Adding New Cloud Providers\n\n1. Implement the `CloudProvider` interface in `internal/interfaces/interfaces.go`\n2. Add provider-specific configuration\n3. Update service factory methods\n4. Add tests\n\n## Troubleshooting\n\n### Common Issues\n\n1. **Permission Denied**:\n   - Check file permissions on sync directories\n   - Verify AWS IAM permissions\n   - Check systemd service user permissions\n\n2. **High Memory Usage**:\n   - Reduce concurrent worker counts\n   - Decrease chunk sizes\n   - Enable log rotation\n\n3. **Sync Failures**:\n   - Check network connectivity\n   - Verify AWS credentials\n   - Review file filters\n   - Check disk space\n\n### Debug Mode\n\nEnable debug logging for detailed information:\n```bash\n./cloudawsync -log-level=debug -daemon=false\n```\n\n### Health Checks\n\nMonitor service health using:\n- Systemd status: `systemctl status cloudawsync`\n- Logs: `journalctl -u cloudawsync -f`\n- Metrics: `curl http://localhost:9090/metrics`\n\n---\n\n## Performance Optimization\n\n### Tuning Guidelines\n\n1. **Concurrent Workers**: Start with 5, adjust based on system resources\n2. **Chunk Size**: 5MB works well for most use cases\n3. **Batch Delay**: 2-5 seconds for event batching\n4. **Bandwidth Limiting**: Set if network capacity is limited\n\n### Resource Requirements\n\n- **CPU**: 1-2 cores recommended\n- **Memory**: 256MB minimum, 512MB recommended\n- **Disk**: Minimal (logs and temporary files)\n- **Network**: Varies based on sync volume\n\n## License\n\nThis project is licensed under the GNU General Public License v3.0 or later. See the [LICENSE](LICENSE) file for details.\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests\n5. Submit a pull request\n\n## Support\n\n- **Issues**: GitHub Issues\n- **Documentation**: This README and inline code comments\n- **Security**: Report security issues privately\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faaronlmathis%2Fcloudawsync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faaronlmathis%2Fcloudawsync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faaronlmathis%2Fcloudawsync/lists"}