Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mstgnz/cdn
Cdn Api With Minio
https://github.com/mstgnz/cdn
aws cdn docker fiber go imagick minio s3
Last synced: 1 day ago
JSON representation
Cdn Api With Minio
- Host: GitHub
- URL: https://github.com/mstgnz/cdn
- Owner: mstgnz
- License: apache-2.0
- Created: 2022-02-21T13:26:58.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-12-29T17:12:51.000Z (17 days ago)
- Last Synced: 2024-12-29T17:24:58.471Z (17 days ago)
- Topics: aws, cdn, docker, fiber, go, imagick, minio, s3
- Language: Go
- Homepage:
- Size: 7.46 MB
- Stars: 33
- Watchers: 2
- Forks: 10
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
# CDN Service
## About
CDN Service is a high-performance, cloud-native content delivery solution built with Go. It provides a robust and scalable platform for managing and delivering digital assets across multiple cloud providers. The service combines modern architectural patterns with enterprise-grade features to offer:
- **Multi-Cloud Storage**: Seamless integration with MinIO and AWS S3, including Glacier support for cost-effective archival
- **Advanced Image Processing**: Real-time image resizing and optimization with support for various formats
- **Enterprise Features**: Circuit breaker pattern, rate limiting, and batch operations
- **High Performance**: Redis caching, worker pools, and optimized file handling
- **Real-Time Monitoring**: WebSocket-based live monitoring, Prometheus metrics, and comprehensive health checks
- **Developer-Friendly**: Swagger documentation, standardized API responses, and easy deployment options
Perfect for organizations needing a reliable, scalable, and feature-rich content delivery solution with support for multiple cloud providers and advanced monitoring capabilities.
## Features
### Storage
- Multi-cloud storage support (MinIO, AWS S3)
- Glacier archive support
- Bucket management
- Automatic file type detection
- Secure file handling
### Image Processing
- Real-time image resizing
- Batch processing capabilities
- Worker pool for concurrent operations
- Support for multiple image formats
- URL-based image processing
### Performance
- Redis caching layer with optimized storage
- Batch processing with configurable sizes
- Worker pool for parallel processing
- Rate limiting and request throttling
- Performance metrics and monitoring
- Dynamic ImageMagick version management
- Circuit breaker pattern for fault tolerance
- Automatic failure detection
- Graceful service degradation
- Self-healing capabilities
- Configurable thresholds and timeouts
- Real-time state monitoring
### Security
- Token-based authentication
- CORS configuration
- Rate limiting per endpoint with bypass protection
- Redis-based rate limit storage
- Request size limitations
- Trusted proxy support
### Monitoring & Observability
- Prometheus metrics
- Jaeger tracing integration
- Structured logging with zerolog
- Health check endpoints
- Detailed error tracking
- Real-time system monitoring via WebSocket
- Live performance metrics
- Active uploads count
- Upload speed
- Cache hit rate
- CPU usage
- Memory usage
- Disk usage by mount point
- Recent error logs
### Additional Features
- Environment variable configuration
- Hot reload for configuration changes
- Swagger documentation
- Docker support
- Graceful shutdown
### API Standardization
- Consistent response formats across all endpoints
- Detailed error messages and codes
- Standardized success/error patterns
- Comprehensive request validation
## Quick Start
### Prerequisites
- Go 1.22+
- Docker and Docker Compose
- MinIO Server (or AWS S3 access)
- Redis Server
### Installation
1. Clone the repository:
```bash
git clone https://github.com/mstgnz/cdn.git
cd cdn
```
2. Copy the example environment file:
```bash
cp .env.example .env
```
3. Start the services using Docker Compose:
```bash
docker-compose up -d
```
### Configuration
Edit the `.env` file with your settings:
```env
APP_PORT=9090
APP_NAME=cdn
TOKEN=your-secure-token
# MinIO Configuration
MINIO_ENDPOINT=localhost:9000
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin
MINIO_USE_SSL=false
# AWS Configuration (optional)
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_REGION=your-region
# Redis Configuration
REDIS_URL=redis://localhost:6379
# Feature Flags
DISABLE_DELETE=false
DISABLE_UPLOAD=false
DISABLE_GET=false
```
### API Usage
#### Image Operations
1. Upload an image:
```bash
curl -X POST http://localhost:9090/upload \
-H "Authorization: your-token" \
-F "[email protected]" \
-F "bucket=your-bucket" \
-F "path=your/path"
```
2. Get an image with resizing:
```bash
# Original size
http://localhost:9090/your-bucket/image.jpg
# Resize with width
http://localhost:9090/your-bucket/w:300/image.jpg
# Resize with height
http://localhost:9090/your-bucket/h:200/image.jpg
# Resize with both
http://localhost:9090/your-bucket/w:300/h:200/image.jpg
```
3. Delete an image:
```bash
curl -X DELETE http://localhost:9090/your-bucket/image.jpg \
-H "Authorization: your-token"
```
#### Bucket Operations
1. List buckets:
```bash
curl http://localhost:9090/minio/bucket-list \
-H "Authorization: your-token"
```
2. Create bucket:
```bash
curl http://localhost:9090/minio/your-bucket/create \
-H "Authorization: your-token"
```
### Monitoring
1. Connect to WebSocket for real-time updates:
```javascript
const ws = new WebSocket('ws://localhost:9090/ws');
ws.onmessage = (event) => {
const stats = JSON.parse(event.data);
console.log('System stats:', stats);
// Example stats data:
// {
// "timestamp": "2024-01-15T10:30:00Z",
// "active_uploads": 5,
// "upload_speed": 1048576, // bytes/sec
// "cache_hit_rate": 85.5, // percentage
// "cpu_usage": 45.2, // percentage
// "memory_usage": 60.8, // percentage
// "disk_usage": {
// "/data": 75, // percentage
// "/uploads": 45
// },
// "errors": [
// "Failed to process image: invalid format"
// ]
// }
};
```
2. Get current monitoring stats:
```bash
curl -H "Authorization: your-token" http://localhost:9090/monitor
```
## Kubernetes Deployment
For production deployments, we provide comprehensive Kubernetes configurations with:
- Horizontal Pod Autoscaling (3-10 pods)
- Resource quotas and limits
- Health monitoring and readiness probes
- Load balancing strategies
- Secrets management
- Persistent volume claims
For detailed instructions, see [Kubernetes Deployment Guide](k8s/README.md)
## Documentation
For detailed information, please refer to:
- [Testing Guide](docs/testing.md)
- [Troubleshooting Guide](docs/troubleshooting.md)
- [Migration Guide](docs/migration.md)
- [Changelog](CHANGELOG.md)
- [Kubernetes Deployment Guide](k8s/README.md)
- [Contributing](CONTRIBUTING.md)
## License
This project is licensed under the Apache License - see the [LICENSE](LICENSE) file for details.