{"id":50671177,"url":"https://github.com/punitkumar4871/distributed-file-upload-system","last_synced_at":"2026-06-08T11:04:41.290Z","repository":{"id":359210299,"uuid":"1244866644","full_name":"punitkumar4871/distributed-file-upload-system","owner":"punitkumar4871","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-20T21:07:11.000Z","size":37,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-21T02:58:14.300Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/punitkumar4871.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-20T17:16:26.000Z","updated_at":"2026-05-20T21:07:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/punitkumar4871/distributed-file-upload-system","commit_stats":null,"previous_names":["punitkumar4871/distributed-file-upload-system"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/punitkumar4871/distributed-file-upload-system","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/punitkumar4871%2Fdistributed-file-upload-system","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/punitkumar4871%2Fdistributed-file-upload-system/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/punitkumar4871%2Fdistributed-file-upload-system/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/punitkumar4871%2Fdistributed-file-upload-system/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/punitkumar4871","download_url":"https://codeload.github.com/punitkumar4871/distributed-file-upload-system/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/punitkumar4871%2Fdistributed-file-upload-system/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34059192,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-08T02:00:07.615Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-08T11:04:39.807Z","updated_at":"2026-06-08T11:04:41.286Z","avatar_url":"https://github.com/punitkumar4871.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Distributed Upload System — DevOps Setup\n\nSpring Boot chunked file upload system with full DevOps pipeline:\n**Maven → Docker → GitHub Actions → Kubernetes**\n\n---\n\n## Architecture Overview\n\n```\n┌─────────────────────────────────────────────────────────┐\n│                  TWO SEPARATE PIPELINES                 │\n└─────────────────────────────────────────────────────────┘\n\nPIPELINE 1 — Code Deployment (CI/CD)\n─────────────────────────────────────\nDeveloper pushes code\n       ↓\nGitHub Actions (ci-cd.yml)\n       ↓\nmvn test  →  mvn package\n       ↓\nDocker build \u0026 push to Docker Hub\n       ↓\nkubectl apply → Kubernetes rolling deploy\n\n\nPIPELINE 2 — File Storage (file-storage.yml)\n─────────────────────────────────────────────\nUser uploads file via browser\n       ↓\nSpring Boot receives chunks → saves to /uploads/temp/\u003cfileId\u003e/\n       ↓\nAll chunks received → merge into /uploads/final/\u003cfileName\u003e\n       ↓\nGitService sends repository_dispatch to GitHub API\n       ↓\nfile-storage.yml workflow triggers\n       ↓\nWorkflow clones private storage repo\n       ↓\ngit add / commit / push  →  file saved in storage repo\n```\n\n---\n\n## Project Structure\n\n```\n.\n├── .github/\n│   └── workflows/\n│       ├── ci-cd.yml           ← Build, test, Docker, K8s deploy\n│       └── file-storage.yml    ← Auto-commit uploaded files to GitHub\n├── k8s/\n│   ├── namespace.yml\n│   ├── configmap.yml\n│   ├── deployment.yml          ← 2 replicas, RollingUpdate, HPA\n│   ├── service.yml             ← LoadBalancer on port 80\n│   ├── hpa.yml                 ← Auto-scale 2–6 pods on CPU/memory\n│   └── secret.yml.template     ← Fill and apply manually; never commit\n├── src/\n│   └── main/java/com/example/upload/\n│       ├── UploadSystemApplication.java\n│       ├── UploadController.java   ← REST endpoints\n│       ├── FileService.java        ← Chunk save + merge + triggers Git\n│       ├── GitService.java         ← Fires repository_dispatch to GitHub\n│       ├── ChunkUploadRequest.java\n│       ├── ApiResponse.java\n│       └── WebConfig.java\n├── Dockerfile                  ← Multi-stage: Maven builder + JRE runtime\n├── pom.xml\n└── README.md\n```\n\n---\n\n## Quick Start (Local)\n\n```bash\n# 1. Build\nmvn clean package -DskipTests\n\n# 2. Run with Docker\ndocker build -t upload-system .\ndocker run -p 8080:8080 \\\n  -e GITHUB_TOKEN=ghp_yourtoken \\\n  -e GITHUB_STORAGE_REPO_OWNER=your-username \\\n  -e GITHUB_STORAGE_REPO_NAME=your-storage-repo \\\n  upload-system\n\n# 3. Open browser\nopen http://localhost:8080\n```\n\n---\n\n## GitHub Actions Setup\n\n### Secrets to add in your repo (Settings → Secrets → Actions)\n\n| Secret | Description |\n|---|---|\n| `DOCKERHUB_USERNAME` | Your Docker Hub username |\n| `DOCKERHUB_TOKEN` | Docker Hub access token |\n| `KUBECONFIG` | Base64-encoded kubeconfig file |\n| `STORAGE_REPO_TOKEN` | PAT with `repo` scope for storage repo |\n| `STORAGE_REPO_OWNER` | GitHub username/org owning storage repo |\n| `STORAGE_REPO_NAME` | Name of the private storage repository |\n\n### Encode your kubeconfig\n\n```bash\ncat ~/.kube/config | base64 | tr -d '\\n'\n# Paste output as KUBECONFIG secret\n```\n\n---\n\n## Kubernetes Deployment\n\n```bash\n# 1. Create secrets (fill in secret.yml.template first)\ncp k8s/secret.yml.template k8s/secret.yml\n# Edit k8s/secret.yml with base64-encoded values\nkubectl apply -f k8s/secret.yml\n\n# 2. Deploy everything\nkubectl apply -f k8s/namespace.yml\nkubectl apply -f k8s/configmap.yml\nkubectl apply -f k8s/deployment.yml\nkubectl apply -f k8s/service.yml\nkubectl apply -f k8s/hpa.yml\n\n# 3. Watch rollout\nkubectl rollout status deployment/upload-system -n upload-system\n\n# 4. Get external IP\nkubectl get svc -n upload-system\n```\n\n### Important: Shared Storage for Multi-Replica Deployments\n\nWith 2+ replicas, a chunk uploaded to Pod A must be readable by Pod B for the merge.\nThe `deployment.yml` uses a `PersistentVolumeClaim` with `ReadWriteMany`.\n\n**Local (Minikube/Kind):** use NFS or Longhorn\n**Cloud (EKS/GKE/AKS):** use EFS (AWS) / Filestore (GCP) / Azure Files\n\n---\n\n## File Storage to GitHub — How It Works\n\n1. User finishes uploading → Spring Boot merges all chunks\n2. `FileService` calls `GitService.triggerFileStorageWorkflow(fileName)`\n3. `GitService` POSTs a `repository_dispatch` event to GitHub API\n4. `file-storage.yml` workflow wakes up, clones your private storage repo\n5. File is copied and committed with a timestamp message\n6. Files are organized by date: `uploads/2026/05/21/filename.mp4`\n\n### Environment variables for GitService\n\n```bash\nGITHUB_TOKEN=ghp_your_token_with_repo_scope\nGITHUB_STORAGE_REPO_OWNER=your-github-username\nGITHUB_STORAGE_REPO_NAME=your-private-storage-repo\n```\n\n---\n\n## REST API Endpoints\n\n| Method | Path | Description |\n|---|---|---|\n| `POST` | `/api/upload/chunk` | Upload a single chunk |\n| `POST` | `/api/upload/merge/{fileId}` | Merge all chunks into final file |\n| `GET` | `/api/download/{fileName}` | Download merged file |\n| `GET` | `/api/health` | Health check (used by K8s probes) |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpunitkumar4871%2Fdistributed-file-upload-system","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpunitkumar4871%2Fdistributed-file-upload-system","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpunitkumar4871%2Fdistributed-file-upload-system/lists"}