{"id":50559601,"url":"https://github.com/devxoshakya/singularity-ingestion-pipeline","last_synced_at":"2026-06-04T11:01:10.201Z","repository":{"id":341564471,"uuid":"1169915330","full_name":"devxoshakya/singularity-ingestion-pipeline","owner":"devxoshakya","description":" A high-performance data ingestion pipeline built with Bun for syncing local files to Cloudflare R2 storage. Designed for speed, reliability, and zero-copy streaming, this pipeline efficiently handles hundreds of files with intelligent deduplication and retry logic.","archived":false,"fork":false,"pushed_at":"2026-03-02T11:44:27.000Z","size":997,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-02T14:33:54.910Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devxoshakya.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-01T12:26:22.000Z","updated_at":"2026-03-02T11:53:57.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/devxoshakya/singularity-ingestion-pipeline","commit_stats":null,"previous_names":["devxoshakya/singularity-ingestion-pipeline"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/devxoshakya/singularity-ingestion-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devxoshakya%2Fsingularity-ingestion-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devxoshakya%2Fsingularity-ingestion-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devxoshakya%2Fsingularity-ingestion-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devxoshakya%2Fsingularity-ingestion-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devxoshakya","download_url":"https://codeload.github.com/devxoshakya/singularity-ingestion-pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devxoshakya%2Fsingularity-ingestion-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33901305,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-04T02:00:06.755Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-04T11:01:09.316Z","updated_at":"2026-06-04T11:01:10.191Z","avatar_url":"https://github.com/devxoshakya.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Singularity Ingestion Pipeline\n\nA high-performance data ingestion pipeline built with Bun for syncing local files to Cloudflare R2 storage. Designed for speed, reliability, and zero-copy streaming, this pipeline efficiently handles hundreds of files with intelligent deduplication and retry logic.\n\n## 🚀 Features\n\n- **⚡ Blazing Fast**: Built on Bun's zero-copy streaming for maximum throughput\n- **🔄 Recursive Directory Scanning**: Automatically syncs all files in subdirectories\n- **🎯 Smart Deduplication**: Bulk-fetches existing R2 objects to skip re-uploads\n- **🔁 Intelligent Retry Logic**: Exponential backoff for transient failures\n- **   Configurable Concurrency**: Optimized pool-based parallelism (default: 30 concurrent uploads)\n- **📝 Auto Content-Type Detection**: Automatic MIME type assignment based on file extensions\n- **📊 Real-time Progress**: Detailed logging with upload/skip/failure counts\n- **🛡️ Production Ready**: Exit codes for CI/CD integration\n\n## 📋 Prerequisites\n\n- [Bun](https://bun.sh) v1.0 or higher\n- Cloudflare R2 account with API credentials\n- Node.js v18+ (for compatibility)\n\n## 🔧 Installation\n\n1. **Clone the repository**:\n   ```bash\n   git clone \u003crepository-url\u003e\n   cd singularity-ingestion-pipeline\n   ```\n\n2. **Install dependencies**:\n   ```bash\n   bun install\n   ```\n\n3. **Configure environment variables**:\n   \n   Create a `.env` file in the project root:\n   ```env\n   R2_ACCESS_KEY_ID=your_access_key_id_here\n   R2_SECRET_ACCESS_KEY=your_secret_access_key_here\n   ```\n\n4. **Update R2 configuration** (optional):\n   \n   Edit `index.ts` to match your R2 bucket settings:\n   ```typescript\n   const r2 = new S3Client({\n     accessKeyId: process.env.R2_ACCESS_KEY_ID!,\n     secretAccessKey: process.env.R2_SECRET_ACCESS_KEY!,\n     bucket: \"your-bucket-name\",\n     endpoint: \"https://YOUR_ACCOUNT_ID.r2.cloudflarestorage.com\",\n   });\n   ```\n\n## 🎯 Usage\n\n### Basic Sync\n\nSync all files from the `docs/` directory to R2:\n\n```bash\nbun run index.ts\n```\n\n### Expected Output\n\n```\n🔍 Scanning ./docs (including all subdirectories)…\n📦 Found 15 local files (including subdirectories)\n☁️  Fetching existing R2 keys under \"docs\"…\n   3 objects already in bucket\n⏭️  Skipped (exists): docs/README.md\n✅ Uploaded: docs/api/endpoints.md\n✅ Uploaded: docs/guides/getting-started.md\n✅ Uploaded: docs/guides/advanced/configuration.md\n...\n\n─────────────────────────────────\n🎉 Sync complete!\n   ✅ Uploaded : 12\n   ⏭️  Skipped  : 3\n   ❌ Failed   : 0\n─────────────────────────────────\n```\n\n## 📁 Project Structure\n\n```\nsingularity-ingestion-pipeline/\n├── docs/                          # Files to be synced to R2\n│   ├── api/                       # API documentation\n│   │   └── endpoints.md\n│   ├── guides/                    # User guides\n│   │   ├── getting-started.md\n│   │   └── advanced/\n│   │       └── configuration.md\n│   └── README.md\n├── index.ts                       # Main pipeline script\n├── package.json                   # Dependencies\n├── tsconfig.json                  # TypeScript config\n└── README.md                      # This file\n```\n\n## ⚙️ Configuration\n\n### Performance Tuning\n\nAdjust these constants in `index.ts` based on your needs:\n\n```typescript\nconst CONCURRENCY = 30;       // Parallel upload limit\nconst MAX_RETRIES = 3;        // Retry attempts per file\nconst RETRY_BASE_MS = 500;    // Base delay for exponential backoff\n```\n\n**Recommended Settings by File Size:**\n\n| File Size    | Concurrency | Use Case              |\n|--------------|-------------|-----------------------|\n| \u003c 100 KB     | 50          | Small docs/configs    |\n| 100 KB - 1 MB| 30          | Medium docs/images    |\n| 1 MB - 10 MB | 10          | Large images/PDFs     |\n| \u003e 10 MB      | 5           | Videos/archives       |\n\n### Supported File Types\n\nThe pipeline automatically detects and sets content types for:\n\n- **Markdown**: `.md`, `.markdown` → `text/markdown; charset=utf-8`\n- **Text**: `.txt` → `text/plain; charset=utf-8`\n- **HTML**: `.html` → `text/html; charset=utf-8`\n- **JSON**: `.json` → `application/json`\n- **Images**: `.jpg`, `.jpeg`, `.png`, `.svg`\n- **PDF**: `.pdf` → `application/pdf`\n- **Default**: `application/octet-stream` (for unrecognized types)\n\nExtend the `getContentType()` function to support additional formats.\n\n## 🔄 How It Works\n\n1. **Recursive Scan**: Reads all files in `docs/` and subdirectories\n2. **Bulk Fetch**: Fetches all existing R2 object keys in one request\n3. **Smart Skip**: Compares local files against R2 to avoid re-uploads\n4. **Pooled Upload**: Processes files with bounded concurrency\n5. **Retry Logic**: Retries failed uploads with exponential backoff\n6. **Report Results**: Displays detailed statistics on completion\n\n## 🛠️ Advanced Usage\n\n### Custom Directory and Prefix\n\nModify the last line in `index.ts`:\n\n```typescript\n// Sync a different directory to a different R2 prefix\nawait syncDirectoryToR2(\"./my-assets\", \"assets\");\n```\n\n### Multiple bucket permissions**: Apply least-privilege access to your R2 bucket\n5. **Enable encryption**: Use R2's encryption at rest features\n\n## 🤝 Contributing\n\nContributions are welcome! Please:\n\n1. Fork the repository\n2. Create a feature branch: `git checkout -b feature/my-feature`\n3. Commit changes: `git commit -am 'Add new feature'`\n4. Push to branch: `git push origin feature/my-feature`\n5. Open a Pull Request\n\n## 📚 Additional Resources\n\n- [Bun Documentation](https://bun.sh/docs)\n- [Cloudflare R2 Documentation](https://developers.cloudflare.com/r2/)\n- [S3-Compatible API Reference](https://docs.aws.amazon.com/AmazonS3/latest/API/)\n- [Getting Started Guide](./docs/guides/getting-started.md)\n- [Advanced Configuration](./docs/guides/advanced/configuration.md)\n\n## 📝 License\n\nThis project is created using Bun v1.3.10. [Bun](https://bun.com) is a fast all-in-one JavaScript runtime.\n\n## 🙏 Acknowledgments\n\n- Built with [Bun](https://bun.sh) for maximum performance\n- Powered by [Cloudflare R2](https://www.cloudflare.com/products/r2/) for scalable object storage\n- Inspired by modern DevOps practices for efficient data pipelines\n\n---\n\n**Made by the Singularity team**","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevxoshakya%2Fsingularity-ingestion-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevxoshakya%2Fsingularity-ingestion-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevxoshakya%2Fsingularity-ingestion-pipeline/lists"}