{"id":19533189,"url":"https://github.com/anuveyatsu/cloudflare-data-fabric","last_synced_at":"2026-06-29T11:31:22.942Z","repository":{"id":262051806,"uuid":"886087403","full_name":"anuveyatsu/cloudflare-data-fabric","owner":"anuveyatsu","description":"Cloudflare Data Fabric: Use Cloudflare's global infrastructure to build a flexible, resilient framework for data solutions.","archived":false,"fork":false,"pushed_at":"2024-11-10T06:54:00.000Z","size":4,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-12T16:15:37.029Z","etag":null,"topics":["cloudflare","data","data-lake","fabric","lakehouse","mesh"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anuveyatsu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-10T06:41:19.000Z","updated_at":"2024-11-29T17:55:24.000Z","dependencies_parsed_at":"2024-11-10T07:31:42.819Z","dependency_job_id":"3c0ec9f1-0e27-4c23-9834-6161c5348eb5","html_url":"https://github.com/anuveyatsu/cloudflare-data-fabric","commit_stats":null,"previous_names":["anuveyatsu/cloudflare-data-fabric"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/anuveyatsu/cloudflare-data-fabric","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anuveyatsu%2Fcloudflare-data-fabric","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anuveyatsu%2Fcloudflare-data-fabric/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anuveyatsu%2Fcloudflare-data-fabric/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anuveyatsu%2Fcloudflare-data-fabric/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anuveyatsu","download_url":"https://codeload.github.com/anuveyatsu/cloudflare-data-fabric/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anuveyatsu%2Fcloudflare-data-fabric/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34925718,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-29T02:00:05.398Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudflare","data","data-lake","fabric","lakehouse","mesh"],"created_at":"2024-11-11T02:05:51.498Z","updated_at":"2026-06-29T11:31:22.923Z","avatar_url":"https://github.com/anuveyatsu.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cloudflare Data Fabric\n\n**Cloudflare Data Fabric** is an open-source framework designed to help teams build scalable data solutions, such as data lakes, data lakehouses, and data meshes, using Cloudflare’s suite of products. This framework offers a data fabric approach that combines storage, compute, and security to create a seamless data layer accessible across distributed environments.\n\n## Key Features\n\n- **Centralized Data Storage**: Utilizes [Cloudflare R2](https://www.cloudflare.com/products/r2/) as the primary storage layer for data objects, offering cost-effective, serverless data storage.\n- **Serverless Data Processing**: Enables real-time data transformations and processing using [Cloudflare Workers](https://developers.cloudflare.com/workers/).\n- **Traditional ETL Workflows**: Supports traditional, long-running ETL tasks through GitHub Actions with a self-hosted runner on Hetzner servers, providing powerful compute resources for data extraction, transformation, and loading at a low cost.\n- **Authentication \u0026 Authorization**: Implements fine-grained access control with [Cloudflare Access](https://www.cloudflare.com/products/zero-trust/) for secure data access and policy enforcement.\n- **Edge Caching and Optimization**: Leverages Cloudflare’s global CDN and edge network for efficient data delivery.\n- **Multi-Purpose Data Buckets**: Provides separate public and private buckets for open and restricted data access, with role-based permissions.\n\n## Use Cases\n\n- **Data Lakes and Lakehouses**: Store raw and processed data for analysis, training, and model serving.\n- **Data Mesh Architecture**: Decentralize data storage across domains with independent data governance.\n- **Real-time Data Transformation**: Process and transform data on the fly as it’s accessed.\n\n## Getting Started\n\n### Prerequisites\n\nTo use Cloudflare Data Fabric, ensure you have:\n- A [Cloudflare account](https://dash.cloudflare.com/sign-up) with access to R2 and Workers.\n- Git installed locally for cloning and managing the repository.\n- Node.js installed (for running the setup script).\n\n### Installation\n\n1. **Clone the Repository**:\n   ```bash\n   git clone https://github.com/yourusername/cloudflare-data-fabric.git\n   cd cloudflare-data-fabric\n   ```\n\n2. **Configure Cloudflare Services**:\n   - Set up Cloudflare R2 buckets (public and private) in your Cloudflare account.\n   - Enable Cloudflare Workers for your account.\n\n3. **Install Dependencies**:\n   Run the following command to install required dependencies:\n   ```bash\n   npm install\n   ```\n\n4. **Configure Environment Variables**:\n   Create a `.env` file with your Cloudflare account details:\n   ```env\n   CLOUDFLARE_ACCOUNT_ID=your_account_id\n   CLOUDFLARE_API_TOKEN=your_api_token\n   ```\n\n### ETL Process with GitHub Actions and Hetzner\n\nFor traditional ETL tasks, we use GitHub Actions with a self-hosted runner on Hetzner servers, providing powerful resources at an affordable cost. Here’s how to set this up:\n\n1. **Set Up Hetzner Runner**:\n   - Deploy a virtual server on Hetzner with sufficient CPU, memory, and storage for ETL workloads.\n   - Install the GitHub Actions runner on your Hetzner server by following [GitHub’s self-hosted runner setup](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners).\n\n2. **Runner Configuration**:\n   - Configure your runner to scale according to ETL job requirements, allowing for efficient resource usage.\n   - Use environment variables or secrets to securely manage credentials for data sources and destinations.\n\n3. **ETL Workflow**:\n   - Define ETL workflows using GitHub Actions. Schedule them with cron syntax for regular, automated runs:\n     ```yaml\n     on:\n       schedule:\n         - cron: '0 0 * * *' # Run daily at midnight\n     ```\n   - Use GitHub’s workflow visualization to monitor each run and set up notifications for errors or job completions.\n\n4. **Data Access and Security**:\n   - Securely access data sources from the runner, potentially using VPN or SSH tunneling if needed.\n   - Store all sensitive information (API keys, credentials) in GitHub secrets for secure access during workflows.\n\n### Example Code\n\n```javascript\n// Example: Access a file in the public R2 bucket\nconst fetchData = async () =\u003e {\n  const response = await fetch(\"https://data-cloudflare.example.com/myfile.csv\");\n  const data = await response.text();\n  console.log(data);\n};\nfetchData();\n```\n\n## Contributing\n\nWe welcome contributions! Please check the [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on how to contribute to the Cloudflare Data Fabric project.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanuveyatsu%2Fcloudflare-data-fabric","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanuveyatsu%2Fcloudflare-data-fabric","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanuveyatsu%2Fcloudflare-data-fabric/lists"}