{"id":13548506,"url":"https://github.com/google/crfs","last_synced_at":"2025-05-15T03:05:45.593Z","repository":{"id":38419604,"uuid":"177216162","full_name":"google/crfs","owner":"google","description":"CRFS: Container Registry Filesystem","archived":false,"fork":false,"pushed_at":"2024-10-09T18:52:42.000Z","size":109,"stargazers_count":1384,"open_issues_count":5,"forks_count":65,"subscribers_count":38,"default_branch":"master","last_synced_at":"2025-05-13T12:09:46.863Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-22T22:19:22.000Z","updated_at":"2025-05-02T08:52:36.000Z","dependencies_parsed_at":"2024-06-18T12:32:09.243Z","dependency_job_id":"1733f233-98d2-460a-ad8c-6fecd7414197","html_url":"https://github.com/google/crfs","commit_stats":{"total_commits":36,"total_committers":9,"mean_commits":4.0,"dds":0.5555555555555556,"last_synced_commit":"71d77da419c90be7b05d12e59945ac7a8c94a543"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fcrfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fcrfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fcrfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Fcrfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google","download_url":"https://codeload.github.com/google/crfs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254264765,"owners_count":22041793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T12:01:11.264Z","updated_at":"2025-05-15T03:05:45.567Z","avatar_url":"https://github.com/google.png","language":"Go","funding_links":[],"categories":["Go","Image"],"sub_categories":[],"readme":"# CRFS: Container Registry Filesystem\n\nDiscussion: https://github.com/golang/go/issues/30829\n\n## Overview\n\n**CRFS** is a read-only FUSE filesystem that lets you mount a\ncontainer image, served directly from a container registry (such as\n[gcr.io](https://gcr.io/)), without pulling it all locally first.\n\n## Background\n\nStarting a container should be fast. Currently, however, starting a\ncontainer in many environments requires doing a `pull` operation from\na container registry to read the entire container image from the\nregistry and write the entire container image to the local machine's\ndisk. It's pretty silly (and wasteful) that a read operation becomes a\nwrite operation. For small containers, this problem is rarely noticed.\nFor larger containers, though, the pull operation quickly becomes the\nslowest part of launching a container, especially on a cold node.\nContrast this with launching a VM on major cloud providers: even with\na VM image that's hundreds of gigabytes, the VM boots in seconds.\nThat's because the hypervisors' block devices are reading from the\nnetwork on demand. The cloud providers all have great internal\nnetworks. Why aren't we using those great internal networks to read\nour container images on demand?\n\n## Why does Go want this?\n\nGo's continuous build system tests Go on [many operating systems and\narchitectures](https://build.golang.org/), using a mix of containers\n(mostly for Linux) and VMs (for other operating systems). We\nprioritize fast builds, targeting 5 minute turnaround for pre-submit\ntests when testing new changes. For isolation and other reasons, we\nrun all our containers in a single-use fresh VMs. Generally our\ncontainers do start quickly, but some of our containers are very large\nand take a long time to start. To work around that, we've automated\nthe creation of VM images where our heavy containers are pre-pulled.\nThis is all a silly workaround. It'd be much better if we could just\nread the bytes over the network from the right place, without the all\nthe hoops.\n\n## Tar files\n\nOne reason that reading the bytes directly from the source on demand\nis somewhat non-trivial is that container images are, somewhat\nregrettably, represented by *tar.gz* files, and tar files are\nunindexed, and gzip streams are not seekable. This means that trying\nto read 1KB out of a file named `/var/lib/foo/data` still involves\npulling hundreds of gigabytes to uncompress the stream, to decode the\nentire tar file until you find the entry you're looking for. You can't\nlook it up by its path name.\n\n## Introducing Stargz\n\nFortunately, we can fix the fact that *tar.gz* files are unindexed and\nunseekable, while still making the file a valid *tar.gz* file by\ntaking advantage of the fact that two gzip streams can be concatenated\nand still be a valid gzip stream. So you can just make a tar file\nwhere each tar entry is its own gzip stream.\n\nWe introduce a format, **Stargz**, a **S**eekable\n**tar.gz** format that's still a valid tar.gz file for everything else\nthat's unaware of these details.\n\nIn summary:\n\n* That traditional `*.tar.gz` format is: `Gzip(TarF(file1) + TarF(file2) + TarF(file3) + TarFooter))`\n* Stargz's format is: `Gzip(TarF(file1)) + Gzip(TarF(file2)) + Gzip(TarF(file3_chunk1)) + Gzip(F(file3_chunk2)) + Gzip(F(index of earlier files in magic file), TarFooter)`, where the trailing ZIP-like index contains offsets for each file/chunk's GZIP header in the overall **stargz** file.\n\nThis makes images a few percent larger (due to more gzip headers and\nloss of compression context between files), but it's plenty\nacceptable.\n\n## Converting images\n\nIf you're using `docker push` to push to a registry, you can't use\nCRFS to mount the image. Maybe one day `docker push` will push\n*stargz* files (or something with similar properties) by default, but\nnot yet. So for now we need to convert the storage image layers from\n*tar.gz* into *stargz*. There is a tool that does that. **TODO: examples**\n\n## Operation\n\nWhen mounting an image, the FUSE filesystem makes a couple Docker\nRegistry HTTP API requests to the container registry to get the\nmetadata for the container and all its layers.\n\nIt then does HTTP Range requests to read just the **stargz** index out\nof the end of each of the layers. The index is stored similar to how\nthe ZIP format's TOC is stored, storing a pointer to the index at the\nvery end of the file. Generally it takes 1 HTTP request to read the\nindex, but no more than 2. In any case, we're assuming a fast network\n(GCE VMs to gcr.io, or similar) with low latency to the container\nregistry. Each layer needs these 1 or 2 HTTP requests, but they can\nall be done in parallel.\n\nFrom that, we keep the index in memory, so `readdir`, `stat`, and\nfriends are all served from memory. For reading data, the index\ncontains the offset of each file's `GZIP(TAR(file data))` range of the\noverall *stargz* file. To make it possible to efficiently read a small\namount of data from large files, there can actually be multiple\n**stargz** index entries for large files. (e.g. a new gzip stream\nevery 16MB of a large file).\n\n## Union/overlay filesystems\n\nCRFS can do the aufs/overlay2-ish unification of multiple read-only\n*stargz* layers, but it will stop short of trying to unify a writable\nfilesystem layer atop. For that, you can just use the traditional\nLinux filesystems.\n\n## Using with Docker, without modifying Docker\n\nIdeally container runtimes would support something like this whole\nscheme natively, but in the meantime a workaround is that when\nconverting an image into *stargz* format, the converter tool can also\nproduce an image variant that only has metadata (environment,\nentrypoints, etc) and no file contents. Then you can bind mount in the\ncontents from the CRFS FUSE filesystem.\n\nThat is, the convert tool can do:\n\n**Input**: `gcr.io/your-proj/container:v2`\n\n**Output**: `gcr.io/your-proj/container:v2meta` + `gcr.io/your-proj/container:v2stargz`\n\nWhat you actually run on Docker or Kubernetes then is the `v2meta`\nversion, so your container host's `docker pull` or equivalent only\npulls a few KB. The gigabytes of remaining data is read lazily via\nCRFS from the `v2stargz` layer directly from the container registry.\n\n## Status\n\nWIP. Enough parts are implemented \u0026 tested for me to realize this\nisn't crazy. I'm publishing this document first for discussion while I\nfinish things up. Maybe somebody will point me to an existing\nimplementation, which would be great.\n\n## Discussion\n\nSee https://github.com/golang/go/issues/30829\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Fcrfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle%2Fcrfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Fcrfs/lists"}