{"id":15459663,"url":"https://github.com/jorropo/linux2ipfs","last_synced_at":"2025-06-13T05:33:02.777Z","repository":{"id":39256151,"uuid":"450677719","full_name":"Jorropo/linux2ipfs","owner":"Jorropo","description":"Small pipeline and extreme-performance oriented IPFS implementation to upload files and deltas to pinning services very fast.","archived":false,"fork":false,"pushed_at":"2023-08-29T18:18:07.000Z","size":733,"stargazers_count":31,"open_issues_count":3,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-08T11:12:20.439Z","etag":null,"topics":["ipfs","pinning"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jorropo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-22T00:22:22.000Z","updated_at":"2024-08-10T13:54:32.000Z","dependencies_parsed_at":"2024-06-19T16:26:39.756Z","dependency_job_id":"f47e5538-b1ef-444c-b9b2-38d6e0331eb9","html_url":"https://github.com/Jorropo/linux2ipfs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Jorropo/linux2ipfs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jorropo%2Flinux2ipfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jorropo%2Flinux2ipfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jorropo%2Flinux2ipfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jorropo%2Flinux2ipfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jorropo","download_url":"https://codeload.github.com/Jorropo/linux2ipfs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jorropo%2Flinux2ipfs/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259589295,"owners_count":22880945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ipfs","pinning"],"created_at":"2024-10-01T23:09:29.529Z","updated_at":"2025-06-13T05:33:02.743Z","avatar_url":"https://github.com/Jorropo.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# linux2ipfs\n\nLinux to IPFS is a project that aims to pipeline streaming a linux repository to [estuary.tech](https://estuary.tech/).\n\n# Requirement\n\n- Linux \u003e= v5.3\n- A reflinking-able FS (btrfs, XFS (optional) or ZFS).\n- Modifications times enabled on your FS.\n- A reasonably updated time clock.\n- A 64bits kernel.\n\n# Performance\n\nThis run has been recorded on the `9ad2437383dfe9e1c8851e4d4f35f0bdd8580027` (Feb 14 2022) commit.\n\nIt compares to [ipld/go-car](https://github.com/ipld/go-car) `v2.1.1`.\n\nThe test was adding a ~32Gig files of `/dev/urandom` data.\n\nIt was done on an RAID1 btrfs of 2 sata SSDs.\n\n## linux2ipfs\n\n```console\n$ time linux2ipfs-test -driver car -concurrent-chunkers 32 f\nbafybeigpau6jiycwadkj74svqbpaibqblmttkjzb53avaxohmahnfojzgm\nupdated\n\nreal\t1m25.407s\nuser\t3m18.423s\nsys\t0m46.643s\n```\nFor info that is faster than the write speed of my disks.\n\nThis is achieved thx to reflinking, basically instead of copying the data to the car. It \"reflinks\" it. That means creating a Copy-On-Write copy, so they share the same on disk data, however in case of any modification a copy on write of that precise part of the file is created.\n\n```console\n$ btrfs fi du *\n     Total   Exclusive  Set shared  Filename\n  31.91GiB       0.00B    31.91GiB  f\n  31.97GiB    65.37MiB    31.91GiB  out.1.car\n```\nThe overhead is only `65.37MiB` the rest `31.91GiB` data is shared between the original `f` file and the car.\n\n## go-car\n\n```console\n$ time car c -o car.car f\n\nreal\t14m59.512s\nuser\t7m47.838s\nsys\t1m20.531s\n```\nAs you can see, **linux2ipfs is 10 times faster !**\n\nThat easy to explain why, most of it is because `go-car` actually copies the file (that the main point however there are other things):\n```console\n$ btrfs fi du *\n     Total   Exclusive  Set shared  Filename\n  31.92GiB    31.92GiB       0.00B  car.car\n  31.91GiB    31.91GiB       0.00B  f\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjorropo%2Flinux2ipfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjorropo%2Flinux2ipfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjorropo%2Flinux2ipfs/lists"}