{"id":17965196,"url":"https://github.com/tokenrove/punchy-the-log","last_synced_at":"2026-04-28T08:03:16.150Z","repository":{"id":147613655,"uuid":"107586200","full_name":"tokenrove/punchy-the-log","owner":"tokenrove","description":"Simple demonstration of hole punching for logging","archived":false,"fork":false,"pushed_at":"2018-03-29T14:58:16.000Z","size":12,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-03T19:47:05.509Z","etag":null,"topics":["c","journaling","linux"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tokenrove.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-19T18:55:36.000Z","updated_at":"2021-03-16T22:47:26.000Z","dependencies_parsed_at":"2023-05-31T19:01:02.083Z","dependency_job_id":null,"html_url":"https://github.com/tokenrove/punchy-the-log","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tokenrove/punchy-the-log","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenrove%2Fpunchy-the-log","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenrove%2Fpunchy-the-log/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenrove%2Fpunchy-the-log/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenrove%2Fpunchy-the-log/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tokenrove","download_url":"https://codeload.github.com/tokenrove/punchy-the-log/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenrove%2Fpunchy-the-log/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32371673,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-27T20:07:02.737Z","status":"online","status_checked_at":"2026-04-28T02:00:07.250Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c","journaling","linux"],"created_at":"2024-10-29T12:10:27.071Z","updated_at":"2026-04-28T08:03:16.123Z","avatar_url":"https://github.com/tokenrove.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"This is a demonstration of the [`fallocate(FALLOC_FL_PUNCH_HOLE,\n...)`] technique for keeping an \"infinite scroll\" journal of\nmanageable size, as well as a kind of persistent pipe.  The code is\nintentionally simple and avoids many performance optimizations.\n\nThe `log` directory contains programs that work on a log where the\noffset of the next available message is written at the beginning of\nthe file, while `pipe` contains programs that work on a sort of\n\"persistent pipe\" [as described by Carlo Alberto Ferrari], where we\nuse `SEEK_DATA` to find the next message, and trim the logical size\nwith `FALLOC_FL_COLLAPSE_RANGE`.\n\nIn both cases, each invocation of `producer` writes a length-prefixed\nmessage into the log, and `consumer` reads one out, trimming the log\nas it goes.  With `-f`, `consumer` will consume continuously.  Exempli\ngratia: (requires [pv])\n\n``` shell\n$ (IFS= yes | while read x; do echo \"$x\" | ./producer ./loggy.log; done) \u0026\n$ ./consumer -f ./loggy.log | pv \u003e/dev/null\n$ kill $!\n```\n\nThis is extremely Linux-specific.  Portability patches would be\ninteresting.\n\n[Sparse files] are useful for all kinds of things ([this LWN comment]\ngives an example of using this for rewinding live TV), and maybe\naren't as well known as they should be.  Many modern filesystems\nsupport this kind of thing.\n\nDepending on how often data is produced, at EOF you may want the\nconsumer to spin (repeatedly read) or use `inotify` to get notified of\na change.  The former will tend to give lower latency, but burns a lot\nof CPU (maybe yield between reads?); the latter is friendly to other\nprocesses but introduces significant latency.  In this implementation,\nwe spin a few times and then block.\n\nThis consumer trims on every message, but it would be much faster to\nonly trim every so often, if you have a way to deal with re-reading\nduplicates in the case of a crash.\n\nIn the multiple-consumer case, you probably want a separate log per\nconsumer, although you could have some other synchronization\nmechanism.  One I've used before is having a separate trim process run\nfrom cron, when the data had timestamps and there were known freshness\nconstraints.  That doesn't look as cool as this implementation,\nthough.\n\nIn the multiple-producer case, you want to make sure you're writing\nyour whole message in one `write` call, and if you're really paranoid,\nmake sure you're [writing less than PIPE_BUF bytes].\n\nParticularly if you're okay with at-least-once consuming, you could\navoid the offset at the beginning by using [`lseek`]`(..., SEEK_DATA,\n...)` in the consumer, and starting the file with a hole.  This is the\napproach the `pipe` consumer takes.\n\n\n[`fallocate(FALLOC_FL_PUNCH_HOLE, ...)`]: http://man7.org/linux/man-pages/man2/fallocate.2.html\n[as described by Carlo Alberto Ferrari]: https://gist.github.com/CAFxX/571a1558db9a7b393579\n[pv]: http://www.ivarch.com/programs/pv.shtml\n[Sparse files]: https://en.wikipedia.org/wiki/Sparse_file\n[this LWN comment]: https://lwn.net/Articles/416234/\n[writing less than PIPE_BUF bytes]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html\n[`lseek`]: http://man7.org/linux/man-pages/man2/lseek.2.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftokenrove%2Fpunchy-the-log","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftokenrove%2Fpunchy-the-log","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftokenrove%2Fpunchy-the-log/lists"}