{"id":28386572,"url":"https://github.com/blitzcode/conduit-chunked","last_synced_at":"2025-07-28T12:38:05.917Z","repository":{"id":18346157,"uuid":"21525948","full_name":"blitzcode/conduit-chunked","owner":"blitzcode","description":"Efficiently processing data byte-by-byte with conduit","archived":false,"fork":false,"pushed_at":"2014-07-31T07:52:18.000Z","size":144,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-30T19:17:52.517Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blitzcode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-07-05T18:43:58.000Z","updated_at":"2020-01-20T02:34:09.000Z","dependencies_parsed_at":"2022-08-25T12:11:18.044Z","dependency_job_id":null,"html_url":"https://github.com/blitzcode/conduit-chunked","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/blitzcode/conduit-chunked","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blitzcode%2Fconduit-chunked","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blitzcode%2Fconduit-chunked/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blitzcode%2Fconduit-chunked/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blitzcode%2Fconduit-chunked/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blitzcode","download_url":"https://codeload.github.com/blitzcode/conduit-chunked/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blitzcode%2Fconduit-chunked/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262094397,"owners_count":23257954,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-30T15:11:33.909Z","updated_at":"2025-06-26T15:31:37.394Z","avatar_url":"https://github.com/blitzcode.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# conduit-chunking\n\n### The Problem\n\nhttp://www.reddit.com/r/haskell/comments/29nvsx/how_to_get_good_performance_when_processing/\n\n### The Solution\n\nGetting decent performance requires verbose, unsafe, low-level code. Anything with my preferred await / yield semantics is really slow, even just the overhead from the monad stack is enough to make it unworkable. Using a high-level language and a library like conduit is kinda pointless if I have to do the work with `mallocByteString` and `pokeByteOff` and can't even be in the conduit monad to transparently yield the finished chunk. At that point, I'm just writing clumsy C code in Haskell. Will have to give this one another look in the future.\n\n#### Update ####\n\nhttps://www.fpcomplete.com/user/snoyberg/library-documentation/vectorbuilder\n\nThe `conduit-combinators` package has been update with a new conduit named `vectorBuilder`, addressing this very problem. The solution is nice and performance is close enough. Solved!\n\n### Benchmark\n\n```\nbenchmarking 1M file size/Speed-of-light\nmean: 21.33466 ms, lb 19.66938 ms, ub 23.58114 ms, ci 0.950\nstd dev: 3.271833 ms, lb 2.207320 ms, ub 4.711354 ms, ci 0.950\nvariance introduced by outliers: 48.339%\nvariance is moderately inflated by outliers\n\nbenchmarking 1M file size/Chunking with list\ncollecting 10 samples, 1 iterations each, in estimated 8.989661 s\nmean: 919.7806 ms, lb 917.1447 ms, ub 923.0766 ms, ci 0.950\nstd dev: 5.015212 ms, lb 3.335758 ms, ub 7.501738 ms, ci 0.950\nvariance introduced by outliers: 9.000%\nvariance is slightly inflated by outliers\n\nbenchmarking 1M file size/Chunking with blaze-builder\nmean: 339.6164 ms, lb 336.6435 ms, ub 344.2831 ms, ci 0.950\nstd dev: 6.200981 ms, lb 3.655564 ms, ub 9.273673 ms, ci 0.950\nfound 1 outliers among 10 samples (10.0%)\n  1 (10.0%) high mild\nvariance introduced by outliers: 9.000%\nvariance is slightly inflated by outliers\n\nbenchmarking 1M file size/Chunking with raw memory buffer\nmean: 24.95672 ms, lb 20.45549 ms, ub 34.69759 ms, ci 0.950\nstd dev: 10.78405 ms, lb 4.345717 ms, ub 17.91868 ms, ci 0.950\nfound 1 outliers among 10 samples (10.0%)\n  1 (10.0%) high severe\nvariance introduced by outliers: 89.322%\nvariance is severely inflated by outliers\n\nbenchmarking 1M file size/Chunking with chunk builder\ncollecting 10 samples, 1 iterations each, in estimated 26.95480 s\nmean: 2.724945 s, lb 2.718691 s, ub 2.730379 s, ci 0.950\nstd dev: 9.911362 ms, lb 6.669448 ms, ub 15.00682 ms, ci 0.950\nvariance introduced by outliers: 9.000%\nvariance is slightly inflated by outliers\n\nbenchmarking 1M file size/Chunking with state\ncollecting 10 samples, 1 iterations each, in estimated 17.52748 s\nmean: 1.716773 s, lb 1.711659 s, ub 1.722977 s, ci 0.950\nstd dev: 9.605795 ms, lb 7.260816 ms, ub 12.36112 ms, ci 0.950\nvariance introduced by outliers: 9.000%\nvariance is slightly inflated by outliers\n\nbenchmarking 1M file size/Chunking with vectorBuilder\nmean: 43.72639 ms, lb 42.32925 ms, ub 45.59146 ms, ci 0.950\nstd dev: 2.739349 ms, lb 1.923031 ms, ub 3.703952 ms, ci 0.950\nvariance introduced by outliers: 18.408%\nvariance is moderately inflated by outliers\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblitzcode%2Fconduit-chunked","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblitzcode%2Fconduit-chunked","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblitzcode%2Fconduit-chunked/lists"}