{"id":22762354,"url":"https://github.com/computermouth/kv-chunker","last_synced_at":"2025-08-17T01:06:29.113Z","repository":{"id":222873651,"uuid":"758600634","full_name":"computermouth/kv-chunker","owner":"computermouth","description":null,"archived":false,"fork":false,"pushed_at":"2024-02-21T03:08:23.000Z","size":11,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-19T05:08:54.063Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/computermouth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-16T17:05:26.000Z","updated_at":"2024-02-16T17:06:05.000Z","dependencies_parsed_at":"2024-12-11T10:07:42.787Z","dependency_job_id":"b48a910c-e3c1-4f0f-a0e9-bc4cb93aaa72","html_url":"https://github.com/computermouth/kv-chunker","commit_stats":null,"previous_names":["computermouth/kv-chunker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/computermouth/kv-chunker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/computermouth%2Fkv-chunker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/computermouth%2Fkv-chunker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/computermouth%2Fkv-chunker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/computermouth%2Fkv-chunker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/computermouth","download_url":"https://codeload.github.com/computermouth/kv-chunker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/computermouth%2Fkv-chunker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270792037,"owners_count":24645953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-11T10:07:37.279Z","updated_at":"2025-08-17T01:06:29.067Z","avatar_url":"https://github.com/computermouth.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kv-chunker\n\nA fastly compute app which stitches together bodies of chunked artifacts for bulk hosted files in KVStore.\n\n## how does it work\n\nA given file is chunked into 25MiB segments with the `uploader.sh` script, and is `PUT` in the KVStore along with a metadata file which just states how many chunks exist. Example:\n\n```\n$ ls -lsah myfile\n48M myfile\n$ stat --printf=\"%s\" myfile \n50000000\n$ ./uploader.sh myfile ${PASSWORD} ## PASSWORD must be set in src/pw.rs\n```\n\nWith this particular file, 3 files will be inserted in your fastly KVStore\n```\n└── mykvstore\n    ├── myfile_pcs\n    ├── myfile_0\n    └── myfile_1\n```\n\n`myfile_pcs` contains a printable text number of the chunks. In this case: 2.\n`myfile_0` is the first 25MiB(26214400B) of `myfile`\n`myfile_1` is the remaining ~23MiB(23785600B) of `myfile`\n\nLarger files will be `n` number of chunks, with the final chunk always being `${FILE_SIZE_IN_BYTES} % 26214400` bytes long.\n\nOn GETs, `myfile_pcs` is fetched, and it's contents parsed. The compute app will create N number of lookups, and append the bodies to the outbound response body.\n\n`TODO` -- use lookup_async, and gather a vector of handles, before waiting on all of them. (Currently a bug preventing this where PendingInsertHandle is not exported correctly).\n\n## getting started\n\nCreate a file `src/pw.rs`, and add the following\n\n```\npub const PASSWORD: \u0026str = \"mysupersecretpassword\";\n```\n\n`DISCLAIMER`\nThis password gets embedded in the application. As there's no rate limiting involved in the application, this is incredibly susceptible to a brute force attack. If used in any production scenario, at a minimum, please please please disable the `put()` logic once you're done with your uploads, or better yet, devise an actual authentication mechanism.\n\n## recommended usage\n\n0. (create fastly account, create a kv store)\n1. set `PASSWORD`, enable the `PUT` path in `main.rs`\n2a. deploy\n2b. (link your kv store with the compute service)\n3. use uploader `$ ./uploader.sh myfile ${PASSWORD}`\n4. test your GET -- `curl -XGET https://your-app-subdomain.edgecompute.app/myfile -O`\n5. disable `PUT` path\n6. deploy","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomputermouth%2Fkv-chunker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcomputermouth%2Fkv-chunker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcomputermouth%2Fkv-chunker/lists"}