{"id":18657935,"url":"https://github.com/lightsofapollo/downloadur","last_synced_at":"2025-07-10T13:36:17.734Z","repository":{"id":11761830,"uuid":"14295821","full_name":"lightsofapollo/downloadur","owner":"lightsofapollo","description":"Experimental download accelerator via http byte range fetching.","archived":false,"fork":false,"pushed_at":"2013-11-11T12:06:10.000Z","size":124,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2024-12-27T15:12:50.105Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lightsofapollo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-11-11T08:55:00.000Z","updated_at":"2013-11-11T12:06:11.000Z","dependencies_parsed_at":"2022-09-05T14:21:48.656Z","dependency_job_id":null,"html_url":"https://github.com/lightsofapollo/downloadur","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightsofapollo%2Fdownloadur","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightsofapollo%2Fdownloadur/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightsofapollo%2Fdownloadur/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightsofapollo%2Fdownloadur/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lightsofapollo","download_url":"https://codeload.github.com/lightsofapollo/downloadur/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239475960,"owners_count":19645041,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T07:30:40.254Z","updated_at":"2025-02-18T12:53:18.681Z","avatar_url":"https://github.com/lightsofapollo.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# downloadur\n\nExpriemental download accelerator via http byte range fetching.\n\n## Current Plan / Braindump\n\nHow its (probably) going to work:\n\n### State: detect\n  1. detect if range is available if it is -\u003e fast path || slow_path\n  2. detect if size is large enough (10mb?? no idea yet) if its big enough fast path || slow_path\n  \n### State: Slow Path\n  1. http.get -\u003e pipe to dest (one get)\n\n### State: Fast Path\n\n#### Terminology\n  * worker = single get request (readable stream)\n  * pool = collection of workers collectively working to donwload a single resource\n  * pool size = number of workers in the pool\n  * worker size = number of bytes a single worker is trying to get\n \n  \n#### Steps\n\n  1. start with pool of N workers with a very small (\u003c1mb?) worker size\n  2. measure latency of connection/time to data packets\n  3. measure bytes over time with current pool size\n  4. increase pool size in such a way to increase the number of parallel workers\n  5. utilize data of measurements to increase / decrease pool size / worker size\n\nTo make sure this works:\n\n  *  Identify a set of sufficently large enough files for remote testing (think order of 100mb files on S3)\n  *  TravisCI is the target of optimizations (for the moment)\n  *  measurements should take place very quickly during the initialization the download and stop therafter. \n     There maybe cases where network fluctuates significantly while we are download a massive file but the idea\n     is to optimize quickly (best effort) then stop measuring to keep overhead very low. If overhead is not an issue\n     we can optimize on the fly (maybe with a less expensive algorithm)\n  * expirement with both pool size AND worker size\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flightsofapollo%2Fdownloadur","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flightsofapollo%2Fdownloadur","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flightsofapollo%2Fdownloadur/lists"}