{"id":15937380,"url":"https://github.com/colindean/homebrew-size-analysis","last_synced_at":"2025-07-17T17:39:53.903Z","repository":{"id":191220863,"uuid":"627451505","full_name":"colindean/homebrew-size-analysis","owner":"colindean","description":"Analyzing the size of Homebrew formulae bottles","archived":false,"fork":false,"pushed_at":"2025-02-03T16:28:07.000Z","size":28,"stargazers_count":0,"open_issues_count":7,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T22:23:19.882Z","etag":null,"topics":["data-science","hacktoberfest","homebrew"],"latest_commit_sha":null,"homepage":"","language":"Makefile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/colindean.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["colindean"],"ko_fi":"colindean","liberapay":"colindean","open_collective":"colindean","custom":["https://venmo.com/colindean"]}},"created_at":"2023-04-13T13:49:51.000Z","updated_at":"2024-12-23T18:34:20.000Z","dependencies_parsed_at":"2025-01-20T16:20:47.287Z","dependency_job_id":"be07dd05-dbeb-4f6c-841a-1209920cbb7f","html_url":"https://github.com/colindean/homebrew-size-analysis","commit_stats":null,"previous_names":["colindean/homebrew-size-analysis"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colindean%2Fhomebrew-size-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colindean%2Fhomebrew-size-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colindean%2Fhomebrew-size-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colindean%2Fhomebrew-size-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/colindean","download_url":"https://codeload.github.com/colindean/homebrew-size-analysis/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252532893,"owners_count":21763494,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","hacktoberfest","homebrew"],"created_at":"2024-10-07T05:02:39.671Z","updated_at":"2025-05-05T16:26:19.324Z","avatar_url":"https://github.com/colindean.png","language":"Makefile","funding_links":["https://github.com/sponsors/colindean","https://ko-fi.com/colindean","https://liberapay.com/colindean","https://opencollective.com/colindean","https://venmo.com/colindean"],"categories":[],"sub_categories":[],"readme":"# Homebrew Bottle Size Analysis\n\nAnalyzing the size of Homebrew formulae bottles\n\n## Intent\n\nThe [Homebrew formula JSON API](https://formulae.brew.sh/docs/api) does not provide package size information for bottles[^def_bottle].\nI aim to retrieve package sizes regularly in order to build a database of `(package, version, bottle_arch) -\u003e size` pairs for future analysis.\nThis analysis could capture:\n\n- Package growth over time\n- Total estimated archive size\n- Spikes in package size indicating significant changes warranting further inspection\n- Expired or broken package URL for rarely-updated formulae with rarely-downloaded bottles\n- Packages to target for size optimization, from individual relief to [humanity-scale savings](https://daniel.haxx.se/blog/2022/12/06/faster-base64-in-curl/).\n\nThis is **currently** mostly an experiment in using simple CLI tools like Make and curl\nto do some data engineering and science that has the above useful implications.\n\n### Current principles\n\n- KISS, to the level of probably dumb.\n- Use CLI tools as much as possible; keep code to a minimum.\n- [Anything that can be installed via Homebrew](https://formulae.brew.sh) is fair to use.\n- Prioritize concurrency using simple tools such as Make `-j`, xargs, parallel, fd, ripgrep, etc.\n- Rebuilding the database from scratch means losing data, so avoid that.\n\n## Usage\n\n```sh\nmake formula.json  # get the data file\nmake urls          # split it out\nmake sizes         # get the sizes\n```\n\n## Architecture\n\n```mermaid\nstateDiagram-v2\n   Formulajson : Homebrew API \\n formula.json\n   Urls : One URL file per URL\n   Database : Database (Unspecified Format)\n\n    [*] --\u003e Formulajson : retrieve latest database\n    Formulajson --\u003e Urls : extract triplets, write URL files\n    state fork_state \u003c\u003cfork\u003e\u003e\n\n    state join_state \u003c\u003cjoin\u003e\u003e\n    Urls --\u003e fork_state : list URL files\n    fork_state --\u003e HTTPRequest1 : retrieve package size\n    fork_state --\u003e HTTPRequest2 : through HEAD requests\n    fork_state --\u003e HTTPRequestN : to all URLs in files\n    HTTPRequest1 --\u003e Sizes1 : write size file\n    HTTPRequest2 --\u003e Sizes2 : write size file\n    HTTPRequestN --\u003e SizesN : write size file\n    Sizes1 --\u003e join_state\n    Sizes2 --\u003e join_state\n    SizesN --\u003e join_state\n    join_state --\u003e Database\n    Database --\u003e [*]\n\n    note left of fork_state\n      One size file per retrieved URL\n    end note\n```\n\n## Performance notes\n\nIt takes around 80 minutes to run for me two requests at a time in order not to\ntrigger some kind of speed limit at my ISP level [^not_ghcr].\n\nYou can check the counts of urls and size files by running something like this:\n\n```sh\nfd .url data | wc -l\nfd .size data | wc -l\n```\n\nIf the numbers are the same, you've got the data for the current `formula.json`.\n\n[^def_bottle]: A _bottle_ is a pre-packaged archive of a formula available in Homebrew.\n    See \u003chttps://docs.brew.sh/Bottles\u003e for more information.\n\n[^not_ghcr]: It's not ghcr.io rate-limiting me.\n    My gateway is working fine but my ISP drops the upstream connection.\n    It's probably some kind of DDOS protection at the DNS level.\n    See notes.txt for ways I might get around this since curl does\n    a DNS lookup every time it launches.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcolindean%2Fhomebrew-size-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcolindean%2Fhomebrew-size-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcolindean%2Fhomebrew-size-analysis/lists"}