{"id":16115516,"url":"https://github.com/byron/github-star-counter","last_synced_at":"2025-03-16T08:32:45.301Z","repository":{"id":62439515,"uuid":"201894223","full_name":"Byron/github-star-counter","owner":"Byron","description":"A tool to query direct (and indirect) stars of a GitHub user or organization","archived":false,"fork":false,"pushed_at":"2020-12-29T16:04:06.000Z","size":273,"stargazers_count":61,"open_issues_count":0,"forks_count":6,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-16T00:23:11.627Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Byron.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-12T08:59:17.000Z","updated_at":"2025-03-09T11:42:29.000Z","dependencies_parsed_at":"2022-11-01T21:52:58.332Z","dependency_job_id":null,"html_url":"https://github.com/Byron/github-star-counter","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Byron%2Fgithub-star-counter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Byron%2Fgithub-star-counter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Byron%2Fgithub-star-counter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Byron%2Fgithub-star-counter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Byron","download_url":"https://codeload.github.com/Byron/github-star-counter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243846982,"owners_count":20357295,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-09T20:19:03.100Z","updated_at":"2025-03-16T08:32:44.915Z","avatar_url":"https://github.com/Byron.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Rust](https://github.com/Byron/github-star-counter/workflows/Rust/badge.svg)\n[![Crates.io](https://img.shields.io/crates/v/github-star-counter.svg)](https://crates.io/crates/github-star-counter)\n\n\nThis program is made just for trying async-await code in the current ecosystem.\nIt features the following capabilities:\n\n * do https requests\n * do multiple requests at a time, one per page\n * use async closures\n\nThe code was done synchronously first, and then moved to async with a surprisingly small amount of\nchanges.\nIt was interesting to see how the [`async` constructs](https://github.com/Byron/github-star-counter/blob/e3746b9182a28a9e9a9e8dd55cdb660f6b1b97df/src/lib.rs#L90)\nallow to control parallelism precisely, to the point where I was able to design interdependent\nfutures to match the data dependency. That way, things run concurrently when they can run concurrently, \nwhich can be visualized neatly with a dependency graph.\n\nThe greatest difficulties were around getting https to work. Besides, it's clearly a learning process\nto understand the implications of futures better. Constructs with `async` tend to _look_ synchronous,\nbut show their teeth with closures and ownership. Everything is solvable, just own everything, yet I think\nmore borrowing will be enabled once `async` lands on _stable_.\n\nSomething I absolutely agree with is the [statements in the async book](https://rust-lang.github.io/async-book/01_getting_started/02_why_async.html)\nwhich indicate that not everything needs to be async. Personally, I would probably start `sync`, and\nwait for performance requirements to change before making the switch. However, threads I would avoid in _future_,\nunless it truly is the simpler solution.\n\nSomething I look forward to is to see fully-async libraries emerge, for example, to interact with `git`,\nwhich will probably perform better than existing libraries. _Using_ `async` libraries already is a breeze!\n\nWhen thinking about the parallelism of this simple application it already becomes evident that one would want to control the amount of in-flight futures. Just imagine\nthe adverse effects of making too may concurrent connections to the same host, or the limits of resources imposed by the operating system itself. One would want to \nhave executors who are aware of what kind of future they are running, and have them limit the amount of concurrently running ones.\n\nWith `async`, Rust can be even more so change the game!\n\n### Installation\n\n```bash\ncargo install github-star-counter\n```\n\n### Running and usage\n\n```bash\ncount-github-stars Byron\n```\n\n```bash\ncount-github-stars --help\n```\n\nA more complete example, showing how massive the speedups can be. However, please keep in mind that this can also be contention, e.g. there\nare simply too many concurrent requests which are much slower together than they would be individually.\n```\n2019-08-15 08:47:49,553 INFO  [github_star_counter] Total bytes received in body: 11.5 MB\n2019-08-15 08:47:49,553 INFO  [github_star_counter] Total time spent in network requests: 366.84s\n2019-08-15 08:47:49,553 INFO  [github_star_counter] Wallclock time for future processing: 22.62s\n2019-08-15 08:47:49,553 INFO  [github_star_counter] Speedup due to networking concurrency: 16.22x\nTotal: 214379\nTotal for seanmonstar: 3818\nTotal for orgs: 210561\n\nmozilla/pdf.js         ★  27611\nmozilla/DeepSpeech     ★  10899\nmozilla/BrowserQuest   ★  8249\nmozilla/send           ★  8165\nmozilla/togetherjs     ★  6393\nmozilla/nunjucks       ★  6207\ntokio-rs/tokio         ★  5598\nlinkerd/linkerd        ★  5042\nhyperium/hyper         ★  5031\nlinkerd/linkerd2       ★  4342\n➜\n```\n\n### Development\n\n```bash\ngit clone https://github.com/Byron/github-star-counter\ncd github-star-counter\n# Print all available targets \nmake\n```\n\nAll other interactions can be done via `cargo`.\n\n### Difficulties on the way...\n\nPlease note that at the time of writing, 2019-08-13, the ecosystem wasn't ready.\nSearch the code for `TODO` to learn about workarounds/issues still present.\n\n* `async || {}` _(without move)_ is not yet ready, and needs to be move. This comes with the additional limitation that references can't be passed as argument, everything it sees must be owned.\n* `reqwest` with await support is absolutely needed. The low-level hyper based client we are using right now will start failing once github gzips its payload. For now I pin a working hyper version, which hopefully keeps working with Tokio.\n* Pinning of git repositories is not as easy as I had hoped - I ended up creating my own forks which are set to the correct version. However, it should also work with the `foo = { git = \"https://github.com/foo/foo\", rev = \"hash\" }` syntax. Maybe my ignorance though.\n* I would be interested in something like `collect::Result\u003cVec\u003cValue\u003e, Error\u003e` for `Vec\u003cFuture\u003cOutput = Result\u003cValue, Error\u003e\u003e\u003e`. `join_all` won't abort on first error, but I think it should be possible to implement such functionality based on it.\n* Defining a closure with `let mut closure: impl FnMut(User, usize) -\u003e impl Future\u003cOutput = Value\u003e` doesn't seem to work. The closure return type must be a type parameter.\n\n### Changelog\n\nFor the parallelism diagrams, a data point prefixed with `*` signals that multiple data is handled at the same time.\n\n#### v1.1.0 - Support for 'tera' templates\n\nThanks to the generous contribution of @mre there now is support for rendering to custom tera\ntemplates. [Look here](https://endler.dev/about/) for an example.\n\n#### v1.0.6 - Assurance of correctness\n\nGithub can silently adjust the page size, e.g. one asks for 1000 items per page and generates queries accordingly, but it will respond only with 100.\nNow we check and abort with a suggested page size, if the given one was not correct. The current page size seems to be limited to 100.\n\n#### v1.0.5 - Better performance metrics\n\n#### v1.0.4 - Even better progress - less is more\n\nJust show the aggregated result\n\n#### v1.0.3 - Better progress messages\n\nEven though the header is parsed and received relatively quickly, the body is read afterwards which takes additional time.\nThis will now be logged as well.\n\n#### v1.0.2 - Even more parallel query of user's repositories\n\nParallelism looks like this:\n```\n user-info+----\u003eorgs-info+----\u003e*(user-of-orgs+----\u003e*repo-info-page)\n          |\n          |\n          +----\u003e*repo-info-page\n```\nNow it's as parallel as it can be, based on the data dependency. This is real nice actually!\n\n#### v1.0.1 - More parallel query of user's repositories\n\nParallelism looks like this:\n```\nuser-info+----\u003eorgs-info+-+--\u003e*(user-of-orgs+----\u003e*repo-info-page)\n         |                |                       ^\n         |          wait  |                       |\n         +----------------+-----------------------^\n```\nWe don't wait for fetching org user info, but still wait for orgs information before anything makes progress.\nFetching repo information for the main user waits longer than needed.\n\n#### v1.0.0 - Initial Release\n\nParallelism looks like this:\n```\nuser-info+----\u003eorgs-info+---\u003e*(user-of-orgs-and-main-user+----\u003e*repo-info-page)\n```\n\n### Reference\n\n[This gist](https://gist.github.com/yyx990803/7745157) got me interested in writing a Rust version of it.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbyron%2Fgithub-star-counter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbyron%2Fgithub-star-counter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbyron%2Fgithub-star-counter/lists"}