{"id":13611447,"url":"https://github.com/cjrh/lifter","last_synced_at":"2025-03-15T14:30:34.429Z","repository":{"id":37016593,"uuid":"292973597","full_name":"cjrh/lifter","owner":"cjrh","description":"Download and sync new releases of single-file binaries from Github Releases and other sites","archived":false,"fork":false,"pushed_at":"2025-03-07T22:29:20.000Z","size":678,"stargazers_count":13,"open_issues_count":58,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-07T23:25:14.856Z","etag":null,"topics":["binaries","cli","package-manager","sync"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cjrh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["cjrh"]}},"created_at":"2020-09-05T00:41:17.000Z","updated_at":"2025-03-07T22:29:25.000Z","dependencies_parsed_at":"2023-02-18T00:02:19.488Z","dependency_job_id":"5239f3e4-a620-45aa-94d1-88880204b16b","html_url":"https://github.com/cjrh/lifter","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cjrh%2Flifter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cjrh%2Flifter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cjrh%2Flifter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cjrh%2Flifter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cjrh","download_url":"https://codeload.github.com/cjrh/lifter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243742552,"owners_count":20340665,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binaries","cli","package-manager","sync"],"created_at":"2024-08-01T19:01:55.467Z","updated_at":"2025-03-15T14:30:34.035Z","avatar_url":"https://github.com/cjrh.png","language":"Rust","funding_links":["https://github.com/sponsors/cjrh"],"categories":["Rust","cli"],"sub_categories":[],"readme":"\u003e [!IMPORTANT]\n\u003e This is an *alpha-quality hobby project*. I do use this\n\u003e tool myself, but I started this project mainly to learn rust. While I\n\u003e appreciate community input, I don't have much extra time to spend on this and\n\u003e I'll be unresponsive to issue reports. I will however happily merge PRs with\n\u003e improvements.\n\n# lifter\n\nWould you like to automatically download cool binaries like ripgrep,\nfzf, bat, as soon as a new version is posted on their Github Releases\npages? And would you like this to work for all websites where\nsuch binaries are posted, not only Github?\n\n*lifter* is a CLI tool for downloading single-file executables\nfrom sites like Github that make them available, *and only downloading\nnewer versions when available*.\n\nNearly all projects that make CLI tools, like say _ripgrep_,\nput those binary artifacts in Github releases; but then we\nhave to wait until someone packages those binaries\ninto various OS distro package managers so that we can\nget them via _apt_ or _yum_ or _chocolatey_.  No more\nwaiting! _lifter_ will download directly from the\nGithub Releases page, if there is a new version released.\n\n### Why the name?\n\n```\n$ dictomatic lifter\nlifter  noun    a thief who steals goods that are in a store\nlifter  noun    an athlete who lifts barbells\n```\n\nTake your pick.  (By the way, `dictomatic` is another one of my\nhobby projects, and `lifter` will download that binary for you\ntoo.)\n\n## Demo\n\nRequires the presence of a `lifter.config` file alongside the binary. You can\nuse the example one in this repo.\n\n```bash\n$ ls -lh | rg lifter\n.rwxrwxr-x  6.9M caleb  2 Apr 12:27  lifter\n.rw-rw-r--   14k caleb  2 Apr 14:41  lifter.config\n$ ./lifter -vv\nINFO - [thesauromatic.exe] Found a match on versions tag: Alpha, includes bumpversion\nINFO - [thesauromatic.exe] Found version is not newer: Alpha, includes bumpversion; Skipping.\nINFO - [tokei] Found a match on versions tag: v12.1.2\nINFO - [tokei] Found version is not newer: v12.1.2; Skipping.\nINFO - [ncspot] Found a match on versions tag: v0.7.3\nINFO - [ncspot] Found version is not newer: v0.7.3; Skipping.\nINFO - [starship.exe] Found a match on versions tag: v0.55.0\nINFO - [starship.exe] Found version is not newer: v0.55.0; Skipping.\nINFO - [caddy] Found a match on versions tag: v2.4.3\nINFO - [caddy] Found version is not newer: v2.4.3; Skipping.\nINFO - [gitea] Found a match on versions tag: v1.14.3\nINFO - [gitea] Found version is not newer: v1.14.3; Skipping.\nINFO - [ripgrep] Found a match on versions tag: 13.0.0\nINFO - [ripgrep] Found version is not newer: 13.0.0; Skipping.\nINFO - [sd] Found a match on versions tag: v0.7.6\nINFO - [sd] Found version is not newer: v0.7.6; Skipping.\nINFO - [fzf] Found a match on versions tag: 0.27.2\nINFO - [fzf] Found version is not newer: 0.27.2; Skipping.\nINFO - [bat] Found a match on versions tag: v0.18.1\nINFO - [bat] Found version is not newer: v0.18.1; Skipping.\nINFO - [fcp] Found a match on versions tag: v0.1.0\nINFO - [fcp] Found version is not newer: v0.1.0; Skipping.\nINFO - [ripgrep Windows] Found a match on versions tag: 13.0.0\nINFO - [ripgrep Windows] Found version is not newer: 13.0.0; Skipping.\nINFO - [dictomatic] Found a match on versions tag: First release\nINFO - [dictomatic] Found version is not newer: First release; Skipping.\n...\n$ ls -l | rg rg\n.rwxr-xr-x  5.5M caleb  8 Feb  0:26  rg\n.rwxrwxr-x  5.1M caleb  8 Feb  0:26  rg.exe\n...\n```\n\nUnlike most package managers like *apt*, *scoop*, *brew*, *chocolatey*\nand many others that focus on a single operating system, *lifter* can\ndownload binaries for multiple operating\nsystems and simply place those in a directory. I regularly work on\ncomputers with different operating systems and I like my tools to travel\nwith me. By merely copying (or syncing) my \"binaries\" directory, I have\neverything available regardless of whether I'm on Linux or Windows.\n\nThis design only works because these applications can be deployed as\n*single-file exectuables*. For more complex applications, a heavier\nOS-specific package manager will be required.\n\n## Usage\n\nYou just run the `lifter` binary, and it'll download the binaries.\n\nThere is a sample configuration file, `lifter.config` that lets you specify which\napplications you want, and from where. *lifter* will keep track of the most recent\nversion, so it is cheap to rerun if nothing's changed.\n\nThis repo contains an example `lifter.config` file that you can use as a\nstarting point. It already contains sections for many popular golang and\nrustlang single-file-executable programs, like\n[ripgrep](https://github.com/BurntSushi/ripgrep),\n[fzf](https://github.com/junegunn/fzf),\n[starship](https://github.com/starship/starship),\nand many others.\n\n*lifter* works with other sites besides Github. The sample `lifter.config`\nincludes a definition for downloading the amazing _redbean_ binary\nfrom @jart's site `https://justine.lol/redbean/`. You should check\nout that project, it's wild.\n\n### Automation\n\nYou can automate `lifter` using cron. Run `$ crontab -e` and then add:\n\n```\nSHELL=/bin/bash\n24 22 * * * /path/to/lifter -w /path/for/downloads/\n```\n\n## Details\n\nI said that *lifter* is for fetching CLI binaries. That's what I'm *using* it\nfor, but it's more than that. It's an engine for downloading things from\nweb pages. It works like a web scraper.  There is a declarative mechanism\nfor specifying how to find the download item on a page. You do have to\ndo a bit of work to figure out the right CSS to target the download\nlink correctly.\n\n*NOTE: this section is out of date because of the switch from page\nscraping to calling the Github API*\n\nLet's look at the ripgrep configuration entry:\n\n```ini\n[ripgrep]\npage_url = https://github.com/BurntSushi/ripgrep/releases/\nanchor_tag = html main div.release-entry div.release-main-section details a\nanchor_text = ripgrep-(\\d+\\.\\d+\\.\\d+)-x86_64-unknown-linux-musl.tar.gz\nversion_tag = div.release-header a\ntarget_filename_to_extract_from_archive = rg\nversion = 12.1.1\n\n[ripgrep Windows]\npage_url = https://github.com/BurntSushi/ripgrep/releases/\nanchor_tag = html main div.release-entry div.release-main-section details a\nanchor_text = ripgrep-(\\d+\\.\\d+\\.\\d+)-x86_64-pc-windows-msvc.zip\nversion_tag = div.release-header a\ntarget_filename_to_extract_from_archive = rg.exe\nversion = 12.1.1\n```\n\nEach section will download a file; one for Linux and one for Windows.\nThe `anchor_tag` is the CSS selector for finding a section that contains\nthe target download link.\n\nIf there are many tags matching the `anchor_tag`, all of them will be\nchecked to match the required `anchor_text`. This is how the Github\nReleases page works. In one \"release\" section, there can be many file\ndownloads available. For example, one for each target architecture.\nSo the `anchor_tag`, alone, is not enough to target the specific target\nfile.\n\nFor that, we have the `anchor_text`: a regular expression that\nwill try to match the text of a specific `\u003ca\u003e` tag for all the `\u003ca\u003e` tags\nin the items contained within the `anchor_tag` section. We're looking for\nthe specific `\u003ca\u003e` tag for the final download. In our *ripgrep* example,\nthe text regex contains placeholders for the version number (the `\\d` values\nfor integers as part of the filename). These are discarded because we find\nthe version number in a different way, further below.\n\nThat's all that is needed to download the target file.\n\nTwo more details require explanation: tracking the version number,\nand dealing with archives. For the version number, we have the `version_tag`,\nwhich is also a CSS selector to find a DOM element containing the version\nnumber to attach to the downloaded file. This version will also be **stored\nand updated** in `lifter.config`. It is plausible that you might have a\nsituation with a (non-Github) target page where the version number does\nnot exist in its own DOM element. This scenario is currently unsupported.\nI think I've come across it on a Sourceforge page, for example.\n\nFinally, archives. Not all Github Releases artifacts are archives, some are\njust the executables themselves. But in the ripgrep examples above, the Linux\ndownload is a `.tar.gz` file, while the Windows download is a `.zip`.\nBy default, *lifter* will search within the archive to find a file that\nmatches the *name* of that section. So if a section is called `[sd]` then\n*lifter* will search for a file called `sd` inside the `.tar.gz`\narchive for that item. Likewise, for the section called `[sd.exe]`,\nit'll look for `sd.exe` inside the zipfile for that section.\n\nTo override this, all you have to do is set the field\n`target_filename_to_extract_from_archive`. If this is present, *lifter* will\nuse that name, rather than the name of the section to find the target file.\narchive. For example, in our ripgrep examples, we called the\nsection name, say, `[ripgrep Windows]`, but the file that we intend\nto extract from the archive is called `rg.exe`. This is why we\nset the target filename for extraction, explicitly. For ripgrep,\nwe could remove the target filename setting if the section names were\nchanged to `[rg]` and `[rg.exe]`. In this case, the section names would\nbe the filenames lookup up in each respective archive.\n\nSometimes things aren't so neat and we'd prefer to rename whatever\nis inside the archive. Consider the config for `[fcp]`:\n\n```ini\n[fcp]\ntemplate = github_release_latest\nproject = Svetlitski/fcp\ntarget_filename_to_extract_from_archive = fcp-0.1.0-x86_64-unknown-linux-gnu\ndesired_filename = fcp\nanchor_text = fcp-(\\d+\\.\\d+\\.\\d+)-x86_64-unknown-linux-gnu.zip\nversion = v0.1.0\n```\n\nIn this case, the name of the target executable as it appears inside the\nrelease archive is `fcp-0.1.0-x86_64-unknown-linux-gnu`. We would\nprefer that it be called `fcp` after extraction. To force this,\nset the `desired_filename` field. The extracted executable will\nbe renamed to this after extraction.\n\n## Templates\n\nThe description given in the *Details* section above is accurate but\nlaborious. It turns out that the CSS targeting is common for all\nprojects on the same site, e.g., Github Releases pages. Thus, there\nis support for templates in the config file definition.\n\nIf you look at the example `lifter.config` file in this repo, what\nyou actually see for ripgrep is the following:\n\n```ini\n[template:github_release_latest]\npage_url = https://github.com/{project}/releases\nanchor_tag = html main details a\nversion_tag = div.release-header a\n\n[ripgrep]\ntemplate = github_release_latest\nproject = BurntSushi/ripgrep\nanchor_text = ripgrep-(\\d+\\.\\d+\\.\\d+)-x86_64-unknown-linux-musl.tar.gz\ntarget_filename_to_extract_from_archive = rg\nversion = 13.0.0\n\n[starship.exe]\ntemplate = github_release_latest\nproject = starship/starship\nanchor_text = starship-x86_64-pc-windows-msvc.zip\nversion = v0.55.0\n```\n\nWhat actually happens at runtime is that if a section, like `ripgrep`,\nassigns a `template`, all the fields from that template are copied\ninto that section's declarations. In the example above, `page_url`,\n`anchor_tag`, and `version_tag` will be copied into each of the\nsections for `[ripgrep]` and `[starship.exe]`.\n\nIf you look carefully, you'll see that the template value for\n`page_url` above contains the variable `{project}`. That will\nbe substituted for the value of `project` that is declared\ninside each of the sections. In the above example, `page_url`\nwill be expanded to\n\n```\npage_url = https://github.com/BurntSushi/ripgrep/releases\n```\n\nfor the `[ripgrep]` section, and expanded to\n\n```\npage_url = https://github.com/starship/starship/releases\n```\n\nfor the `[starship.exe]` project.\n\n## Github API\n\nGithub made a change to their _Releases_ pages that requires running\nJavaScript to get the page to fully render. This change was likely\nmade to break scrapers like lifter. I have a working branch that uses\nembedded Chrome to fully render pages (with JS) that works---but for\nnow I've implemented a method that uses the Github API to download\nbinaries, rather than scrape. I will monitor how smoothly this goes\nand if it becomes too tedious I'll switch back from the API to\nscraping with the embedded browser engine.\n\nUsing the API has both benefits and downsides. The only benefit for\nlifter is that there might be more stability in the API than in the\n_Releases_ HTML page structure. Scrapers usually suffer if websites\nare updated frequently, in incompatible ways. There are several\ndownsides to using the API:\n- There are more severe rate limits. This is particularly true for\nunauthenticated requests, and for a tool like lifter which makes\na bunch of requests as its normal operation, is unusable, which means...\n- You pretty much have to use authenticated requests, which means you\nwill need to provide a [Personal Access Token](https://github.com/settings/tokens)\n- Tokens expire, which means you will have to periodically make a new\none and update your cron to use that. They _should_ expire because\ntokens that never expire are a security risk. However, if a token\nwasn't necessary you also wouldn't have the security risk. Needing\na token to get around the rate limits now also means you need to\nmanage token lifetime.\n- Authentication means you can and will be tracked.\n\nBecause of these changes, the earlier description of how to configure\nlifter will no longer work. However, the configuration is nearly the\nsame, except for two differences.\n\nThe first difference is in the config file, `lifter.config`. The\ntemplate section near the top must be written like this:\n\n```inifile\n[template:github_api_latest]\nmethod = api_json\npage_url = https://api.github.com/repos/{project}/releases/latest\nversion_tag = $.tag_name\nanchor_tag = $.assets.*.browser_download_url\n```\n\nNote the change from `github_release_latest` to `github_api_latest`.\nThen, simply change the `template` value only. Here's the example\nfor ripgrep:\n\n```inifile\n[ripgrep]\ntemplate = github_api_latest\nproject = BurntSushi/ripgrep\nanchor_text = ripgrep-(\\d+\\.\\d+\\.\\d+)-x86_64-unknown-linux-musl.tar.gz\ntarget_filename_to_extract_from_archive = rg\nversion = 13.0.0\n```\n\nIt is identical, except for the `template` value which now refers\nto the new one.\n\nThe second change is that you must provide a personal access token\nas an env var:\n\n```bash\n$ GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx lifter -vv\n```\n\nIt will run without specifying the token, but the rate limits come\nvery quickly, after only a handful of repos are checked.\n\n## Geek creds\n\nLifter can update itself. The config entry required to allow lifter to\nupdate itself looks like:\n\n```ini\n[lifter]\ntemplate = github_api_latest\nproject = cjrh/lifter\nanchor_text = lifter-(\\d+\\.\\d+\\.\\d+)-x86_64-unknown-linux-musl.tar.gz\nversion = 0.1.1\n\n[lifter.exe]\ntemplate = github_api_latest\nproject = cjrh/lifter\nanchor_text = lifter-(\\d+\\.\\d+\\.\\d+)-x86_64-pc-windows-msvc.zip\nversion = 0.1.1\n```\n\n## Other alternatives\n\n### Huber\n\nIf all you want is to download binaries from Github, then\n*Huber* is a probably a better choice than *lifter*.\n\n[Huber](https://github.com/innobead/huber) is a similar project that\nalso uses the Github API to download binaries. It has a lot more\nfeatures than *lifter*, and is more mature.\n\nInstead of a more general tool that is build around a \"scraping\"\nmindset, *Huber* focuses specifically on Github releases via the Github\nAPI. *Huber* makes it quite easy to list out various projects, and for\neach project to list the available versions, based on what is available\non the Github Releases page for that project.\n\n*Huber* has a \"managed\" list of projects that it can download. I have\nbeen thinking about adding a similar feature to *lifter*, but I haven't\ndone it yet. For now you have to manage your `lifter.config` yourself.\n\nGiven that *Huber* exists, I'm going to focus lifter on being a more\ngeneral tool that can download from any site, not just Github.\n\n### webinstall\n\nA pre-existing project doing something similar is\n[webinstall](https://github.com/webinstall/webi-installers). By comparison,\n*lifter*:\n- has fewer features\n- has fewer options\n- has fewer developers\n\n*webinstall* is however more complex than *lifter*. *lifter* needs only \nitself (binary) and the `lifter.config` file to work.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcjrh%2Flifter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcjrh%2Flifter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcjrh%2Flifter/lists"}