{"id":17160237,"url":"https://github.com/obfusk/fdroid-misc-scripts","last_synced_at":"2025-04-13T14:10:24.234Z","repository":{"id":145667068,"uuid":"576109022","full_name":"obfusk/fdroid-misc-scripts","owner":"obfusk","description":"fdroid-misc-scripts - miscellaneous scripts to analyse f-droid app data","archived":false,"fork":false,"pushed_at":"2023-10-06T02:10:03.000Z","size":3568,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-13T14:09:59.046Z","etag":null,"topics":["f-droid","reproducible-builds"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/obfusk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.AGPLv3","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-12-09T02:58:33.000Z","updated_at":"2024-03-30T23:43:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"c71c90a7-a4db-49c4-89a3-7972d641755b","html_url":"https://github.com/obfusk/fdroid-misc-scripts","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obfusk%2Ffdroid-misc-scripts","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obfusk%2Ffdroid-misc-scripts/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obfusk%2Ffdroid-misc-scripts/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/obfusk%2Ffdroid-misc-scripts/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/obfusk","download_url":"https://codeload.github.com/obfusk/fdroid-misc-scripts/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248724629,"owners_count":21151561,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["f-droid","reproducible-builds"],"created_at":"2024-10-14T22:24:09.899Z","updated_at":"2025-04-13T14:10:24.210Z","avatar_url":"https://github.com/obfusk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- SPDX-FileCopyrightText: 2023 FC Stegerman \u003cflx@obfusk.net\u003e --\u003e\n\u003c!-- SPDX-License-Identifier: AGPL-3.0-or-later --\u003e\n\n[![AGPLv3+](https://img.shields.io/badge/license-AGPLv3+-blue.svg)](https://www.gnu.org/licenses/agpl-3.0.html)\n\n# fdroid-misc-scripts\n\nmiscellaneous scripts to analyse f-droid app data\n\n![rb](graphs/rb.png)\n\n\u0026rarr; [Overview of apps published with Reproducible Builds](reproducible/overview.md)\n\n\u0026rarr; [Graphs of apps verified by the Verification Server](verification/graphs.md)\n\n## Setup\n\n```sh\n$ git clone https://github.com/obfusk/fdroid-misc-scripts.git\n$ cd fdroid-misc-scripts\n$ git clone https://gitlab.com/fdroid/fdroiddata.git\n$ git clone https://gitlab.com/fdroid/f-droid.org-transparency-log.git\n```\n\n### Dependencies\n\n`detect-blocks-fdroiddata.sh`, `detect-blocks.sh`, and `detect-signflinger.sh`\nrequire [`apksigtool`](https://github.com/obfusk/apksigtool),\n`download-index.sh` uses it when available; `create-graphs.py` requires\n`matplotlib` (e.g. `apt install python3-matplotlib`).\n\n## Scripts\n\n### Index \u0026 metadata (v1)\n\n#### download-index.sh\n\nDownloads F-Droid's `index-v1.jar` \u0026 extracts `index-v1.json` from it.\n\n```sh\n$ ./scripts/download-index.sh\n```\n\n#### update-index-and-metadata-apps.sh\n\nCreates/updates `apps/index-apps`, `apps/metadata-apps`, etc.\n\n```sh\n$ ./scripts/update-index-and-metadata-apps.sh\ngetting apps from index-v1.json...\nlisting apps from metadata...\ndiffing...\n$ ls -1 apps/\nindex-apps\nindex-apps-not-in-metadata\nmetadata-apps\nmetadata-apps-archived-and-disabled\nmetadata-apps-not-archived-or-disabled\nmetadata-apps-not-in-index\n```\n\n#### apps-status.py\n\nReads a list of appids from stdin and parses the metadata YAML for each app to\nshow its status: `disabled`, `archived`, `all builds disabled`, or `version=NAME\ncode=CODE` for the latest (non-disabled) build.\n\n```sh\n$ ./scripts/apps-status.py \u003c apps/metadata-apps-not-in-index\nsome.app.id                                                   version=4.2 code=42\nsome.other.app.id                                             all builds disabled\n[...]\n```\n\n#### detect-permissions.py\n\nLists apps in the index that use the specified permission(s).\n\n```sh\n$ ./scripts/detect-permissions.py REQUEST_INSTALL_PACKAGES\nsome.app.id: android.permission.REQUEST_INSTALL_PACKAGES\n[...]\n```\n\n#### update-stats.sh\n\nUpdate `stats/YYYY-MM-DD-apps`.\n\nNB: this doesn't *modify* `fdroiddata`, but it does check out the first commit\non the specified date (and then `master`).\n\n```sh\n$ ./scripts/update-stats.sh 2022-11-01\n$ ./scripts/update-stats.sh 2022-12-01\n```\n\n#### update-diffs.sh\n\nUpdate `stats/YYYY-MM-DD-{adds,rems}` \u0026\n`reproducible/YYYY-MM-DD-{bins,sigs}-{adds,rems}`.\n\n```sh\n$ ./scripts/update-diffs.sh\n```\n\n#### all-rb.sh\n\nCreate `reproducible/YYYY-MM-DD-all.rb` from\n`reproducible/YYYY-MM-DD-{bins,sigs}`.\n\n```sh\n$ ./scripts/all-rb.sh 2023-09-01\n```\n\n### Index \u0026 metadata (v2)\n\n#### download-v2.sh\n\nDownloads F-Droid's `entry.jar` (\u0026 extracts `entry.json` from it) and\n`index-v2.json` (\u0026 checks the signatures and checksums), for both `repo/` and\n`archive`, and saves them in `v2/`.\n\n```sh\n$ ./scripts/download-v2.sh\n```\n\n#### v2-apks.py\n\nChecks for inconsistencies between (version codes of) APKs in `fdroiddata` and\nthe `v2/` index.\n\n```sh\n$ ./scripts/v2-apks.py -v\n```\n\n### Reproducible Builds: Overview\n\n#### update-rb.sh\n\nCreates `reproducible/YYYY-MM-DD-{bins,sigs}`: an overview of the apps using\n`Binaries`/`signatures` on that date.\n\nNB: this doesn't *modify* `fdroiddata`, but it does check out the first commit\non the specified date (and then `master`).\n\n```sh\n$ ./scripts/update-rb.sh 2022-11-01\n$ ./scripts/update-rb.sh 2022-12-01\n```\n\n\u003cdetails\u003e\n\n```sh\n$ cd reproducible\n$ head 2022-12-01-bins\nandrodns.android.leetdreams.ch.androdns\nch.admin.bag.covidcertificate.verifier\nch.admin.bag.covidcertificate.wallet\ncom.dhaval.bookland\ncom.github.bmx666.appcachecleaner [signflinger]\ncom.markuspage.android.certtools [missing]\ncom.mishiranu.dashchan\ncom.rafapps.earthviewformuzei [signflinger]\ncom.zionhuang.music\nde.corona.tracing\n$ head 2022-12-01-sigs\nde.schildbach.wallet\nde.schildbach.wallet_test\ndev.obfusk.jiten\ndev.obfusk.jiten_webview\ndev.obfusk.sokobang\norg.schabi.newpipe [no longer RB]\norg.torproject.torservices\n```\n\n\u003c/details\u003e\n\n#### create-graphs.py\n\nCreate `graphs/{bins,sigs,rb}.png` graphs from the\n`reproducible/YYYY-MM-DD-{bins,sigs}` files and `graphs/adds.png` from the\n`stats/YYYY-MM-DD-{adds,rems}` \u0026\n`reproducible/YYYY-MM-DD-{bins,sigs}-{adds,rems}` files.\n\n```sh\n$ ./scripts/create-graphs.py\n```\n\n#### update-rb-signflinger.sh\n\nUpdates `reproducible/signflinger` using `detect-virtual-entry.sh`.\n\n```sh\n$ ./scripts/update-rb-signflinger.sh\n```\n\nNB: `reproducible/{disabled,missing,no-longer-rb}` are updated manually.\n\n### Reproducible Builds: Binaries\n\n#### download-binaries.sh\n\nDownloads APKs for apps using `Binaries:` into `binaries/`.\n\n```sh\n$ ./scripts/download-binaries.sh\n==\u003e fdroiddata/metadata/some.app.id.yml\nversion=4.2 code=42\n[...]\n\n==\u003e fdroiddata/metadata/some.other.app.id.yml\nall versions disabled\n[...]\n```\n\n#### compare-binaries.sh\n\nCompares upstream and F-Droid APKs in `binaries/` (when both are available).\n\n```sh\n$ cd binaries\n$ ../scripts/compare-binaries.sh cmp\nsome.app.id_42                                                          OK\nsome.other.app.id_37                                                    skipped\n[...]\n```\n\n#### detect-blocks.sh\n\nLists APKs in `binaries/` that contain blocks of other types than\n`APKSignatureSchemeBlock` or `VerityPaddingBlock` in their APK Signing Block.\n\n```sh\n$ cd binaries\n$ ../scripts/detect-blocks.sh\nsome.app.id_42_fdroid.apk: DependencyInfoBlock\n[...]\n```\n\n#### detect-signflinger.sh\n\nLists APKs in `binaries/` that are signed by Signflinger according to their\nmanifest, which is extracted using `apksigtool`.\n\n```sh\n$ cd binaries\n$ ../scripts/detect-signflinger.sh\nsome.app.id_42_fdroid.apk\nsome.app.id_42_upstream.apk\n[...]\n```\n\nNB: most -- but not all! -- of these APKs will start with a zipflinger virtual\nentry (see next script).\n\n#### detect-virtual-entry.sh\n\nLists APKs in `binaries/` whose first 28 bytes indicate they start with a\nzipflinger virtual entry.\n\n```sh\n$ cd binaries\n$ ../scripts/detect-virtual-entry.sh\nsome.app.id_42_fdroid.apk\nsome.app.id_42_upstream.apk\n[...]\n```\n\n### Reproducible Builds: Signatures in fdroiddata\n\n#### detect-blocks-fdroiddata.sh\n\nLists `fdroiddata/metadata/*/signatures/*/APKSigningBlock` that contain blocks\nof other types than `APKSignatureSchemeBlock` or `VerityPaddingBlock`.\n\n```sh\n$ ./scripts/detect-blocks-fdroiddata.sh\nfdroiddata/metadata/some.app.id/signatures/42/APKSigningBlock: DependencyInfoBlock\n[...]\n```\n\n### Reproducible Builds: Verification Server\n\n#### download-verified.py\n\nFIXME: work in progress.\n\n```sh\n$ ./scripts/download-verified.py\n```\n\n#### update-index-apks.sh\n\nFIXME: work in progress.\n\n```sh\n$ ./scripts/update-index-apks.sh 2023-01-\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobfusk%2Ffdroid-misc-scripts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fobfusk%2Ffdroid-misc-scripts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobfusk%2Ffdroid-misc-scripts/lists"}