{"id":45186180,"url":"https://github.com/jbangdev/jdkdb-scraper","last_synced_at":"2026-04-30T01:10:22.360Z","repository":{"id":335559866,"uuid":"1146126061","full_name":"jbangdev/jdkdb-scraper","owner":"jbangdev","description":"Scraper for Java Metadata","archived":false,"fork":false,"pushed_at":"2026-02-20T12:47:37.000Z","size":677,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-20T12:55:46.918Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jbangdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"commonhaus","open_collective":"commonhaus-foundation"}},"created_at":"2026-01-30T16:52:39.000Z","updated_at":"2026-02-20T12:47:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jbangdev/jdkdb-scraper","commit_stats":null,"previous_names":["jbangdev/java-metadata-scraper","jbangdev/jdkdb-scraper"],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/jbangdev/jdkdb-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbangdev%2Fjdkdb-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbangdev%2Fjdkdb-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbangdev%2Fjdkdb-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbangdev%2Fjdkdb-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jbangdev","download_url":"https://codeload.github.com/jbangdev/jdkdb-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jbangdev%2Fjdkdb-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29720315,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-22T15:10:41.462Z","status":"ssl_error","status_checked_at":"2026-02-22T15:10:04.636Z","response_time":110,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-20T11:01:48.470Z","updated_at":"2026-04-30T01:10:22.350Z","avatar_url":"https://github.com/jbangdev.png","language":"Java","funding_links":["https://github.com/sponsors/commonhaus","https://opencollective.com/commonhaus-foundation"],"categories":[],"sub_categories":[],"readme":"# jdkdb-scraper - JDK Metadata DB Scraper\n\nA Java-based application for scraping JDK metadata from various distros.\n\nThis application is used daily to update the contents of the [jdkdb-data](https://github.com/jbangdev/jdkdb-data) repository which is then used to generate indices in the [jdkdb-index](https://github.com/jbangdev/jdkdb-index) repository that finally powers the [OpenAPI](https://jbangdev.github.io/jdkdb-index/) page.\n\nThis project is based on [Joschi's Java Metadata project](https://github.com/joschi/java-metadata) and incorporates ideas from the [Foojay's Disco API project](https://github.com/foojayio/discoapi).\n\n## Features\n\n- **Parallel Execution**: Run multiple distro scrapers concurrently for improved performance\n- **Selective Scraping**: Run all scrapers or select specific distros\n- **Extensible Architecture**: Easy to add new distro scrapers\n- **Archive Extraction**: Automatically extracts release information from JDK archives\n\n## Running using JBang\n\nRunning the released version of the scraper application can simply be done with [JBang](jbang.dev):\n\n```bash\n# Update: Run all scrapers\njbang scraper@jbangdev/jdkdb-scraper update\n\n# Update: List available scrapers\njbang scraper@jbangdev/jdkdb-scraper update --list\n\n# Update: Run specific scrapers\njbang scraper@jbangdev/jdkdb-scraper update --scrapers microsoft,semeru,temurin\n\n# Index: Generate all.json files for all distros\njbang scraper@jbangdev/jdkdb-scraper index\n\n# Download: Download and compute missing checksums for all distros\njbang scraper@jbangdev/jdkdb-scraper download\n\n# Clean: Remove incomplete metadata files and prune old EA releases (dry-run)\njbang scraper@jbangdev/jdkdb-scraper clean --dry-run\n\n# Clean: Actually remove incomplete files and prune EA releases older than 6 months\njbang scraper@jbangdev/jdkdb-scraper clean --remove-incomplete=all --prune-ea=6m\n```\n\n## Usage\n\nThe application provides four main commands:\n\n- **`update`** - Scrape JDK metadata from various distros and update metadata files\n- **`index`** - Generate aggregated all.json files for distro directories\n- **`download`** - Download and compute checksums for metadata files with missing checksums\n- **`clean`** - Clean up metadata by removing incomplete files and pruning old EA releases\n\n### GitHub Authentication\n\nMany scrapers fetch data from GitHub APIs, which have rate limits. To avoid hitting these limits, the application supports GitHub authentication through:\n\n1. **Environment Variable**: Set `GITHUB_TOKEN` to your GitHub personal access token\n2. **GitHub CLI**: If you have the GitHub CLI (`gh`) installed and authenticated, the token will be automatically obtained from `gh auth token`\n\n```bash\n# Option 1: Using environment variable\nexport GITHUB_TOKEN=your_github_token_here\njava -jar build/libs/jdkdb-scraper-1.0.0-SNAPSHOT-standalone.jar update\n\n# Option 2: Using GitHub CLI (automatic if gh is installed and authenticated)\ngh auth login\njava -jar build/libs/jdkdb-scraper-1.0.0-SNAPSHOT-standalone.jar update\n```\n\nThe application checks for tokens in this order: environment variable first, then GitHub CLI.\n\n### Typical usage\n\n- You can simply run `update` in the root of the data repository (where the `metadata/` folder is located) and let it do its work. It will scrape all the distro sites, obtain the latest metadata, download the jdk distributions, calculate checksums and update all the indices. Nothing else to be done. But this can take some time.\n- You can split the work into two steps:\n\n1. You run `update --no-download` which will do the scraping and will make sure that we have all the latest distributions cataloged. It will write all the metadata but with _missing_ checksums (and release info).\n2. You then run the much heavier and time-consuming `download` process that will look for any metadata file that has missing checksums (or release info) and download the archive to do the calculations. You can limit this step to only process a maximum amount of downloads before stopping. And then you can repeat it after a certain wait time to avoid rate limits and such.\n\nThe `index` command can be run any time you think the indices might be out of date and it will generate them all from scratch. _NB:_ if you use the second of the two strategies mentioned above then you will have metadata files with missing checksums, in that case you will need to add the option `--allow-incomplete` or they won't show up in the indices!\n\nAnd finally the `clean` command can be used to get rid of any invalid or orphaned data. If you run it without any flags it will give an overview of everything that could be cleaned up without actually doing any of the cleaning yet.\n\n## Command Line Options\n\n### Main Command\n\n```bash\nUsage: jdkdb-scraper [-hV] [COMMAND]\nScrapes JDK metadata from various distros and generates index files\n-h, --help      Show this help message and exit.\n-V, --version   Print version information and exit.\nCommands:\nupdate    Scrape JDK metadata from various distros and update metadata files\nindex     Generate all.json files for distro directories by aggregating\n\t\t\tindividual metadata files\ndownload  Download and compute checksums for metadata files that have missing\n\t\t\tchecksum values\nclean     Clean up metadata by removing incomplete files and pruning old EA\n\t\t\treleases\n```\n\n### Update Command\n\n```bash\nUsage: jdkdb-scraper update [-hlV] [--from-start] [--no-download] [--no-index]\n\t\t\t\t\t\t\t[-c=\u003cchecksumDir\u003e] [-x=\u003cindexDir\u003e]\n\t\t\t\t\t\t\t[--exclude=\u003cexcludeFileTypes\u003e[,\u003cexcludeFileTypes\u003e...]]...\n\t\t\t\t\t\t\t[--include=\u003cincludeFileTypes\u003e[,\u003cincludeFileTypes\u003e...]]...\n\t\t\t\t\t\t\t[--limit-progress=\u003climitProgress\u003e]\n\t\t\t\t\t\t\t[--limit-total=\u003climitTotal\u003e]\n\t\t\t\t\t\t\t[-m=\u003cmetadataDir\u003e] [--max-failures=\u003cmaxFailures\u003e]\n\t\t\t\t\t\t\t[--skip-ea=\u003cskipEa\u003e] [-t=\u003cmaxThreads\u003e]\n\t\t\t\t\t\t\t[-s=\u003cscraperIds\u003e[,\u003cscraperIds\u003e...]]...\n\nScrape JDK metadata from various distros and update metadata files\n\nOptions:\n-c, --checksum-dir=\u003cchecksumDir\u003e\n\t\t\t\t\tDirectory to store checksum files (default: db/checksums)\n\t--exclude=\u003cexcludeFileTypes\u003e[,\u003cexcludeFileTypes\u003e...]\n\t\t\t\t\tExclude these file types (e.g., msi,exe). These types will\n\t\t\t\t\tnot be downloaded.\n\t--from-start   Ignore existing metadata files and scrape all items from\n\t\t\t\t\tthe start\n-h, --help         Show this help message and exit.\n\t--include=\u003cincludeFileTypes\u003e[,\u003cincludeFileTypes\u003e...]\n\t\t\t\t\tInclude only these file types (e.g., tar_gz,zip). If\n\t\t\t\t\tspecified, only these types will be downloaded.\n-l, --list         List all available scraper IDs and exit\n\t--limit-progress=\u003climitProgress\u003e\n\t\t\t\t\tMaximum number of metadata items to process per scraper\n\t\t\t\t\tbefore aborting (default: unlimited)\n\t--limit-total=\u003climitTotal\u003e\n\t\t\t\t\tMaximum total number of downloads to accept before\n\t\t\t\t\tstopping (default: unlimited)\n-m, --metadata-dir=\u003cmetadataDir\u003e\n\t\t\t\t\tDirectory to store metadata files (default: db/metadata)\n\t--max-failures=\u003cmaxFailures\u003e\n\t\t\t\t\tMaximum number of allowed failures per scraper before\n\t\t\t\t\taborting that scraper (default: 10)\n\t--no-download  Skip downloading files and only generate metadata (for\n\t\t\t\t\ttesting/dry-run)\n\t--no-index     Skip generating index files (for testing/dry-run)\n-s, --scrapers=\u003cscraperIds\u003e[,\u003cscraperIds\u003e...]\n\t\t\t\t\tComma-separated list of scraper IDs to run (if not\n\t\t\t\t\tspecified, all scrapers run)\n\t--skip-ea=\u003cskipEa\u003e\n\t\t\t\t\tSkip early access (EA) releases older than the specified\n\t\t\t\t\tduration (e.g., '6m' for 6 months, '1y' for 1 year)\n\t\t\t\t\t(default: 6m)\n-t, --threads=\u003cmaxThreads\u003e\n\t\t\t\t\tMaximum number of parallel scraper threads (default:\n\t\t\t\t\tnumber of processors)\n-V, --version      Print version information and exit.\n-x, --index-dir=\u003cindexDir\u003e\n\t\t\t\t\tDirectory to write generated index files to (default:\n\t\t\t\t\tdb/metadata)\n```\n\n### Index Command\n\n```bash\nUsage: jdkdb-scraper index [-hV] [--allow-incomplete] [-m=\u003cmetadataDir\u003e]\n\t\t\t\t\t\t[-x=\u003cindexDir\u003e] [-v=\u003cdistroNames\u003e[,\u003cdistroNames\u003e...]]...\n\nGenerate all.json files for distro directories by aggregating individual\nmetadata files\n\nOptions:\n\t--allow-incomplete\n\t\t\t\t\tAllow incomplete metadata files (missing checksums) to be\n\t\t\t\t\tincluded\n-h, --help         Show this help message and exit.\n-m, --metadata-dir=\u003cmetadataDir\u003e\n\t\t\t\t\tDirectory containing metadata files (default:\n\t\t\t\t\tdb/metadata)\n-v, --distros=\u003cdistroNames\u003e[,\u003cdistroNames\u003e...]\n\t\t\t\t\tComma-separated list of distro names to regenerate\n\t\t\t\t\tall.json for (if not specified, all distros are\n\t\t\t\t\tprocessed)\n-V, --version      Print version information and exit.\n-x, --index-dir=\u003cindexDir\u003e\n\t\t\t\t\tDirectory to write generated index files to (default:\n\t\t\t\t\tdb/metadata)\n```\n\n### Download Command\n\n```bash\nUsage: jdkdb-scraper download [-hV] [--randomize] [--stats-only]\n\t\t\t\t\t\t\t[-c=\u003cchecksumDir\u003e]\n\t\t\t\t\t\t\t[--exclude=\u003cexcludeFileTypes\u003e[,\u003cexcludeFileTypes\u003e...]]...\n\t\t\t\t\t\t\t[--include=\u003cincludeFileTypes\u003e[,\u003cincludeFileTypes\u003e...]]...\n\t\t\t\t\t\t\t[--limit-progress=\u003climitProgress\u003e]\n\t\t\t\t\t\t\t[--limit-total=\u003climitTotal\u003e]\n\t\t\t\t\t\t\t[-m=\u003cmetadataDir\u003e] [-t=\u003cmaxThreads\u003e]\n\t\t\t\t\t\t\t[-v=\u003cdistroNames\u003e[,\u003cdistroNames\u003e...]]...\n\nDownload and compute checksums for metadata files that have missing checksum\nvalues\n\nOptions:\n-c, --checksum-dir=\u003cchecksumDir\u003e\n\t\t\t\t\tDirectory to store checksum files (default: db/checksums)\n\t--exclude=\u003cexcludeFileTypes\u003e[,\u003cexcludeFileTypes\u003e...]\n\t\t\t\t\tExclude these file types (e.g., msi,exe). These types will\n\t\t\t\t\tnot be downloaded.\n-h, --help         Show this help message and exit.\n\t--include=\u003cincludeFileTypes\u003e[,\u003cincludeFileTypes\u003e...]\n\t\t\t\t\tInclude only these file types (e.g., tar_gz,zip). If\n\t\t\t\t\tspecified, only these types will be downloaded.\n\t--limit-progress=\u003climitProgress\u003e\n\t\t\t\t\tMaximum number of metadata items to process per scraper\n\t\t\t\t\tbefore aborting (default: unlimited)\n\t--limit-total=\u003climitTotal\u003e\n\t\t\t\t\tMaximum total number of downloads to accept before\n\t\t\t\t\tstopping (default: unlimited)\n-m, --metadata-dir=\u003cmetadataDir\u003e\n\t\t\t\t\tDirectory containing metadata files (default:\n\t\t\t\t\tdb/metadata)\n\t--randomize    Randomize the order of downloads instead of processing\n\t\t\t\t\tfiles in order\n\t--stats-only   Skip downloading files and only show statistics (for\n\t\t\t\t\ttesting/dry-run)\n-t, --threads=\u003cmaxThreads\u003e\n\t\t\t\t\tMaximum number of parallel download threads (default:\n\t\t\t\t\tnumber of processors)\n-v, --distros=\u003cdistroNames\u003e[,\u003cdistroNames\u003e...]\n\t\t\t\t\tComma-separated list of distro names to process (if not\n\t\t\t\t\tspecified, all distros are processed)\n-V, --version      Print version information and exit.\n```\n\n### Clean Command\n\n```bash\nUsage: jdkdb-scraper clean [-hV] [--dry-run] [--prune-checksums]\n\t\t\t\t\t\t[--remove-invalid] [-c=\u003cchecksumDir\u003e]\n\t\t\t\t\t\t[-m=\u003cmetadataDir\u003e] [--prune-ea=\u003cpruneEa\u003e]\n\t\t\t\t\t\t[--remove-incomplete=\u003cremoveIncomplete\u003e]\n\nClean up metadata by removing incomplete files and pruning old EA releases\n\nOptions:\n-c, --checksum-dir=\u003cchecksumDir\u003e\n\t\t\t\t\tDirectory containing checksum files (default:\n\t\t\t\t\tdb/checksums)\n\t--dry-run      Show statistics without actually deleting files\n-h, --help         Show this help message and exit.\n-m, --metadata-dir=\u003cmetadataDir\u003e\n\t\t\t\t\tDirectory containing metadata files (default:\n\t\t\t\t\tdb/metadata)\n\t--prune-checksums\n\t\t\t\t\tRemove orphaned checksum files that don't have a matching\n\t\t\t\t\tmetadata file\n\t--prune-ea=\u003cpruneEa\u003e\n\t\t\t\t\tPrune EA releases older than specified duration (e.g.,\n\t\t\t\t\t30d, 3w, 6m, 1y). Duration format: [number][d|w|m|y]\n\t--remove-incomplete=\u003cremoveIncomplete\u003e\n\t\t\t\t\tRemove metadata files with incomplete data. Options:\n\t\t\t\t\tchecksums (missing checksums), release-info (missing\n\t\t\t\t\trelease info), all (either missing checksums or release\n\t\t\t\t\tinfo) (default: all)\n\t--remove-invalid\n\t\t\t\t\tRemove metadata files that fail validation\n\t\t\t\t\t(MetadataUtils.isValidMetadata)\n-V, --version      Print version information and exit.\n```\n\n## Development\n\n### Requirements\n\n- Needs Java 21 or higher\n\n### Building\n\nThis project uses Gradle for dependency management and building.\n\n```bash\n# Build the project\n./gradlew spotlessApply build\n\n# This creates two jars:\n# - jdkdb-scraper-1.0.0-SNAPSHOT.jar (regular jar)\n# - jdkdb-scraper-1.0.0-SNAPSHOT-standalone.jar (fat jar with all dependencies)\n```\n\n### Running using the standalone JAR\n\n```bash\n# Show available commands\njava -jar build/libs/jdkdb-scraper-1.0.0-SNAPSHOT-standalone.jar --help\n\n# Update: Run all scrapers\njava -jar build/libs/jdkdb-scraper-1.0.0-SNAPSHOT-standalone.jar update\n```\n\n## Architecture\n\n### Core Components\n\n- **Main**: Entry point with Picocli command dispatcher\n- **UpdateCommand**: Scrapes JDK metadata from distros and updates files\n- **IndexCommand**: Aggregates individual metadata files into all.json files\n- **DownloadCommand**: Downloads JDK files to compute missing checksums\n- **CleanCommand**: Cleans up incomplete metadata and prunes old EA releases\n- **ProgressReporter**: Central reporting thread that receives and logs progress events from all scrapers\n- **ScraperConfig**: Configuration record encapsulating metadata directory, checksum directory, logger, etc\n- **BaseScraper**: Abstract base class for all scrapers with common functionality (downloading, hashing, metadata saving)\n- **GitHubReleaseScraper**: Specialized base class for scrapers that fetch releases from GitHub\n- **AdoptiumMarketplaceScraper**: Specialized base class for scrapers using Adoptium Marketplace API\n- **JavaNetBaseScraper**: Specialized base class for scrapers that fetch from java.net archives\n- **ScraperFactory**: Factory class that uses ServiceLoader to dynamically discover and instantiate scrapers\n- **Scraper.Discovery**: Service provider interface for scraper registration via Java ServiceLoader\n- **DownloadManager**: Interface for downloading JDK files (with default and no-op implementations)\n\n### Distro Scrapers\n\nThe project includes **35 distro scrapers**, supporting all major JDK distributions:\n\n#### Scraper IDs and Distros\n\n| Scraper ID | Distro | Notes |\n|------------|--------|-------|\n| `adoptopenjdk` | AdoptOpenJDK | Legacy |\n| `bisheng` | Bisheng | Huawei |\n| `corretto` | Amazon Corretto | |\n| `debian` | Debian | |\n| `dragonwell` | Alibaba Dragonwell | |\n| `gluon-graalvm` | Gluon GraalVM | |\n| `graalvm-ce` | GraalVM CE | Early Access |\n| `graalvm-community` | GraalVM Community | |\n| `graalvm-community-ea` | GraalVM Community | Early Access |\n| `graalvm-legacy` | GraalVM | Legacy versions |\n| `ibm` | IBM JDK | |\n| `java-se-ri` | Java SE RI | Reference Implementation |\n| `jetbrains` | JetBrains Runtime | |\n| `kona` | Tencent Kona | |\n| `liberica` | BellSoft Liberica | |\n| `liberica-native` | BellSoft Liberica Native | Native Image Kit |\n| `mandrel` | Mandrel | Red Hat's GraalVM |\n| `microsoft` | Microsoft | Microsoft Build of OpenJDK |\n| `openjdk` | OpenJDK | |\n| `openjdk-leyden` | OpenJDK Leyden | Project Leyden |\n| `openjdk-loom` | OpenJDK Loom | Project Loom |\n| `openjdk-valhalla` | OpenJDK Valhalla | Project Valhalla |\n| `openlogic` | OpenLogic | |\n| `oracle` | Oracle | |\n| `oracle-graalvm` | Oracle GraalVM | |\n| `oracle-graalvm-ea` | Oracle GraalVM | Early Access |\n| `redhat` | Red Hat | |\n| `sapmachine` | SAP SapMachine | |\n| `semeru` | IBM Semeru | |\n| `semeru-certified` | IBM Semeru Certified | |\n| `temurin` | Eclipse Temurin | Eclipse Adoptium |\n| `trava-8` | TravaOpenJDK | Java 8 |\n| `trava-11` | TravaOpenJDK | Java 11 |\n| `zulu` | Azul Zulu | |\n| `zulu-prime` | Azul Zulu Prime | |\n\n**Total: 35 scrapers**\n\nEach scraper is registered via Java's ServiceLoader mechanism in `META-INF/services`.\n\n### Adding New Scrapers\n\n1. Create a new class in `src/main/java/dev/jbang/jdkdb/scraper/impl/` extending `BaseScraper`, `GitHubReleaseScraper`, `AdoptiumMarketplaceScraper`, or `JavaNetBaseScraper`\n2. Implement required abstract methods\n3. Add an inner `Discovery` class implementing `Scraper.Discovery`\n4. Register the discovery class in `src/main/resources/META-INF/services/dev.jbang.jdkdb.scraper.Scraper$Discovery`\n\nExample:\n\n```java\npackage dev.jbang.jdkdb.scraper.impl;\n\npublic class NewDistro extends BaseScraper {\n\tpublic NewDistro(ScraperConfig config) {\n\t\tsuper(config);\n\t}\n\n\t@Override\n\tprotected List\u003cJdkMetadata\u003e scrape() throws Exception {\n\t\tList\u003cJdkMetadata\u003e allMetadata = new ArrayList\u003c\u003e();\n\n\t\ttry {\n\t\t\t// Figure out what items to scrape here...\n\t\t\tvar allItems = ...;\n\t\t\tfor (item in allItems) {\n\t\t\t\ttry {\n\t\t\t\t\tJdkMetadata metadata = scrapeItem(item);\n\t\t\t\t\tallMetadata.add(metadata);\n\t\t\t\t} catch (InterruptedProgressException|TooManyFailuresException e) {\n\t\t\t\t\t// We don't want these exceptions to mark the item as failed\n\t\t\t\t\tthrow e;\n\t\t\t\t} catch (Exception e) {\n\t\t\t\t\t// For recoverable problems we call fail():\n\t\t\t\t\tfail(filename, e);\n\t\t\t\t}\n\t\t\t}\n\t\t} catch (InterruptedProgressException ex) {\n\t\t\t// When we receive this exception we stop scraping\n\t\t\t// but we return whatever is already processed\n\t\t}\n\n\t\treturn allMetadata;\n\t}\n\n\tprivate JdkMetadata scrapeItem(...) throws Exception {\n\t\t// Here we scrape a single item\n\t\tJdkMetadata metadata = ...;\n\t\t// For each successful item created we call:\n\t\tsaveMetadataFile(metadata);\n\t\tsuccess(filename);\n\t\treturn metadata;\n\t}\n\n\t// ServiceLoader discovery\n\tpublic static class Discovery implements Scraper.Discovery {\n\t\t@Override\n\t\tpublic String name() {\n\t\t\treturn \"new-distro\";\n\t\t}\n\n\t\t@Override\n\t\tpublic String distro() {\n\t\t\treturn \"New Distro\";\n\t\t}\n\n\t\t@Override\n\t\tpublic Scraper create(ScraperConfig config) {\n\t\t\treturn new NewDistro(config);\n\t\t}\n\t}\n}\n```\n\n## Output\n\nThe scrapers and indexer generate structured output in the `metadata/` directory:\n\n### Metadata Files (`db/metadata`)\n\n1. **Top-level aggregated indexes**:\n- `all.json` - All JDK releases across all distros\n- `ga.json` - General Availability (stable) releases only\n- `ea.json` - Early Access releases only\n- `latest.json` - Latest releases per distro\n2. **Organized by release type** (`all/`, `ea/`, `ga/`):\n- OS-specific files: `linux.json`, `macosx.json`, `windows.json`, `aix.json`, `solaris.json`\n- Architecture-specific subdirectories with further breakdowns\n3. **Distro-specific metadata** (`\u003cdistro-name\u003e/`):\n- Individual `.json` files for each JDK release\n- `all.json` file combining all releases for that distro\n\n### Checksum Files (`db/checksums/`)\n\n- Stored in distro-specific directories: `db/checksums/\u003cdistro-name\u003e/`\n- Contains MD5, SHA1, SHA256, and SHA512 checksum files\n- Organized to match the corresponding metadata files\n\n## Logging\n\nLogs are written to:\n\n- Console (STDOUT) - Real-time progress\n- File (`logs/jdkdb-scraper.log`) - Detailed execution log\n\nThe logging configuration can be customized in `src/main/resources/logback.xml`.\n\n## Requirements\n\n- Java 21 or higher\n- Gradle 8.x (included via wrapper)\n\n## License\n\nSee [LICENSE](./LICENSE) file\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjbangdev%2Fjdkdb-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjbangdev%2Fjdkdb-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjbangdev%2Fjdkdb-scraper/lists"}