{"id":16931175,"url":"https://github.com/maxgio92/krawler","last_synced_at":"2025-03-22T11:31:20.924Z","repository":{"id":37006323,"uuid":"450841195","full_name":"maxgio92/krawler","owner":"maxgio92","description":"A crawler for kernel releases distributed by the major Linux distributions.","archived":false,"fork":false,"pushed_at":"2024-10-18T08:13:00.000Z","size":1413,"stargazers_count":13,"open_issues_count":7,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-18T11:04:15.601Z","etag":null,"topics":["crawler","kernel","linux","scraping"],"latest_commit_sha":null,"homepage":"https://krawler.dev","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maxgio92.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-22T14:31:40.000Z","updated_at":"2024-10-14T09:47:02.000Z","dependencies_parsed_at":"2024-03-16T12:26:57.393Z","dependency_job_id":"64ff9db5-b0b2-43db-b926-eac55a1a24d5","html_url":"https://github.com/maxgio92/krawler","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxgio92%2Fkrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxgio92%2Fkrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxgio92%2Fkrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxgio92%2Fkrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maxgio92","download_url":"https://codeload.github.com/maxgio92/krawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244951427,"owners_count":20537385,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","kernel","linux","scraping"],"created_at":"2024-10-13T20:43:22.810Z","updated_at":"2025-03-22T11:31:20.170Z","avatar_url":"https://github.com/maxgio92.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Krawler: a kernel releases crawler ![logo](images/madagascar_small.png)\n\n[![Latest](https://img.shields.io/github/v/release/maxgio92/krawler)](https://github.com/maxgio92/krawler/releases/latest)\n[![CI](https://github.com/maxgio92/krawler/actions/workflows/ci.yaml/badge.svg)](https://github.com/maxgio92/krawler/actions/workflows/ci.yaml)\n[![Release](https://github.com/maxgio92/krawler/actions/workflows/release.yaml/badge.svg)](https://github.com/maxgio92/krawler/actions/workflows/release.yaml)\n[![Database update](https://github.com/maxgio92/krawler/actions/workflows/publish.yaml/badge.svg)](https://github.com/maxgio92/krawler/actions/workflows/publish.yaml)\n\n![](images/debian.png) ![](images/ubuntu.png) ![](images/centos.png) ![](images/fedora.png) ![](images/oracle.png) ![](images/opensuse.png) ![](images/amazonlinux.png) ![](images/archlinux.png)\n\nA crawler for kernel releases distributed by the major Linux distributions.\n\nIt supports, Amazon Linux v1, Amazon Linux v2, Amazon Linux 2022, Centos, Debian, Ubuntu, Fedora, Oracle Linux, OpenSUSE Linux, Arch Linux.\n\nThe crawling data is continuously published and is available at [db.krawler.dev](https://db.krawler.dev).\n\n## Usage\n\n```\nkrawler [options] \u003ccommand\u003e\n```\n\n### Options\n- `-c, --config file`: (optional) the config file to customize the list of mirrors to scrape for kernel releases (by default it looks at *$HOME/.krawler.yaml*).\n- `-v, --verbosity level`: (optional) the verbosity level (*debug*, *info*, *warn*, *error*, *fatal*, *panic*). By (default *warning*).\n\n### Commands\n\n#### `list`|`ls`\n\nList available kernel releases with distributed headers, by Linux distribution.\nIt returns a list of `kernelRelease` objects. The output format can be specified by flag parameter.\n\n```\nkrawler [options] list|ls \u003cdistribution\u003e [-o \u003cformat\u003e] \n```\n\n#### Parameters\n\n`distribution`: (**required**) The Linux distribution for which the release has been pubished.\nAvailable distributions:\n\n- *amazonlinux*\n- *amazonlinux2*\n- *amazonlinux2022*\n- *amazonlinux2023*\n- *centos*\n- *debian*\n- *ubuntu*\n- *fedora*\n- *oracle*\n- *opensuse*\n- *archlinux*\n\n#### Options\n\n`-o, --output format`: (optional) the format of the output of the list of kernel releases (one of *text*, *json* or *yaml*). By default *yaml*.\n\n#### Output\n\nThe `list`|`ls` command prints on standard ouput a is a list of kernel release objects of type [`KernelRelease`](https://github.com/maxgio92/krawler/blob/main/pkg/kernelrelease/kernelrelease.go#L16).\n\nAn example of a `json` result entry:\n\n```\n{\n  \"full_version\": \"4.18.0\",\n  \"version\": 4,\n  \"patch_level\": 18,\n  \"sublevel\": 0,\n  \"extra_version\": \"331\",\n  \"full_extra_version\": \"-331.el8.aarch64\",\n  \"architecture\": \"aarch64\",\n  \"package_name\": \"kernel-devel\",\n  \"package_url\": \"https://mirrors.edge.kernel.org/centos/8-stream/BaseOS/aarch64/os/Packages/kernel-devel-4.18.0-331.el8.aarch64.rpm\",\n  \"compiler_version\": \"80500\"\n}\n```\n\n## Getting started\n\nLet's imagine you want to list the available CentOS kernel releases, scraping default mirrors. You do it by running:\n\n```\nkrawler ls centos\n```\n\n## Configuration\n\nA configuration lets you configure parameters for the crawling, like the mirrors to scrape.\n\nThe default configuration file path is `$HOME/.krawler.yaml`. You can specify a custom path with the `--config` option.\n\nWhen a configuration is not present, a default configurations for repositories are used (for example [this](https://github.com/maxgio92/krawler/blob/main/pkg/distro/centos/constants.go#L20) is the default for Centos).\n\nFor a detailed overview see the [**reference**](docs/reference/CONFIG.md).\n\nMoreover, sample configurations are available [here](./config/samples).\n\n## Roadmap\n\n- [ ] Provide GCC versions for all releases\n- [ ] Support new distributions\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxgio92%2Fkrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxgio92%2Fkrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxgio92%2Fkrawler/lists"}