{"id":18976604,"url":"https://github.com/waseigo/changelogrex","last_synced_at":"2026-05-18T15:08:44.106Z","repository":{"id":191379803,"uuid":"684526972","full_name":"waseigo/Changelogrex","owner":"waseigo","description":"Fetches and parses Linux kernel ChangeLogs from https://kernel.org/pub/linux/kernel/ into Elixir structs that can later be handled with Ecto.","archived":false,"fork":false,"pushed_at":"2024-02-04T08:56:35.000Z","size":1305,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-02T14:19:52.989Z","etag":null,"topics":["changelogs","elixir","linux-kernel"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/waseigo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-29T10:11:55.000Z","updated_at":"2023-11-19T11:48:56.000Z","dependencies_parsed_at":"2024-01-14T12:43:06.655Z","dependency_job_id":"8d7e5ad3-b934-4c81-bf95-b459fcf1143a","html_url":"https://github.com/waseigo/Changelogrex","commit_stats":null,"previous_names":["waseigo/changelogrex"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/waseigo/Changelogrex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waseigo%2FChangelogrex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waseigo%2FChangelogrex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waseigo%2FChangelogrex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waseigo%2FChangelogrex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/waseigo","download_url":"https://codeload.github.com/waseigo/Changelogrex/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/waseigo%2FChangelogrex/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271400259,"owners_count":24752830,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["changelogs","elixir","linux-kernel"],"created_at":"2024-11-08T15:25:32.351Z","updated_at":"2026-05-18T15:08:39.074Z","avatar_url":"https://github.com/waseigo.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Changelogrex\n\n🐧🌾🗒️🫅 (Linux kernel changelog rex)\n\nFetches and parses Linux kernel ChangeLogs from [https://kernel.org/pub/linux/kernel/](https://kernel.org/pub/linux/kernel/) into Elixir structs that can later be handled with Ecto.\n\n## Front-end stuff\n\nhttps://github.com/waseigo/Changelogrex/assets/76704620/6b9b0d4e-722a-40e0-ac52-98489325f87e\n\n## Back-end stuff\n\n### Batch fetch-and-process Linux kernel v6.x ChangeLogs into properly parsed commits\n\nFirst, check which ChangeLogs are available.\n\n```elixir\n{ :ok, available } = Changelogr.Fetcher.fetch_available(\"v6.x\")\n```\n\n`available` is a %FetchOp{} (_Fetch_ *Op*eration) struct that contains HREFs and dates of all ChangeLogs from the directory listing of [https://kernel.org/pub/linux/kernel/v6.x/](https://kernel.org/pub/linux/kernel/v6.x/).\n\n```elixir\niex\u003e IO.inspect available\n%Changelogr.FetchOp{\n  url: \"https://cdn.kernel.org/pub/linux/kernel/v6.x/\",\n  timestamp: ~U[2023-11-19 10:43:10Z],\n  status: 200,\n  body: \"\u003chtml\u003e\\r\\n\u003chead\u003e\u003ctitle\u003eIndex of /pub/linux/kernel/v6.x/\u003c/title\u003e\u003c/head\u003e ...\",\n  hrefs: %{\n    \"6.3.3\" =\u003e \"https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.3.3\",\n    \"6.5.6\" =\u003e \"https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.5.6\",\n    ...\n  },\n  dates: %{\n    \"6.3.3\" =\u003e ~N[2023-05-17 12:09:00],\n    \"6.5.6\" =\u003e ~N[2023-10-06 11:24:00],\n    ...\n  },\n  errors: nil\n}\n```\n\nTo be able to fetch their content, we convert `available` into a list of `%ChangeLog{}` structs.\n\n```elixir\n{ :ok, fetchable } = Changelogr.Fetcher.fetchop_to_changelogs(available)\n```\n\n`fetchable` is a list of `%ChangeLog{}` structs. The `:body` attribute of the struct is still `nil`.\n\n```elixir\niex\u003e IO.inspect fetchable\n[\n  %Changelogr.ChangeLog{\n    kernel_version: \"6.1.35\",\n    url: \"https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.35\",\n    date: ~N[2023-06-21 14:12:00],\n    timestamp: nil,\n    body: nil,\n    commits: nil\n  },\n  %Changelogr.ChangeLog{\n    kernel_version: \"6.1.40\",\n    url: \"https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.40\",\n    date: ~N[2023-07-23 12:00:00],\n    timestamp: nil,\n    body: nil,\n    commits: nil\n  },\n...\n]\n```\n\nNext, we'll fetch them in sequence.\n\n```elixir\nchangelogs = Enum.map(fetchable, \u0026Changelogr.Fetcher.fetch_changelog/1)\n```\n\nIf the fetch was successful, the `:body` of each `%ChangeLog{}` struct in the list now contains the ChangeLog text, but it is still unparsed (`:commit` is still `nil`).\n\nFor example:\n\n```elixir\niex\u003e hd changelogs\n{:ok,\n %Changelogr.ChangeLog{\n   kernel_version: \"6.1.40\",\n   url: \"https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.40\",\n   date: ~N[2023-07-23 12:00:00],\n   timestamp: ~U[2023-11-19 10:44:24Z],\n   body: \"commit 75389113731bb629fa5e971baf58e422414c8d23\\nAuthor: Greg Kroah-Hartman \u003cgregkh@linuxfoundation.org\u003e\\nDate:   Sun Jul 23 13:49:51 2023 +0200\\n\\n    Linux 6.1.40\\n    \\n    Link: https://lore.kernel.org/ ...\",\n   commits: nil\n }}\n```\n\nNext, we'll process each one of these `%ChangeLog{}` structs into `%Commit{}` structs.\n\n1. First, we filter the list to keep only those with status `:ok`.\n2. Then, we convert it into a list of `%ChangeLog{}` structs by discarding the status.\n3. Finally, we parse the `:body` of each `%ChangeLog{}` structs into a list of `%Commit{}` structs in parallel.\n4. Then, we select only those with status `:ok`, ...\n5. ...and we flatten the list.\n\n```elixir\ncommits =\n  changelogs\n  |\u003e Enum.filter(fn {status, _} -\u003e status == :ok end)\n  |\u003e Enum.map(fn {_, x} -\u003e x end)\n  |\u003e Task.async_stream(\u0026Changelogr.Parser.changelog_to_commits/1)\n  |\u003e Enum.to_list()\n  |\u003e Enum.filter(fn {status, _} -\u003e status == :ok end)\n  |\u003e Enum.map(fn {_, x} -\u003e x end)\n  |\u003e List.flatten()\n```\n\nEach item of the `commits` list is a `%Commit{}` struct in which `:body` and `:commit` are filled in, as well as anything else known from the ChangeLog that contains the commit, and timestamps of when it was fetched, and where it was fetched from.\n\n```elixir\niex\u003e hd commits\n%Changelogr.Commit{\n  kernel_version: \"6.1.35\",\n  changelog_url: \"https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.35\",\n  fetched_timestamp: ~U[2023-11-19 11:23:01Z],\n  changelog_timestamp: ~N[2023-06-21 14:12:00],\n  commit: \"commit e84a4e368abe42cf359fe237f0238820859d5044\\n\",\n  author: nil,\n  date: nil,\n  body: \"Author: Greg Kroah-Hartman \u003cgregkh@linuxfoundation.org\u003e\\nDate:   Wed Jun 21 16:01:03 2023 +0200\\n\\n    ...\",\n  reported_by: nil,\n  tested_by: nil,\n  message_id: nil,\n  noticed_by: nil,\n  suggested_by: nil,\n  fixes: nil,\n  reviewed_by: nil,\n  closes: nil,\n  cc: nil,\n  acked_by: nil,\n  signed_off_by: nil,\n  link: nil,\n  upstream_commit: nil\n}\n```\n\nTo fill in the rest of the attributes and to process the existing fields, we need to extract and process them. Processing is done in parallel.\n\n```elixir\nprocessed =\n commits\n |\u003e Task.async_stream(\u0026Changelogr.Parser.extract_all_fields/1)\n |\u003e Enum.to_list()\n |\u003e Enum.map(fn {:ok, x} -\u003e x end)\n```\n\nAnd so we end up with a list of parsed and processed commits. Here's what such a \"finished\" commit looks like:\n\n```elixir\niex\u003e hd processed\n%Changelogr.Commit{\n  kernel_version: \"6.1.35\",\n  changelog_url: \"https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.35\",\n  fetched_timestamp: ~U[2023-11-19 11:23:01Z],\n  changelog_timestamp: ~N[2023-06-21 14:12:00],\n  commit: \"e84a4e368abe42cf359fe237f0238820859d5044\",\n  author: \"Greg Kroah-Hartman \u003cgregkh@linuxfoundation.org\u003e\",\n  date: #DateTime\u003c2023-06-21 16:01:03+02:00 +02 Etc/UTC+2\u003e,\n  body: [\"Linux 6.1.35\"],\n  reported_by: nil,\n  tested_by: [\"Florian Fainelli \u003cflorian.fainelli@broadcom.com\u003e\",\n   \"Markus Reichelt \u003clkt+2023@mareichelt.com\u003e\",\n   \"Salvatore Bonaccorso \u003ccarnil@debian.org\u003e\",\n   \"Linux Kernel Functional Testing \u003clkft@linaro.org\u003e\",\n   \"Chris Paterson (CIP) \u003cchris.paterson2@renesas.com\u003e\",\n   \"Jon Hunter \u003cjonathanh@nvidia.com\u003e\", \"Ron Economos \u003cre@w6rz.net\u003e\",\n   \"Conor Dooley \u003cconor.dooley@microchip.com\u003e\",\n   \"Sudip Mukherjee \u003csudip.mukherjee@codethink.co.uk\u003e\",\n   \"Takeshi Ogasawara \u003ctakeshi.ogasawara@futuring-girl.com\u003e\",\n   \"Allen Pais \u003capais@linux.microsoft.com\u003e\",\n   \"Shuah Khan \u003cskhan@linuxfoundation.org\u003e\",\n   \"Guenter Roeck \u003clinux@roeck-us.net\u003e\"],\n  message_id: nil,\n  noticed_by: nil,\n  suggested_by: nil,\n  fixes: nil,\n  reviewed_by: nil,\n  closes: nil,\n  cc: nil,\n  acked_by: nil,\n  signed_off_by: [\"Greg Kroah-Hartman \u003cgregkh@linuxfoundation.org\u003e\"],\n  link: [\"https://lore.kernel.org/r/20230619102154.568541872@linuxfoundation.org\"],\n  upstream_commit: nil\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwaseigo%2Fchangelogrex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwaseigo%2Fchangelogrex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwaseigo%2Fchangelogrex/lists"}