{"id":19054628,"url":"https://github.com/datadog/btfhub-online","last_synced_at":"2025-04-17T23:32:22.873Z","repository":{"id":40578489,"uuid":"437351258","full_name":"DataDog/btfhub-online","owner":"DataDog","description":"BTFHub repo","archived":false,"fork":false,"pushed_at":"2023-04-08T10:03:09.000Z","size":135,"stargazers_count":16,"open_issues_count":0,"forks_count":1,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-04-14T03:36:58.543Z","etag":null,"topics":["ebpf","ebpf-co-re","golang"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataDog.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2021-12-11T17:51:18.000Z","updated_at":"2024-01-17T17:15:34.000Z","dependencies_parsed_at":"2022-08-09T23:40:39.156Z","dependency_job_id":null,"html_url":"https://github.com/DataDog/btfhub-online","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataDog%2Fbtfhub-online","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataDog%2Fbtfhub-online/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataDog%2Fbtfhub-online/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataDog%2Fbtfhub-online/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataDog","download_url":"https://codeload.github.com/DataDog/btfhub-online/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223769011,"owners_count":17199390,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ebpf","ebpf-co-re","golang"],"created_at":"2024-11-08T23:39:08.304Z","updated_at":"2024-11-08T23:39:08.951Z","avatar_url":"https://github.com/DataDog.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BTFHub online\nBTFHub online is the online version of the great [BTFHub project](https://github.com/aquasecurity/btfhub).\nThe purpose of the online version is to make it easier for developers to fetch their relevant BTF for their BPF binary.\n\nThe project will allow you to be \"forward compatible\" with new minor versions or patches of your kernel being released.\n\n# The problem\nIn the recent years eBPF became more and more common among developers and companies. Most of them have used BCC toolkit\nit was the common toolkit for eBPF. But BCC have major drawback, among them, we can find portability issue.\n\nTo overcome that issue (and some other issues) there is a movement to libbpf toolkit with the great addition of CORE\n(Compile Once Run Everywhere). To use libbpf+CORE, one must supply a file called BTF (BPF Type Format) which holds data\nthat can tell how to make a BPF code compatible in the current host's kernel (Where to find structs in the kernel, how \nthey are arranged, etc.). Latest kernel versions do have a default BTF file at `/sys/kernel/btf/vmlinux`, but older kernels\ndoes not have such file. The solution for them is to generate such BTF files to all versions of the old kernel.\nGladly, [BTFHub](https://github.com/aquasecurity/btfhub) have done most of the hard work and generated BTFs for old kernels\nand their updates. BTFHub even added a script to generate a minimized and customized BTFs for a given BPF (a BTF that contain\nonly relevant data to your BPF). So you can create a directory with thousands of BTFs with a few MBs.\n\nBut, unfortunately kernel can be updated, and your local directory of the BTFs might be outdated for the latest kernel update.\nHere BTFHub-Online comes in!\n\nBTFHub Online supplies an online server that is able to supply a BTF (customized or not) live!\nYou BPF program just need to send a request to the server, and it will immediately get the relevant BTF.\nUsing the online version you gain:\n* \"Forward compatability\" to new kernel patches\n* No need to generate the local directory of BTFs\n* Easy integration\n\n\n# Access\nThe server is open for free access at https://btfhub.seekret.io (checkout the [SDK list](#list-of-sdks) for easy integration)\n\nOf course, you can set up and host your server by your own (checkout the [deployment section](#deploy))\nCurrently we wrap deployment for:\n* k8s\n\n# Table of Content\n1. [Visual Diagram](#visual-diagram)\n2. [Mode or operation](#mode-of-operation)\n   1. [List of SDKs](#list-of-sdks)\n3. [Examples](#examples)\n4. [Docker build](#docker-build)\n5. [Deploy](#deploy)\n   1. [k8s](#k8s-with-kong-api-gateway-and-gcp-bucket-archive)\n6. [Contributing](#contributing)\n7. [TODO List](#todo-list)\n8. [Credits](#credits)\n\n# Visual Diagram\n![](docs/BTFHubOnline.jpg)\n\n# Mode of operation\nWhile developing your eBPF module, use our SDKs or the api documentation to dynamically pull the BTF that suites\nyour eBPF module.\n\n## List of SDKs\n1. [GO](https://github.com/seek-ret/btfhub-online-go)\n2. [OpenAPI 3 spec](./docs/openapi.yaml)\n\n## Examples\n### Run server locally\n```bash\nmake update-dependencies\nGOOGLE_APPLICATION_CREDENTIALS=\u003cservice account json\u003e go run cmd/btfserver/main.go -t ./tools -p 8080 -b \u003cbucket name\u003e\n```\n\n### List all BTFs in the server\n```bash\ncurl \"https://btfhub.seekret.io/api/v1/list\"\n```\n\n### Download a single BTF\n```bash\ncurl \"https://btfhub.seekret.io/api/v1/download?distribution=ubuntu\u0026distribution_version=20.04\u0026kernel_version=5.11.0-1022-gcp\u0026arch=x86_64\" -o btf.tar.gz\n```\n\n### Customize a single BTF\n```bash\ncurl -X POST \"https://btfhub.seekret.io/api/v1/customize?distribution=ubuntu\u0026distribution_version=20.04\u0026kernel_version=5.11.0-1022-gcp\u0026arch=x86_64\" -F bpf=@\u003cpath to bpf.core.o\u003e -o btf.tar.gz\n```\n\n### Getting metrics\n```bash\ncurl \"https://btfhub.seekret.io/monitoring/metrics\"\n```\n\n### Getting health check\n```bash\ncurl \"https://btfhub.seekret.io/monitoring/health\"\n```\n\n# Docker build\n\n```bash\ndocker build -f btfserver.Dockerfile -t \"\u003cimage\u003e\" .\n```\n\n# Deploy\n## K8S with kong api gateway and gcp bucket archive\n```bash\nhelm dependency build deployments/k8s\nhelm upgrade -i btfhub ./deployments/k8s --set arguments.bucket_name=btfhub           \\\n  --set-file arguments.gcp_bucket.service_account_content=\u003cservice account json file\u003e \\\n  --set-file ingress.tls.cert=\u003ctls certificate\u003e                                       \\\n  --set-file ingress.tls.key=\u003ctls key\u003e                                                \\\n  --set kong.proxy.loadBalancerIP=\u003cexternal IP\u003e                                       \\\n  --set ingress.host=\u003cbtfhub.example.com\u003e                                             \\\n  --set \"ingress.domain=\u003c*.company.com\u003e\" \n```\n\n# Contributing\n\nWe are welcome you to contribute in any manner there is.\nWe care a lot from the community, thus please read our [code of conduct](./CODE_OF_CONDUCT.md) before contributing.\nYou don't have to develop to contribute to the community, and you can do it in one of the following ways:\n\n1. Open issues (bugs, feature requests)\n2. Develop the code (read [contributing.md](./CONTRIBUTING.md))\n   1. Check the [TODO List](#todo-list) to find good issues to start with.\n3. Mention us (twitter, linkedin, blogs)\n\n# TODO List\n - [ ] Add more http metrics\n - [ ] Add python SDK\n - [ ] Add RUST SDK\n - [ ] Add ARM support\n - [ ] Add local directory archive support\n - [ ] Add AWS bucket support\n - [ ] Add cloud run / cloud function deployment in GCP\n - [ ] Add lambda deployment in AWS\n - [ ] Add updater job instead of GitHub Actions\n\n# Credits\n\nThanks to:\n\n* Aqua Security for creating [BTFHub](https://github.com/aquasecurity/btfhub).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatadog%2Fbtfhub-online","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatadog%2Fbtfhub-online","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatadog%2Fbtfhub-online/lists"}