{"id":22004685,"url":"https://github.com/davidcoles/xvs","last_synced_at":"2025-05-05T17:24:29.368Z","repository":{"id":213135171,"uuid":"733075283","full_name":"davidcoles/xvs","owner":"davidcoles","description":"XDP Virtual Server","archived":false,"fork":false,"pushed_at":"2024-07-08T13:40:56.000Z","size":532,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-07-09T12:58:28.555Z","etag":null,"topics":["dsr","ebpf","golang","l4lb","libbpf","linux","load-balancing","networking","xdp"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidcoles.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-18T14:03:32.000Z","updated_at":"2024-07-08T13:40:59.000Z","dependencies_parsed_at":"2023-12-22T16:43:34.813Z","dependency_job_id":"6f3efe8a-30b4-4e66-a52a-0e41e85cbf4e","html_url":"https://github.com/davidcoles/xvs","commit_stats":null,"previous_names":["davidcoles/xvs"],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidcoles%2Fxvs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidcoles%2Fxvs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidcoles%2Fxvs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidcoles%2Fxvs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidcoles","download_url":"https://codeload.github.com/davidcoles/xvs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227239591,"owners_count":17752498,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dsr","ebpf","golang","l4lb","libbpf","linux","load-balancing","networking","xdp"],"created_at":"2024-11-30T00:16:46.560Z","updated_at":"2024-11-30T00:16:46.658Z","avatar_url":"https://github.com/davidcoles.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# XDP Virtual Server\n\nAn [XDP](https://en.wikipedia.org/wiki/Express_Data_Path)/[eBPF](https://en.wikipedia.org/wiki/EBPF)\nload balancer and Go API for Linux.\n\nThis code is originally from the\n[vc5](https://github.com/davidcoles/vc5) load balancer, and has been\nsplit out to be developed seperately.\n\nThis code implements a layer 4 Direct Server Return (DSR) load\nbalancer with an eBPF data plane that is loaded into the kernel, and a\nsupporting Go library to configure the balancer through the XDP\nAPI. Currently connections are steered at layer 2, so backend servers\nneed to share a VLAN with the load balancer. Multiple VLANs/interfaces\nare supported. Only IPv4 is supported for now, but support for layer 3\nDSR and IPv6 is planned.\n\nA compiled BPF ELF object file is committed to this repository (main\nbranch) and is accessed via Go's embed feature, which means that it\ncan be used as a standard Go module without having to build the binary\nas a seperate step. [libbpf](https://github.com/libbpf/libbpf) is\nstill required for linking programs using the library (CGO_CFLAGS and\nCGO_LDFLAGS environment variables may need to be used to specify the\nlocation of the library - see the Makefile for an example of how to do\nthis).\n\n# Portability\n\neBPF code is JITted to the native instruction set at runtime, so this\nshould run on any Linux architecture. Currently AMD64 and ARM\n(Raspberry Pi) are confirmed to work.\n\nDevices with constrained memory might have issues loading in the\ndefault size flow state tables. This can now be overriden with the\nMaxFlows parameter on newer kernels.\n\nRaspberry Pi Wi-Fi load balancer:\n\n`cmd/balancer wlan0 192.168.0.16 192.168.101.1 192.168.0.10 192.168.0.11`\n\n## Documentation\n\nSome notes about design are in the [doc/](doc/) directory, and the [Go\nAPI is described here](https://pkg.go.dev/github.com/davidcoles/xvs).\n\nThe API is loosely modelled on the [Cloudflare IPVS\nlibrary](https://github.com/cloudflare/ipvs) [(Go\nreference)](https://pkg.go.dev/github.com/cloudflare/ipvs).\n\n## Sample application\n\nA simple application in the `cmd/` directory will balance traffic\nto a VIP (TCP port 80 by default, can be changed with flags) to a\nnumber of backend servers on the same IP subnet.\n\nCompile/run with:\n \n* `make example`\n* `cmd/balancer ens192 10.1.2.3 192.168.101.1 10.1.2.10 10.1.2.11 10.1.2.12`\n\nReplace `ens192` with your ethernet interface name, `10.1.2.3` with\nthe address of the machine you are running the program on,\n`192.168.101.1` with the VIP you want to use and `10.1.2.10-12` with\nany number of real server addresses.\n\nOn a seperate client machine on the same subnet you should add a static route for the VIP, eg.:\n\n* `ip r add 192.168.101.1 via 10.1.2.3`\n\nYou should then be able to contact the service:\n\n* `curl http://192.168.101.1/`\n\nNo healthchecking is done, so you'll have to make sure that a\nwebserver is running on the real servers and that the VIP has been\nconfigured on the loopback address (`ip a add 192.168.101.1 dev lo`).\n\nA more complete example with health check and BGP route health\ninjection is currently available at\n[vc5](https://github.com/davidcoles/vc5).\n\n\n## Performance\n\nThis has mostly been tested using Icecast backend servers with clients\npulling a mix of low and high bitrate streams (48kbps - 192kbps).\n\nA VMWare guest (4 core, 8GB) using the XDP generic driver was able to\nsupport 100K concurrent clients, 380Mbps/700Kpps through the load\nbalancer and 8Gbps of traffic from the backends directly to the\nclients. Going above 700Kpps cause connections to be dropped,\nregardless of the number of cores or memory assigned to the VM, so I\nsuspect that there is a limit on the number of interrupts that the VM\nis able to handle per second.\n\nOn a single (non-virtualised) Intel Xeon Gold 6314U CPU (2.30GHz 32\nphysical cores, with hyperthreading enabled for 64 logical cores) and\nan Intel 10G 4P X710-T4L-t ethernet card, I was able to run 700K\nstreams with 2Gbps/3.8Mpps ingress traffic and 46.5Gbps egress. The\nserver was more than 90% idle. Unfortunately I did not have the\nresources available to create more clients/servers. I realised that I\ncarried this out when the server's profile was set to performance\nper-watt. Using the performance mode the CPU usage is barely 2% and\nlatencey is less than 250 nanoseconds.\n\nOn a Raspberry Pi (B+) ... don't get your hopes up!\n\n## Recalcitrant cards\n\nI'm currently investigating issues with the Intel X710 card. We have\nhad issues getting the NIC to bring up links (particularly after\nswitch reboot), though this may be due to SFP+ module/optics. I've\nbeen able to force a renegotiation with ethtool -r, but this then has\nthe effect of breaking XDP. This seems to be fixable by reattaching\nthe BPF section, so I have added a function to carry this out. The\ngeneric driver did not show this problem.\n\nThis has been extremely disappointing as the Intel X520 card worked\nperfectly, and pulling/reinserting cables on a bond behaved exactly as\nI would have hoped.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidcoles%2Fxvs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidcoles%2Fxvs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidcoles%2Fxvs/lists"}