{"id":20289961,"url":"https://github.com/davidcoles/vc5","last_synced_at":"2025-10-04T06:29:21.861Z","repository":{"id":37496988,"uuid":"418163217","full_name":"davidcoles/vc5","owner":"davidcoles","description":"A horizontally scalable Direct Server Return layer 4 load balancer for Linux using XDP/eBPF","archived":false,"fork":false,"pushed_at":"2025-02-18T12:38:28.000Z","size":2013,"stargazers_count":85,"open_issues_count":0,"forks_count":8,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-31T16:14:03.689Z","etag":null,"topics":["bgp","dsr","ebpf","golang","l4lb","linux","load-balancer","networking","xdp"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidcoles.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-17T14:49:02.000Z","updated_at":"2025-03-25T01:07:06.000Z","dependencies_parsed_at":"2023-11-07T17:45:11.594Z","dependency_job_id":"48eb6564-3e65-400c-a324-9d9e3295e480","html_url":"https://github.com/davidcoles/vc5","commit_stats":null,"previous_names":[],"tags_count":44,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidcoles%2Fvc5","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidcoles%2Fvc5/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidcoles%2Fvc5/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidcoles%2Fvc5/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidcoles","download_url":"https://codeload.github.com/davidcoles/vc5/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247694876,"owners_count":20980733,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bgp","dsr","ebpf","golang","l4lb","linux","load-balancer","networking","xdp"],"created_at":"2024-11-14T15:05:41.531Z","updated_at":"2025-10-04T06:29:16.584Z","avatar_url":"https://github.com/davidcoles.png","language":"Go","readme":"# VC5\n\n\u003cpicture\u003e\n  \u003cimg src=\"doc/vc5.drawio.png\" width=\"25%\"\u003e\n\u003c/picture\u003e\n\nA horizontally scalable Direct Server Return\n([DSR](https://www.loadbalancer.org/blog/direct-server-return-is-simply-awesome-and-heres-why/))\nlayer 4 load balancer (L4LB) for Linux using XDP/eBPF.\n\nIf you think that this may be useful or have any\nquestions/suggestions, feel free to contact me at vc5lb@proton.me or\nraise a GitHub issue (or if you're going to [FOSDEM 2025](https://fosdem.org/2025/)).\n\n## Quickstart\n\nFor best results you should disable/uninstall irqbalance.\n\nYou will need to select a primary IP to pass to the balancer. This is\nused for the BGP router ID and, when only using a single subnet, as\nthe source address for healthcheck probe packets.\n\nA simple example on a server with a single, untagged ethernet interface:\n\n* `apt-get install git make libelf-dev golang-1.20 libyaml-perl libjson-perl ethtool` (or your distro's equivalent)\n* `ln -s /usr/lib/go-1.20/bin/go /usr/local/bin/go` (ensure that the Go binary is in your path)\n* `git clone https://github.com/davidcoles/vc5.git`\n* `cd vc5/cmd`\n* `cp config.sample.yaml config.yaml` (edit config.yaml to match your requirements)\n* `make` (pulls down the [libbpf](https://github.com/libbpf/libbpf) library, builds the binary and JSON config file)\n* `./vc5 -a 10.1.10.100 config.json eth0` (amend to use your server's IP address and ethernet interface)\n* A web console will be on your load balancer server's port 80 by default\n* Add your VIP to the loopback device on your backend servers (eg.: `ip addr add 192.168.101.1/32 dev lo`)\n* Configure your network/client to send traffic for your VIP to the load balancer, either via BGP (see config file) or static routing\n\nIt is almost certainly easier to use the binary from the latest Github\nrelease (compiled for x86-64). This will have been tested in\nproduction so should be reliable. Ensure that your configuration is\ncompatible with this version by using the config.pl script from the\ntagged release (or, of course, you can build your own JSON config\nhowever you prefer).\n\nIf you update the YAML config file and regenerate the JSON (`make\nconfig.json`) you can reload the new configuration by sending an a\nSIGINT (Ctrl-C) or SIGUSR2 to the process. SIGQUIT (Ctrl-\\\\) or SIGTERM\nwill cause the process to gracefully shut down BGP connections and\nexit.\n\nA more complex example with an LACP bonded ethernet device consisting\nof two (10Gbps Intel X520 on my test server) interfaces, with native\nXDP driver mode enabled and tagged VLANs:\n\n\n`config.yaml` vlans entry: \n\n```\nvlans:\n  10: 10.1.10.0/24\n  20: 10.1.20.0/24\n  30: 10.1.30.0/24\n  40: 10.1.40.0/24\n```\n\nCommand line:\n\n`./vc5 -n -a 10.1.10.100 config.json enp130s0f0 enp130s0f1`\n\nThe binary will detect your VLAN interfaces by looking for devices\nwith IP addreses which are contained in the VLAN mapping in the\nconfiguration file. If you use separate untagged physical interfaces\nthen this should now work transparently without any extra\nconfiguration, just list all of the interfaces on the command line so\nthat the eBPF code is loaded into each of them.\n\nBecause connection state is tracked on a per-core basis\n(BPF_MAP_TYPE_LRU_PERCPU_HASH), you should ensure that RSS ([Receive\nSide\nScaling](https://www.kernel.org/doc/Documentation/networking/scaling.txt))\nwill consistently route packets for a flow to the same CPU core in the\nevent of your switch slecting a different interface when the LACP\ntopology changes. Disable irqbalance, ensure that channel settings are\nthe same on each interface (ethtool -l/-L) and that RSS flow hash\nindirection matches (ethtool -x/-X).\n\nThe setup can be tested by starting a long running connection\n(eg. using iperf with the -t option) to a set of backend servers, then\n[disabling the chosen backend with an asterisk after the IP address in\nthe config file](doc/servers.md), determining which interface is\nreceiving the flow on the load balancer (eg., `watch -d 'cat\n/proc/interrupts | grep enp130s0f'` and look for the rapidly\nincreasing IRQ counter) and then dropping this interface out of LACP\n(`ifenslave -d bond0 enp130s0f0`). You should see the flow move to the\nother network interface but still hit the same core.\n\nWhen using backends in multiple subnets, for best performance you\nshould ensure that all VLANs are tagged on a single trunk interface\n(LACP bonded if you have more than one physical interface) with\nsubnet/VLAN ID mappings specified in the `vlans` section of the config\nfile.\n\nIf this is not possible (for example creating trunked interfaces on\nvSphere is not simple), then you can assign each subnet to a different\ninterface and use untagged mode (-u). This will use XDP's XDP_REDIRECT\nreturn code to send traffic out of the right interface, rather than\nupdating 802.1Q VLAN ID and using XDP_TX (vlans entry as before).\n\n`./vc5 -u -a 10.1.10.100 config.json eth0 eth1 eth2`\n\n\n## Goals/status\n\n* ✅ Simple deployment with a single binary\n* ✅ Stable backend selection with Maglev hashing algorithm\n* ✅ Route health injection handled automatically; no need to run other software such as ExaBGP\n* ✅ Minimally invasive; does not require any modification of iptables rules on balancer\n* ✅ No modification of backend servers beyond adding the VIP to a loopback device\n* ✅ Health checks are run against the VIP on backend servers, not their real addresses\n* ✅ HTTP/HTTPS, half-open SYN probe and UDP/TCP DNS health checks built in\n* ✅ In-kernel code execution with eBPF/XDP; native mode drivers avoid sk_buff allocation\n* ✅ Multiple VLAN support\n* ✅ Multiple NIC support for lower bandwidth/development applications\n* ✅ Works with bonded network devices to support high-availibility/high-bandwidth\n* ✅ Observability via a web console, Elasticsearch logging (in development) and Prometheus metrics\n\n## Performance\n\nThis has mostly been tested using Icecast backend servers with clients\npulling a mix of low and high bitrate streams (48kbps - 192kbps).\n\nIt seems that a VMWare guest (4 core, 8GB) using the XDP generic\ndriver will support 100K concurrent clients, 380Mbps/700Kpps through\nthe load balancer and 8Gbps of traffic from the backends directly to\nthe clients.\n\nOn a single (non-virtualised) Intel Xeon Gold 6314U CPU (2.30GHz 32\nphysical cores, with hyperthreading enabled for 64 logical cores) and\nan Intel 10G 4P X710-T4L-t ethernet card, I was able to run 700K\nstreams at 2Gbps/3.8Mpps ingress traffic and 46.5Gbps egress. The\nserver was more than 90% idle. Unfortunately I did not have the\nresources available to create more clients/servers.\n\n## About\n\nVC5 is a network load balancer designed to work as replacement for\nlegacy hardware appliances. It allows a service with a Virtual IP\naddress (VIP) to be distributed over a set of real servers. Real\nservers might run the service themselves or act as proxies for another\nlayer of servers (eg. HAProxy serving as a Layer 7 HTTP router/SSL\noffload). The only requirement being that the VIP needs to be\nconfigured on a loopback device on real server, eg.: `ip addr add\n192.168.101.1/32 dev lo`\n\nCurrently only layer 2 load balancing is performed. This means that\nthe load balancer instance needs to have an interface configured for\neach subnet where backend servers are present. This can be achieved\nwith seperate untagged physical NICs, or a trunked/tagged NIC or bond\ndevice with VLAN subinterfaces. For performance reasons, the tagged\nVLAN model is preferable.\n\nOne server with a 10Gbit/s network interface should be capable of\nsupporting an HTTP service in excess of 100Gbit/s egress bandwidth due\nto the asymmetric nature of most internet traffic. For smaller\nservices a modest virtual machine or two will likely handle a service\ngenerating a number of Gbit/s of egress traffic.\n\nIf one instance is not sufficient then more servers may be added to\nhorizontally scale capacity (and provide redundancy) using your\nrouter's ECMP feature. 802.3ad bonded interfaces and 802.1Q VLAN\ntrunking is supported (see [examples/](examples/) directory).\n\nNo kernel modules or complex setups are required, although for best\nperformance a network card driver with XDP native mode support is\nrequired (eg.: mlx4, mlx5, i40e, ixgbe, ixgbevf, nfp, bnxt, thunder,\ndpaa2, qede). A full list is availble at [The XDP Project's driver\nsupport page](https://github.com/xdp-project/xdp-project/blob/master/areas/drivers/README.org).\n\nA good summary of the concepts in use are discussed in [Patrick\nShuff's \"Building a Billion User Load Balancer\"\ntalk](https://www.youtube.com/watch?v=bxhYNfFeVF4\u0026t=1060s) and [Nitika\nShirokov's Katran talk](https://www.youtube.com/watch?v=da9Qw7v5qLM)\n\nA basic web console and Prometheus metrics server is included: ![Console screenshot](doc/console.jpg)\n\nExperimental elasticsearch support for logging (direct to your\ncluster, no need to scrape system logs) is now included. Every probe\nto backend servers is logged, so if one goes down you can see\nprecisely what error was returned, as well all sorts of other\nconditions. This will require a lot of refinement and more sensible\nnaming of log parameters, etc. (if you've got any insights please get\nin touch), but it should lead to being able to get some good insights\ninto what is going on with the system - my very inept first attempt\ncreating a Kibana dashboard as an example: ![Kibana screenshot](doc/kibana.jpg)\n\nA sample utility to render traffic from /20 prefixes going through the\nload-balancer is available at https://github.com/davidcoles/hilbert:\n![https://raw.githubusercontent.com/davidcoles/hilbert/master/hilbert.png](https://raw.githubusercontent.com/davidcoles/hilbert/master/hilbert.png)\n\nA good use for the traffic stats (/prefixes.json endpoint) would be to\ntrack which prefixes are usually active and to generate a table of\nwhich /20s to early drop traffic from in the case of a DoS/DDoS\n(particularly spoofed source addresses).\n\n## Changes from old version\n\nThe code for eBPF/XDP has been split out into the\n[xvs](https://github.com/davidcoles/xvs) repository - the object file\nis now committed to this repository and so does not need to be built\nas a seperate step.\n\nThe code for managing services, carrying out health checks and\nspeaking to BGP peers has been split out to the\n[cue](https://github.com/davidcoles/cue) repository, which allows it\nto be reused by other projects which use a different load balancing\nimplementation\n(eg., [LVS/IPVS](https://en.wikipedia.org/wiki/IP_Virtual_Server)).\n\n## Operation\n\nThere are three modes of operation, simple, VLAN, and multi-NIC\nbased. In simple mode all hosts must be on the same subnet as the\nprimary address of the load balancer. In VLAN mode (enabled by\ndeclaring entries under the \"vlans\" section of the YAML/JSON config\nfile), server entries must match a VLAN/CIDR subnet entry. VLAN tagged\ninterfaces need to be created in the OS and have an IP address\nassigned within the subnet. In multi-NIC mode subnets are given IDs in\nthe same manner as VLANs, but bpf_redirect() is used to send traffic\nout of the appropriately configured interface (rather than changing\nthe VLAN ID and using XDP_TX).\n\nIn VLAN mode, all traffic for the load balancer needs to be on a tagged VLAN (no\npushing or popping of 802.1Q is done - yet).\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidcoles%2Fvc5","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidcoles%2Fvc5","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidcoles%2Fvc5/lists"}