{"id":13508646,"url":"https://github.com/outscale/butterfly","last_synced_at":"2025-04-05T14:31:26.489Z","repository":{"id":31523056,"uuid":"35087520","full_name":"outscale/butterfly","owner":"outscale","description":"Butterfly connects Virtual Machines and control their traffic flow","archived":false,"fork":false,"pushed_at":"2023-03-23T14:37:48.000Z","size":1496,"stargazers_count":61,"open_issues_count":29,"forks_count":19,"subscribers_count":10,"default_branch":"develop","last_synced_at":"2025-03-21T05:43:12.890Z","etag":null,"topics":["api","butterfly","dpdk","firewall","maturity-incubating","network","vhost","virtual-machine","virtual-switch","vm","vxlan"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/outscale.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-05-05T08:30:57.000Z","updated_at":"2025-03-21T05:04:12.000Z","dependencies_parsed_at":"2024-11-01T08:31:21.381Z","dependency_job_id":"c4d1af5d-a8ba-4852-b5f2-4a828cd596ef","html_url":"https://github.com/outscale/butterfly","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outscale%2Fbutterfly","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outscale%2Fbutterfly/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outscale%2Fbutterfly/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outscale%2Fbutterfly/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/outscale","download_url":"https://codeload.github.com/outscale/butterfly/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247352625,"owners_count":20925306,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","butterfly","dpdk","firewall","maturity-incubating","network","vhost","virtual-machine","virtual-switch","vm","vxlan"],"created_at":"2024-08-01T02:00:56.169Z","updated_at":"2025-04-05T14:31:23.639Z","avatar_url":"https://github.com/outscale.png","language":"C++","funding_links":[],"categories":["C++","api"],"sub_categories":[],"readme":"[![Project Incubating](https://docs.outscale.com/fr/userguide/_images/Project-Incubating-blue.svg)](https://docs.outscale.com/en/userguide/Open-Source-Projects.html)\n\nButterfly\n=========\nButterfly connects Virtual Machines (VM) and controls their traffic flow.\n\nEach VM traffic is contained in a specific [VXLAN](https://en.wikipedia.org/wiki/Virtual_Extensible_LAN)\nnetwork and traffic is filtered by (EC2/Openstack-like) security groups.\n\nSecurity groups can be applied to any VM interface, and contain a list of simple\nnetwork rules (dropping traffic by default).\n\n\n# Virtual NICs\n\nIn Butterfly, a Virtual NIC (or vnic) enables you to add a virtual network\ninterface to your Qemu VM through vhost-user.\nEach vnic has a 24 bit network id called VNI.\nIf two vnics with the same VNI are located on different physical hosts,\nButterfly will encapsulate VM packets over VXLAN and send them to the\ncorresponding physical host.\nOnce received, packets will be decapsulated and routed to their final\ndestination.\nAll vnics created with the same VNI are located on the same network.\nIf two vnics with the same VNI are on the same physical host, then packets do\nnot exit to physical network.\n\nButterfly is meant to be connected to the physical network using a dedicated\n[DPDK](http://dpdk.org/) port.\nIt allows Butterfly to have a very low latency between VMs while using physical\nNIC offload capabilities.\n\nFor VM-to-VM communication, checksum and segmentation do not occurs as packets\ndo not transit on a physical network. This enables Butterfly to have a high speed\nand low latency communication between VMs.\n\nExample: create of a new vnic \"vnic_1\" on vni \"1337\":\n```\nbutterfly nic add --ip 42.0.0.1 --mac 52:54:00:12:34:01 --vni 1337 --id vnic_1\n```\n\n# Filtering\n\nVMs traffic is filtered using an integrated firewall within Butterfly\n([NetBSD's NPF](http://www.netbsd.org/~rmind/npf/)) for each vnic.\nFiltering rules are applied to each VM depending on the rules contained in its _Security Groups_.\nA vnic can use several security groups and a security group can be used by several vnics.\nWhen a vnic use several security group, then rules are cumulated.\nA security group contains a list of rules to allow (default policy is to block) and a list\nof members (IP addresses).\n\nA Butterfly rule is mainly described by a protocol/port and source to allow.\nThis source can be either a CIDR block _or_ members of a security group.\n\nExample: Add a rule in the \"mysg\" security group that allows 42.0.3.1 in TCP protocol on port 22:\n```\nbutterfly sg rule add mysg --ip-proto tcp --port 22 --cidr 42.0.3.1/32\n```\n\nExample: Add a rule in the \"mysg\" security group that allows \"users\" security group members in TCP protocol on port 80:\n```\nbutterfly sg rule add mysg --ip-proto tcp --port 80 --sg-members users\n```\n\nNote: When a security group used by one or more vnics is modified, firewalling rules\nattached to each impacted VM are reloaded.\n\n# Using Butterfly\n\nButterfly is a daemon you can control over a network API.\n\nIt is packaged with a client mainly allowing you to add/remove/list vnics and security groups.\n\nYou can of course directly code your calls to Butterfly's API.\nAPI message transport is based on [ZeroMQ](http://zeromq.org/) and messages\nare encoded in [Protobuf](https://github.com/google/protobuf/ \"Google's protobuf\")\nformat. Check out [protocol](https://github.com/outscale/butterfly/tree/master/api/protocol)\nfor more details.\n\nHere is an example of Butterfly with 6 VMs isolated in three networks (VNI 42, 51 and 1337).\n\n![Butterfly execution](doc/butterfly.svg)\n\nButterfly binds a dedicated NIC to send/receive VXLAN packets and binds a socket\n(default: tcp) to listen to queries on its API. If you use a DPDK compatible\ncard, you will not be able to access the API through it.\n\nYou can build this configuration using a few lines of client calls:\n```\nbutterfly nic add --ip 42.0.0.1 --mac 52:54:00:12:34:01 --vni 42 --id vnic_1\nbutterfly nic add --ip 42.0.0.1 --mac 52:54:00:12:34:01 --vni 51 --id vnic_2\nbutterfly nic add --ip 42.0.0.2 --mac 52:54:00:12:34:02 --vni 51 --id vnic_3\nbutterfly nic add --ip 42.0.0.3 --mac 52:54:00:12:34:03 --vni 51 --id vnic_4\nbutterfly nic add --ip 42.0.0.1 --mac 52:54:00:12:34:01 --vni 1337 --id vnic_5\nbutterfly nic add --ip 42.0.0.2 --mac 52:54:00:12:34:02 --vni 1337 --id vnic_6\n```\n\nTip: if you want to see what the graph looks like: run `butterfly status` and copy past the dot diagram in [webgraphviz.com](http://www.webgraphviz.com/)\n\nYou can edit security groups whenever you want, which automatically updates vnics filtering.\nIn the following example, we create a new rule to allow everyone in http protocol and ask some vnics to use this security group.\n```\nbutterfly sg add sg-web\nbutterfly sg rule add sg-web --ip-proto tcp --port 80 --cidr 0.0.0.0/0\nbutterfly nic sg add vnic_1 sg-web\nbutterfly nic sg add vnic_2 sg-web\n```\nNote: Butterfly API uses idempotence, meaning that two calls should produce the\nsame result.\n\n# Installing Butterfly\n\nThe easiest way to install Butterfly is to download and install a package from [github releases](https://github.com/outscale/butterfly/releases).\nYou can also build Butterfly yourself (as shown in the next section).\n\n# Building Butterfly without docker\n\nThis building procedure has been tested on a fresh Centos7.\n\nFirst, install some dependencies (jemalloc needs manual installation:\n```\n$ sudo yum update -y\n$ sudo yum install -y gcc-c++ glibc-devel glib2-devel libtool libpcap-devel automake kernel-headers make git cmake kernel-devel unzip zlib-devel wget libstdc++-static numactl numactl-devel openssl-devel openssl-libs clang\n$ wget http://cbs.centos.org/kojifiles/packages/jemalloc/3.6.0/8.el7.centos/x86_64/jemalloc-devel-3.6.0-8.el7.centos.x86_64.rpm\n$ wget http://cbs.centos.org/kojifiles/packages/jemalloc/3.6.0/8.el7.centos/x86_64/jemalloc-3.6.0-8.el7.centos.x86_64.rpm\n$ sudo rpm -i jemalloc-devel-3.6.0-8.el7.centos.x86_64.rpm jemalloc-3.6.0-8.el7.centos.x86_64.rpm\n```\n\nBuild Butterfly:\n```\n$ git clone https://github.com/outscale/butterfly.git\n$ mkdir butterfly/build\n$ cd butterfly/build\n$ cmake ..\n$ make\n```\n\n# Build Butterfly with docker:\n\nYou can also use Docker in order to build Butterfly. It is based on a Centos7.\nFor developpment, we love to have different Linux distros in order to detect some issues. We could also build for several distro using docker in the future.\n\n# Preparing Your Machine\n\n## Configure Huge Pages\n\nButterfly needs some [huge pages](https://en.wikipedia.org/wiki/Page_%28computer_memory%29#Huge_pages)\n(adjust to your needs):\n\n- Edit your `/etc/sysctl.conf` and add some huge pages:\n```\nvm.nr_hugepages=2000\n```\n- Reload your sysctl configuration:\n```\n$ sudo sysctl -p /etc/sysctl.conf\n```\n- Check that your huge pages are available:\n```\n$ cat /proc/meminfo | grep Huge\n```\n- Mount your huge pages:\n```\n$ sudo mkdir -p /mnt/huge\n$ sudo mount -t hugetlbfs nodev /mnt/huge\n```\n- (optional) Add this mount in your `/etc/fstab`:\n```\nhugetlbfs       /mnt/huge  hugetlbfs       rw,mode=0777        0 0\n```\n\n## Prepare DPDK Compatible NIC\n\nBefore being able to bind your port, you need to enable Intel VT-d in your BIOS and have IOMMU explicitly enabled in your kernel parameters.\nCheck [DPDK compatible NICs](http://dpdk.org/doc/nics) and how to [bind NIC drivers](http://people.redhat.com/~pmatilai/dpdk-guide/setup/binding.html).\nPacketgraph also has an [example](https://github.com/outscale/packetgraph/tree/master/examples/firewall#configure-your-nics) on how to bind DPDK NICs.\n\nAdditionally, you may also want to isolate a specific core for Butterfly, check isolcpus [kernel parameters](https://www.kernel.org/doc/Documentation/kernel-parameters.txt).\n\n# Running Butterfly Server\n\nTo get help, see: `butterflyd --help`\n\nFor example, if you have a DPDK compatible NIC, Butterfly will use the first\navailable DPDK port. If no port is found, a (slow) tap interface is created.\n```\n$ sudo butterflyd -i 192.168.0.1 -s /tmp\n```\n\nIf you do not have a DPDK compatible card, you can also init a DPDK virtual\ndevice (which is _much_ slower than a DPDK compatible hardware).\n\nFor example, we can ask Butterfly to listen to the existing `eth0` interface:\n```\n$ sudo butterflyd -i 192.168.0.1 -s /tmp --dpdk-args \"-c1 -n1 --socket-mem 64 --vdev=eth_pcap0,iface=eth0\"\n```\n\nAlternatively, you can ask Butterfly to read a [configuration file](https://github.com/outscale/butterfly/blob/master/api/server/butterflyd.ini)\nat init:\n```\n$ sudo butterflyd -c /etc/butterfly/butterfly.conf\n```\n\n# F.A.Q.\n\n## Why Another Virtual Switch?\n\nBecause we just want a fast vswitch answering our simple needs:\n- _simple_ API: EC2/Openstack security groups style\n- Have some VXLAN for network isolation\n- Have some firewalling per Virtual Machine based on security groups\n- Use as little CPU as possible (and let Virtual Machines use all other cores)\n- Ease a (Cloud) orchestrator to control the whole thing through a simple API\n\n## What's Behind Butterfly?\n\nButterfly is based on:\n- [Packetgraph](http://github.com/outscale/packetgraph): creates network graph\n- [DPDK](http://dpdk.org/): fast access to NICs (in Packetgraph)\n- [NPF](http://www.netbsd.org/~rmind/npf/): firewalling (in Packetgraph)\n- [ZeroMQ](http://zeromq.org/): message transport\n- [Protobuf](https://github.com/google/protobuf/ \"Google's protobuf\"): message encoding/versioning\n\n## How Fast?\n\nBenchmarks setup:\n- Two physical machines directly connected\n- A third machine remote setup and launch benchmarks using `./benchmarks/run_benchmarks.sh`\n\nMachines:\n- OS: Centos 7 (3.10.0-327.18.2.el7.x86_64)\n- NICs: Intel 82599ES 10-Gigabit SFI/SFP+ (DPDK compatible used with vfio-pci driver)\n- CPU: AMD Opteron(tm) Processor 3350 HE\n\nResults (Juin 2019):\n### QPerf TCP latence Without TSO on VMs:\n```\ncmd: qperf -vvs  \u003cip address\u003e tcp_lat\n\n                | Without Firewall | With Firewall |\n ---------------+------------------+---------------+\n| Same Host     | 18.3 us          | 20.1 us       |\n| Diferent Host | 22.6 us          | 24.1 us       |\n```\n\n\n### QPerf UDP latence With TSO on VMs:\n```\ncmd: qperf -vvs  \u003cip address\u003e udp_lat\n                | Without Firewall | With Firewall |\n ---------------+------------------+---------------+\n| Same Host     | 16.8 us          | 18.5 us       |\n| Diferent Host | 21.7 us          | 22.1 us       |\n```\n\n### QPerf TCP latence With TSO on VMs:\n```\ncmd: qperf -vvs  \u003cip address\u003e tcp_lat\n                | Without Firewall | With Firewall |\n ---------------+------------------+---------------+\n| Same Host     | 18.4 us          | 20.1 us       |\n| Diferent Host | 22.9 us          | 24.1 us       |\n```\n\nResults (august 2017, 60 seconds per tests):\n\n### Without TSO on VMs\n```\n                         |   VMs on same host  |  VMs on remote host |\n ------------------------+---------------------+---------------------|\n| Ping (min/average/max) | 0.072/0.090/0.160ms | 0.106/0.162/0.236ms |\n| TCP (iperf 3)          |     6.00 Gbits/s    |     6.70 Gbits/s    |\n| UDP (iperf 3)          |     2.99 Gbits/s    |     1.41 Gbits/s    |\n```\n\n### With TSO enabled on VMs (--tso-on)\n```\n                         |   VMs on same host  |  VMs on remote host |\n ------------------------+---------------------+---------------------|\n| Ping (min/average/max) | 0.077/0.101/0.447ms | 0.059/0.96/0.203ms  |\n| TCP  (iperf 3)         |     15.9 Gbits/s    |     3.2 Gbits/s     |\n| UDP  (iperf 3)         |     2.99 Gbits/s    |     1.4 Gbits/s     |\n```\n\n### Notes\n- We get these results using iperf, so packets spend a lot of timer going inside VM's kernel, so these benchmarks are sadly not representative of the speed of Butterfly, we are working on new benchmarks\n- UDP is really _bad_ at the moment, we are working on it\n- We can get even faster with zero copy in vhost-user\n- We can get faster by embedding a more recent libc (`make package-fat`)\n- If you try to run some benchmarks, you may want to configure your [CPU throttling](https://en.wikipedia.org/wiki/Dynamic_frequency_scaling). On Centos7, check [cpufreq governors page](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Power_Management_Guide/cpufreq_governors.html)\n\n## How to Connect a Virtual Machine to Butterfly?\n\nButterfly does not launch your Virtual Machine for you, it just creates a special\nnetwork interface (vhost-user) so you can connect your Virtual Machine to it.\nVhost-user interfaces are Unix sockets allowing you to directly communicate with\nVirtual Machines in userland.\n\nWe tested Butterfly with QEMU \u003e= 2.5 and added the following parameters to\nthe machine's arguments (to adapt):\n\nSome shared memory between the guest and Butterfly:\n```\n-object memory-backend-file,id=mem,size=124M,mem-path=/mnt/huge,share=on -numa node,memdev=mem -mem-prealloc\n```\n\nFor each network interface:\n```\n-chardev socket,id=char0,path=/path/to/socket -netdev type=vhost-user,id=mynet0,chardev=char0,vhostforce -device virtio-net-pci,netdev=mynet0,gso=off\n```\n\n***Note about enic:***\n\nAlthough butterfly can run on Cisco enic and, depending of your hadware, enic pmd driver may not support checksum offloading or TSO for inner packets yet (needed for vxlan encapsulation).\nIf you want to use Butterfly with enic, you must desactivate TSO and TX checksum offloading on VM:\n```\nethtool -K ensX tx off\n```\n\nFor more details, check [vhost-user dpdk guide](http://dpdk.org/browse/dpdk/plain/doc/guides/prog_guide/vhost_lib.rst).\n\n## Do You Support Any Containers?\n\nNot yet, Butterfly only supports vhost-user network interfaces.\n\nAnyways, connecting a container should be possible too.\n\n## Does Butterfly Support IPv6?\n\n- Virtual Machine traffic can be in IPv6.\n- Outer network (VXLAN's side) is made in IPv4 for the moment.\n\n## What If My Virtual Machine Crashes/Reboots/Stops?\n\nVhost-user interface will still exist until it is removed from Butterfly.\n\nYou can just restart your VM, it will be reconnected to Butterfly and run as\nbefore.\n\nThis is possible because Butterfly acts as \"server\" in vhost-user communication.\n\n## What If Butterfly Crashes?\n\nToo bad, you can just restart Butterfly but the VM won't reconnect, as vhost-user\n\"server\" is located on Butterfly's side.\n\nFiling an issue is very valuable to the project. Please provide the following information:\n- The operating system with its version\n- The Butterfly version (butterfly --version)\n- The Butterfly logs (check syslogs)\n- Is your system under memory pressure?\n- What was Butterfly doing? (Heavy traffic? Doing nothing? How many VMs?)\n- The estimated Butterfly uptime until crash\n- Do you have a way to reproduce it?\n\nIt may be soon possible to choose which one is the vhost-user server between QEMU and\nButterfly, [comming soon in DPDK](http://dpdk.org/ml/archives/dev/2016-May/038627.html) :)\n\n## What Is Butterfly’s License?\n\nButterfly is licensed under [GPLv3](http://gplv3.fsf.org/).\n\n## Is There Any Authentication on the API or Protection?\n\nBy default, there is no protection on the API but you can configure Butterfly\nin order to have all it's messages encrypted using AES-256.\n\nFor this, you will need to generate a 32 Bytes key (encoded in base 64) in a file and share this file:\n```\nopenssl rand -base64 -out PlaintextKeyMaterial.bin 32\n```\nThen you will need to provide the path to this file using `--key --k` in the command line or `encryption_key_path` option in `butterflyd.ini`.\nOnce a key is correctly loaded, all clear messages will be rejected.\n\nFor encryption format details, check [api/protocol/encrypted.proto](api/protocol/encrypted.proto).\n\n## On Which Port Does Butterfly Listen?\n\nBy default, Butterfly listens on the `tcp://0.0.0.0:9999` port, but it's up to you!\n\nButterfly uses ZeroMQ for message transport and allows you to bind in\n[different ways](http://api.zeromq.org/4-0:zmq-bind). (like tcp, ipc, inproc, pgm, ...)\n\n## Questions? Problems? Contact Us!\n\nButterfly is an open-source project, feel free to [chat with us on IRC](https://webchat.freenode.net/?channels=betterfly\u0026nick=butterfly_user), open\na Github issue or propose a pull request.\n\n\u003e server: irc.freenode.org\n\n\u003e chan: #betterfly\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foutscale%2Fbutterfly","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foutscale%2Fbutterfly","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foutscale%2Fbutterfly/lists"}