https://github.com/cosmonic-labs/netreap
A Cilium controller implementation for Nomad
https://github.com/cosmonic-labs/netreap
cilium cilium-cluster-mesh cilium-operator consul cosmonic netreap nomad
Last synced: about 2 months ago
JSON representation
A Cilium controller implementation for Nomad
- Host: GitHub
- URL: https://github.com/cosmonic-labs/netreap
- Owner: cosmonic-labs
- License: apache-2.0
- Created: 2023-05-02T12:55:30.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-05-21T03:55:07.000Z (over 1 year ago)
- Last Synced: 2024-06-21T21:06:28.740Z (over 1 year ago)
- Topics: cilium, cilium-cluster-mesh, cilium-operator, consul, cosmonic, netreap, nomad
- Language: Go
- Homepage: https://netreap.io
- Size: 771 KB
- Stars: 118
- Watchers: 7
- Forks: 7
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
- awesome-nomad - cosmonic-labs/netreap - Netreap is a Cilium controller designed to manage Cilium networking and policies within Nomad clusters, providing Kubernetes-independent cluster management and policy synchronization. (Infrastructure setup / Automation and Infrastructure Management)
- awesome-cilium - Netreap - Cilium controller implementation for Nomad. (Cilium related projects)
README
# Netreap
Netreap is a non-Kubernetes-based tool for handling Cilium across a cluster,
similar to the functionality of [Cilium
Operator](https://docs.cilium.io/en/v1.13/internals/cilium_operator/#cilium-operator-internals).
It was originally designed just to reap orphaned Cilium
[Endpoints](https://docs.cilium.io/en/v1.13/gettingstarted/terminology/#endpoints),
hence the name of `Netreap`. But we loved the name so much we kept it even
though it does more than reaping.
## So why does this exist?
The current [Cilium
Operator](https://github.com/cilium/cilium/tree/master/operator) only works for
Kubernetes and even when we tried to fork it, Kubernetes was too deeply
ingrained to just pull it out, so we created this little project. This helps
clean up nodes that no longer exist from the KV store, and deletes any
endpoints that no longer have services. Ideally, we will want to make this more
generic and open source so other people can take advantage of this work.
## Running
Instructions for running and configuring Netreap are found below. Please note
that Netreap uses leader election, so multiple copies can (and should) be run.
### Installing
#### Requirements
* A Consul cluster or server
* A running Nomad cluster
* Cilium 1.12.x or 1.13.x
* You will also need to install the [CNI
plugins](https://github.com/containernetworking/plugins/releases/tag/v1.2.0)
alongside Cilium
As of v0.2.0 Consul is no longer required for endpoint reconciliation in
Cilium. You may chose to continue to use Consul as Cilum's KV store, but you
can also use etcd. The install guide assumes you want to use Consul as the
kvstore, since you will need it to distribute Cilium policies.
#### Running Cilium
Due to the way Nomad fingerprinting currently works, you _cannot_ run Cilium as
a system job to provide the CNI plugin. This means you'll need to configure and
run it yourself on every agent that you want to include in the Cilium mesh.
##### Iptables
Make sure that iptables is properly configured on the host:
```bash
cat <<'EOF' | sudo tee /etc/modules-load.d/iptables.conf
iptable_nat
iptable_mangle
iptable_raw
iptable_filter
ip6table_mangle
ip6table_raw
ip6table_filter
EOF
```
##### Cilium Agent
Since you can't run Cilium as a Nomad job right now, the easiest way to run it
is to just use systemd. You can run and enable a job similar to the following:
```systemd
[Unit]
Description=Cilium Agent
After=docker.service
Requires=docker.service
After=consul.service
Wants=consul.service
Before=nomad.service
[Service]
Restart=always
ExecStartPre=-/usr/bin/docker exec %n stop
ExecStartPre=-/usr/bin/docker rm %n
ExecStart=/usr/bin/docker run --rm --name %n \
-v /var/run/cilium:/var/run/cilium \
-v /sys/fs/bpf:/sys/fs/bpf \
--net=host \
--cap-add NET_ADMIN \
--cap-add NET_RAW \
--cap-add IPC_LOCK \
--cap-add SYS_MODULE \
--cap-add SYS_ADMIN \
--cap-add SYS_RESOURCE \
--privileged \
cilium/cilium:v1.13.1 \
cilium-agent --kvstore consul --kvstore-opt consul.address=127.0.0.1:8500 \
--enable-ipv6=false -t geneve \
--enable-l7-proxy=false \
--ipv4-range 172.16.0.0/16
[Install]
WantedBy=multi-user.target
```
Note that this actually runs Cilium with Docker! The reason for this is that
Cilium uses forked versions of some key libraries and needs access to a C
compiler. We found that it is easier to just the container instead of
installing all of Cilium's dependencies.
If you use Consul ACLs, then you will need to add a token to the `Service`
block in the systemd unit so that Cilium can connect to the cluster.
```systemd
[Service]
Environment="CONSUL_HTTP_TOKEN=..."
```
#### Configuring the CNI
The big thing to note is that you need to make sure that the IP CIDR you use
for Cilium does _not_ conflict with what Docker uses if you're using Docker. If
it does or if you want to change Docker's IP range, take a look at the
`default-address-pools` option in `daemon.json`, ex.
```json
{
"default-address-pools": [
{
"base": "192.168.0.0/24",
"size": 24
}
]
}
```
You will then need to make sure you have a CNI configuration for Cilium in
`/opt/cni/config` named `cilium.conflist`:
```json
{
"name": "cilium",
"cniVersion": "1.0.0",
"plugins": [
{
"type": "cilium-cni",
"enable-debug": false
}
]
}
```
Ensure that the Cilium CNI binary is available in `/opt/cni/bin`:
```bash
sudo docker run --rm --entrypoint bash -v /tmp:/out cilium/cilium:v1.13.1 -c \
'cp /usr/bin/cilium* /out; cp /opt/cni/bin/cilium-cni /out'
sudo mv /tmp/cilium-cni /opt/cni/bin/cilium-cni
# Optionally install the other Cilium binaries to /usr/local/bin
sudo mv /tmp/cilium* /usr/local/bin
```
### Running Netreap
Run Netreap as a system job in your cluster similar to the following:
```hcl
job "netreap" {
datacenters = ["dc1"]
priority = 100
type = "system"
constraint {
attribute = "${attr.plugins.cni.version.cilium-cni}"
operator = "is_set"
}
group "netreap" {
restart {
interval = "10m"
attempts = 5
delay = "15s"
mode = "delay"
}
service {
name = "netreap"
tags = ["netreap"]
}
task "netreap" {
driver = "docker"
config {
image = "ghcr.io/cosmonic/netreap:0.2.0"
network_mode = "host"
# You must be able to mount volumes from the host system so that
# Netreap can use the Cilium API over a Unix socket.
# See
# https://developer.hashicorp.com/nomad/docs/drivers/docker#plugin-options
# for more information.
volumes = [
"/var/run/cilium:/var/run/cilium"
]
}
}
}
}
```
The job constraint ensures that Netreap will only run on nodes where the
Cilium CNI is available.
If you use Nomad or Consul ACLs then you will need to set them in the Netreap
job, ex.
```hcl
template {
destination = "secrets/file.env"
env = true
change_mode = "restart"
data = <