{"id":23008264,"url":"https://github.com/cosmonic-labs/netreap","last_synced_at":"2026-03-09T22:02:50.772Z","repository":{"id":160676241,"uuid":"635321739","full_name":"cosmonic-labs/netreap","owner":"cosmonic-labs","description":"A Cilium controller implementation for Nomad","archived":false,"fork":false,"pushed_at":"2024-05-21T03:55:07.000Z","size":789,"stargazers_count":142,"open_issues_count":15,"forks_count":9,"subscribers_count":8,"default_branch":"main","last_synced_at":"2026-01-12T00:28:44.223Z","etag":null,"topics":["cilium","cilium-cluster-mesh","cilium-operator","consul","cosmonic","netreap","nomad"],"latest_commit_sha":null,"homepage":"https://netreap.io","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cosmonic-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-02T12:55:30.000Z","updated_at":"2025-12-03T02:04:00.000Z","dependencies_parsed_at":"2023-05-06T01:08:58.002Z","dependency_job_id":"338f42fa-60a5-4579-88a3-1f62f262f9b9","html_url":"https://github.com/cosmonic-labs/netreap","commit_stats":null,"previous_names":["cosmonic/netreap"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/cosmonic-labs/netreap","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cosmonic-labs%2Fnetreap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cosmonic-labs%2Fnetreap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cosmonic-labs%2Fnetreap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cosmonic-labs%2Fnetreap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cosmonic-labs","download_url":"https://codeload.github.com/cosmonic-labs/netreap/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cosmonic-labs%2Fnetreap/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30314371,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-09T20:05:46.299Z","status":"ssl_error","status_checked_at":"2026-03-09T19:57:04.425Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cilium","cilium-cluster-mesh","cilium-operator","consul","cosmonic","netreap","nomad"],"created_at":"2024-12-15T09:00:47.449Z","updated_at":"2026-03-09T22:02:50.752Z","avatar_url":"https://github.com/cosmonic-labs.png","language":"Go","funding_links":[],"categories":["Infrastructure setup","Go","Cilium related projects"],"sub_categories":["Automation and Infrastructure Management"],"readme":"![netreap-social](https://github.com/cosmonic-labs/netreap/assets/1687902/681901f1-8717-43d3-87e1-311e7a5baf83)\n\n# Netreap\n\nNetreap is a non-Kubernetes-based tool for handling Cilium across a cluster,\nsimilar to the functionality of [Cilium\nOperator](https://docs.cilium.io/en/v1.13/internals/cilium_operator/#cilium-operator-internals).\nIt was originally designed just to reap orphaned Cilium\n[Endpoints](https://docs.cilium.io/en/v1.13/gettingstarted/terminology/#endpoints),\nhence the name of `Netreap`. But we loved the name so much we kept it even\nthough it does more than reaping.\n\n## So why does this exist?\n\nThe current [Cilium\nOperator](https://github.com/cilium/cilium/tree/master/operator) only works for\nKubernetes and even when we tried to fork it, Kubernetes was too deeply\ningrained to just pull it out, so we created this little project. This helps\nclean up nodes that no longer exist from the KV store, and deletes any\nendpoints that no longer have services. Ideally, we will want to make this more\ngeneric and open source so other people can take advantage of this work.\n\n## Running\n\nInstructions for running and configuring Netreap are found below. Please note\nthat Netreap uses leader election, so multiple copies can (and should) be run.\n\n### Installing\n\n#### Requirements\n\n* A Consul cluster or server\n* A running Nomad cluster\n* Cilium 1.12.x or 1.13.x\n  * You will also need to install the [CNI\n    plugins](https://github.com/containernetworking/plugins/releases/tag/v1.2.0)\n    alongside Cilium\n\nAs of v0.2.0 Consul is no longer required for endpoint reconciliation in\nCilium. You may chose to continue to use Consul as Cilum's KV store, but you\ncan also use etcd. The install guide assumes you want to use Consul as the\nkvstore, since you will need it to distribute Cilium policies.\n\n#### Running Cilium\n\nDue to the way Nomad fingerprinting currently works, you _cannot_ run Cilium as\na system job to provide the CNI plugin. This means you'll need to configure and\nrun it yourself on every agent that you want to include in the Cilium mesh.\n\n##### Iptables\n\nMake sure that iptables is properly configured on the host:\n\n```bash\ncat \u003c\u003c'EOF' | sudo tee /etc/modules-load.d/iptables.conf\niptable_nat\niptable_mangle\niptable_raw\niptable_filter\nip6table_mangle\nip6table_raw\nip6table_filter\nEOF\n```\n\n##### Cilium Agent\n\nSince you can't run Cilium as a Nomad job right now, the easiest way to run it\nis to just use systemd. You can run and enable a job similar to the following:\n\n```systemd\n[Unit]\nDescription=Cilium Agent\nAfter=docker.service\nRequires=docker.service\nAfter=consul.service\nWants=consul.service\nBefore=nomad.service\n\n[Service]\nRestart=always\nExecStartPre=-/usr/bin/docker exec %n stop\nExecStartPre=-/usr/bin/docker rm %n\nExecStart=/usr/bin/docker run --rm --name %n \\\n  -v /var/run/cilium:/var/run/cilium \\\n  -v /sys/fs/bpf:/sys/fs/bpf \\\n  --net=host \\\n  --cap-add NET_ADMIN \\\n  --cap-add NET_RAW \\\n  --cap-add IPC_LOCK \\\n  --cap-add SYS_MODULE \\\n  --cap-add SYS_ADMIN \\\n  --cap-add SYS_RESOURCE \\\n  --privileged \\\n  cilium/cilium:v1.13.1 \\\n  cilium-agent --kvstore consul --kvstore-opt consul.address=127.0.0.1:8500 \\\n    --enable-ipv6=false -t geneve \\\n    --enable-l7-proxy=false  \\\n    --ipv4-range 172.16.0.0/16\n\n[Install]\nWantedBy=multi-user.target\n```\n\nNote that this actually runs Cilium with Docker! The reason for this is that\nCilium uses forked versions of some key libraries and needs access to a C\ncompiler. We found that it is easier to just the container instead of\ninstalling all of Cilium's dependencies.\n\nIf you use Consul ACLs, then you will need to add a token to the `Service`\nblock in the systemd unit so that Cilium can connect to the cluster.\n\n```systemd\n[Service]\nEnvironment=\"CONSUL_HTTP_TOKEN=...\"\n```\n\n#### Configuring the CNI\n\nThe big thing to note is that you need to make sure that the IP CIDR you use\nfor Cilium does _not_ conflict with what Docker uses if you're using Docker. If\nit does or if you want to change Docker's IP range, take a look at the\n`default-address-pools` option in `daemon.json`, ex.\n\n```json\n{\n  \"default-address-pools\": [\n    {\n      \"base\": \"192.168.0.0/24\",\n      \"size\": 24\n    }\n  ]\n}\n\n```\n\nYou will then need to make sure you have a CNI configuration for Cilium in\n`/opt/cni/config` named `cilium.conflist`:\n\n```json\n{\n  \"name\": \"cilium\",\n  \"cniVersion\": \"1.0.0\",\n  \"plugins\": [\n     {\n       \"type\": \"cilium-cni\",\n       \"enable-debug\": false\n     }\n  ]\n}\n```\n\nEnsure that the Cilium CNI binary is available in `/opt/cni/bin`:\n\n```bash\nsudo docker run --rm --entrypoint bash -v /tmp:/out cilium/cilium:v1.13.1 -c \\\n  'cp /usr/bin/cilium* /out; cp /opt/cni/bin/cilium-cni /out'\nsudo mv /tmp/cilium-cni /opt/cni/bin/cilium-cni\n# Optionally install the other Cilium binaries to /usr/local/bin\nsudo mv /tmp/cilium* /usr/local/bin\n\n```\n\n### Running Netreap\n\nRun Netreap as a system job in your cluster similar to the following:\n\n```hcl\njob \"netreap\" {\n  datacenters = [\"dc1\"]\n  priority    = 100\n  type        = \"system\"\n\n  constraint {\n    attribute = \"${attr.plugins.cni.version.cilium-cni}\"\n    operator = \"is_set\"\n  }\n\n  group \"netreap\" {\n    restart {\n      interval = \"10m\"\n      attempts = 5\n      delay = \"15s\"\n      mode = \"delay\"\n    }\n    service {\n      name = \"netreap\"\n      tags = [\"netreap\"]\n    }\n\n    task \"netreap\" {\n      driver = \"docker\"\n\n      config {\n        image        = \"ghcr.io/cosmonic/netreap:0.2.0\"\n        network_mode = \"host\"\n\n        # You must be able to mount volumes from the host system so that\n        # Netreap can use the Cilium API over a Unix socket.\n        # See\n        # https://developer.hashicorp.com/nomad/docs/drivers/docker#plugin-options\n        # for more information.\n        volumes = [\n          \"/var/run/cilium:/var/run/cilium\"\n        ]\n      }\n\n    }\n  }\n}\n```\n\nThe job constraint ensures that Netreap will only run on nodes where the\nCilium CNI is available.\n\nIf you use Nomad or Consul ACLs then you will need to set them in the Netreap\njob, ex.\n\n```hcl\n\n      template {\n        destination = \"secrets/file.env\"\n        env         = true\n        change_mode = \"restart\"\n        data        = \u003c\u003cEOT\nCONSUL_HTTP_TOKEN=\"...\"\nNOMAD_TOKEN=\"...\"\nEOT\n      }\n```\n\nNote that all environment variables used to configure the Consul and Nomad API\nclients are available to Netreap.\n\n### Configuring\n\n| Flag                   | Env Var               | Default                       | Description                                                                                                   |\n| ---------------------- | --------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------- |\n| `--debug`              | `NETREAP_DEBUG`       | `false`                       | Turns on debug logging                                                                                        |\n| `--policy-key`         | `NETREAP_POLICY_KEY`  | `netreap.io/policy`           | Consul key that Netreap watches for changes to the Cilium policy JSON value |\n\nPlease note that to configure the Nomad, Consul and Cilium clients that Netreap uses,\nwe leverage the well defined environment variables for\n[Nomad](https://www.nomadproject.io/docs/commands#environment-variables),\n[Consul](https://www.consul.io/commands#environment-variables) and\n[Cilium](https://pkg.go.dev/github.com/cilium/cilium/pkg/client#DefaultSockPath).\n\nRight now we only allow connecting to the local Unix socket endpoint for the\nCilium agent. As we determine how we are going to set things up with Cilium, we\ncan add additional configuration options.\n\n### Cilium Policies\n\nOne of Netreap's key responsibilities is to sync [Cilium\npolicies](https://docs.cilium.io/en/stable/security/policy/) to every node in\nyour Cilium mesh. Normally Cilium policies are configured using Kubernetes\nCRDs, but we don't have that option when we're running Nomad. Normally Cilium\ncombines all of the CRD values in to a single JSON representation which is\nimported by every agent. What this means is that Netreap does the same thing by\nwatching a single Consul key that stores the complete JSON representation of\nall of the Cilium policies in your cluster. The official documentation has\nexamples on how to write policies in JSON.\n\nWhenever you want to update policies in your cluster, simply set the key in\nConsul:\n\n```bash\nconsul kv put netreap.io/policy @policy.json\n```\n\nNetreap automatically picks up any updates to the value and updates the policy\non every node where it is running.\n\n## Development\n\nNetreap is written in pure Go, no other build tools are required other than a\nworking Go toolchain.\n\nOn the other hand, actually using it is a bit more difficult. You need the\nfollowing things set up on a Linux machine:\n\n* Consul agent running (no special configuration required, can just use `-dev`\n  if you want)\n* Nomad configured to use Docker volumes\n* Cilium installed using the directions in [Running Cilium](###Running-Cilium).\n\n### Testing\n\nBecause of all of the necessary pieces described in the previous section, we\ndon't have any automated tests in place yet. For now, here are some steps to\ntest manually:\n\n* Start a job and then start netreap with the `--debug` flag, making sure the\n  logs say that it is labeling it\n* Run `cilium endpoint list` and make sure the endpoint is showing a label that\n  looks something like this: `netreap:job_id=example`\n* Stop the job and make sure the logs note that the reap counter was\n  incremented\n* Start a job and make sure the logs note that it saw the new job. Run `cilium\n  endpoint list` to make sure the endpoint was properly labeled\n* Stop netreap and then start it again, making sure the logs say that it is\n  deleting an endpoint (from the previous job you stopped). Run `cilium\n  endpoint list` to make sure the endpoint was properly deleted.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcosmonic-labs%2Fnetreap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcosmonic-labs%2Fnetreap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcosmonic-labs%2Fnetreap/lists"}