{"id":15056999,"url":"https://github.com/k8sstormcenter/honeycluster","last_synced_at":"2025-04-10T05:06:56.891Z","repository":{"id":226914124,"uuid":"769948033","full_name":"k8sstormcenter/honeycluster","owner":"k8sstormcenter","description":"Threat-informed defense for cloudnative: Reference Implementation of a so-called Honeycluster - for kind (and GKE, RKE2, AKS)","archived":false,"fork":false,"pushed_at":"2025-03-25T19:00:32.000Z","size":4231,"stargazers_count":36,"open_issues_count":23,"forks_count":4,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-04-10T05:06:56.118Z","etag":null,"topics":["cloudnative","cybersecurity","ebpf","kubernetes","threat-intelligence"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/k8sstormcenter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"entlein"}},"created_at":"2024-03-10T14:10:23.000Z","updated_at":"2025-04-07T11:09:58.000Z","dependencies_parsed_at":"2024-05-09T12:40:49.860Z","dependency_job_id":"f149a3e5-ed05-4828-9466-442b036b073c","html_url":"https://github.com/k8sstormcenter/honeycluster","commit_stats":null,"previous_names":["k8sstormcenter/honeycluster"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/k8sstormcenter%2Fhoneycluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/k8sstormcenter%2Fhoneycluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/k8sstormcenter%2Fhoneycluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/k8sstormcenter%2Fhoneycluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/k8sstormcenter","download_url":"https://codeload.github.com/k8sstormcenter/honeycluster/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248161274,"owners_count":21057555,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudnative","cybersecurity","ebpf","kubernetes","threat-intelligence"],"created_at":"2024-09-24T21:59:51.063Z","updated_at":"2025-04-10T05:06:56.867Z","avatar_url":"https://github.com/k8sstormcenter.png","language":"Shell","funding_links":["https://github.com/sponsors/entlein"],"categories":[],"sub_categories":[],"readme":"# K8sStormCenter: Honey Storm and Lightening\n\n\u003e [!NOTE]\n\u003e This deployment is an opinionated collection of offensive and defensive open-source security and observability tools. It contains destructive elements, and it is NOT to be used in production under any circumstances \n\nWelcome to the K8sStormCenter HoneyCluster repository. Here you will find everything you need to set up your own HoneyCluster: a Kubernetes cluster that is instrumented with bait and tripwires to collect data on the attacks carried out against it. With our complimentary [lightening-rod](https://github.com/k8sstormcenter/cti-stix-visualization) , the `attack paths` can be visualized and are made `shareable` using STIX. You can also use the `lightening`-feature to attack yourself to create a threat-model-starting point or to generally scan your setup. The `storm` feature is the collection of CTI from many `HoneyClusters` into a open shared threat-intel collection.\n\nWe expressely thank the upstream maintainers of our main components: KubeHound, KubeScape, Vector, Tetragon, Falco, Oasis/STIX, Kubesploit\n\nIf you like it -\u003e consider leaving a star ⭐\n\n\u003c!--![kubecon25_stormcenter_harbor](https://github.com/user-attachments/assets/262115d6-fbd4-475b-bbb4-4a4a0af84db1) --\u003e\n![StormcenterEnd2EndwithBG](https://github.com/user-attachments/assets/5d6a0614-8397-43c0-874b-6b59de1c819f)\n\n\n## Table of Contents\n\n- [K8sStormCenter HoneyCluster](#k8sstormcenter-honeycluster)\n  - [Table of Contents](#table-of-contents)\n  - [How does it work?](#how-does-it-work)\n    - [Creating a HoneyCluster](#creating-a-honeycluster)\n    - [The four fold path to threat intelligence](#the-four-fold-path-to-threat-intelligence)\n    - [Threat Model](#threat-model)\n      - [Generate a full Threat Tree using OSS tools such as Kubehound](#generate-a-full-threat-tree-using-oss-tools-such-as-kubehound)\n    - [Attack Model](#attack-model)\n  - [Getting started](#getting-started)\n    - [1. Create a Kubernetes Cluster](#1-create-a-kubernetes-cluster)\n    - [2. Set up the HoneyCluster](#2-set-up-the-honeycluster)\n    - [3.  Baselining](#3--baselining)\n    - [4.  Lightening](#4--lightening)\n    - [5. Create STIX observables](#5-create-stix-observables)\n    - [5. Teardown](#5-teardown)\n  - [Tailoring the instrumentation to your needs](#tailoring-the-instrumentation-to-your-needs)\n    - [Tracing Policies](#tracing-policies)\n    - [Application \\\u0026 Audit Logs](#application--audit-logs)\n    - [Mapping and Matching: Stix Observables and Stix Indicators](#mapping-and-matching-stix-observables-and-stix-indicators)\n  - [Experiment: Detect Leaky Vessel on live clusters](#experiment-detect-leaky-vessel-on-live-clusters)\n    - [Bait](#bait)\n    - [Security Considerations](#security-considerations)\n  - [Contributing](#contributing)\n\n\n\n## How does it work?\n\nYou start with your \"normal\" cluster, where you wish to\n\n- verify/quantify theoretical threat modelling assumptions, or to\n- simply observe how your cluster will be attacked by interpreting the anomalous signals\n\nHowever, you don't want to expose your cluster to real threats. That's where the HoneyCluster comes in. A HoneyCluster is a cluster that looks like a normal cluster, but is instrumented with tripwires and bait to collect data on the attacks carried out against it. Because the HoneyCluster looks like a real cluster, the collected data is representative of the attacks that would be carried out against a real cluster and therefore can give us insights into the behaviour of attackers and the ways in which they target our clusters.\nYou decide, how much of your production scope you ll clone into this \"HoneyCluster\". We recommend, using the same IaC you already have and put it into a throw-away tenant(Azure)/project(GCP). I d recommend additionally, not using any IAM/IdP or other identiies or key material that has scopes beyond this tenant. Especially if you want to see critical kind of attacks... \nPlease note, that you are solely responsible for the risk of lateral-movements/pivots, if you deploy the HoneyCluster in shared-\u003canything\u003e configuration.\n\nWIP: I ll add GCP terraform templates including all IAM to work standalone, as a reference.\n\n\n### Creating a HoneyCluster\n\n\u003cimg width=\"1083\" alt=\"Screenshot 2024-04-26 at 22 32 32\" src=\"https://github.com/k8sstormcenter/honeycluster/assets/70207455/f574e663-fb7b-4c43-af6f-b3544b8b63a6\"\u003e\n\nTo set up a HoneyCluster, you start by creating a copy of your \"normal\" cluster. This copy has the same services as the real cluster, but without its sensitive data and with some additional instrumentation to collect data on the attacks carried out against it. This data is then used to create a baseline of normal behaviour, which is used to filter out benign signals. The remaining signals are then used to detect and understand the attacks carried out against the cluster.\n\n\n\n### The four fold path to threat intelligence\n\nThe HoneyCluster is a tool to gather data about attack patterns, but on its own, it does not provide us with any actionable insights. To attain knowledge and insights about attacks, the following steps are necessary:\n\n1. Threat Model -\u003e Attack Model -\u003e Critical Attack Path\n2. Instrument a honeycluster with eBPF tripwires and some bait\n3. Trace and stream events, remove baseline\n4. Disseminate the Threat Intelligence\n\n\n### Threat Model\n\nA relatively generic Threat Model could for example look like this:\n\n```mermaid\nflowchart TD\n    A[Access sensitive data] --\u003e B{Establish \\npersistence}\n    A --\u003e C{Command \\nand Control}\n    B --\u003e BB[Pod with \\nwriteable hostPath]\n    A --\u003e BB\n    A --\u003e H[Pod uses PVC \\nwhich references a \\nhostPath PV]\n    B --\u003e BC[crontab \\non node]\n    B --\u003e BD[OR static pod ]\n    B --\u003e C\n    C --\u003e D{Weapon \\npod}\n    H --\u003e D\n    C --\u003e DE[reverse shell]\n    D --\u003e E[App Vulnerability\\nallows RCE]\n    D --\u003e EE[ServiceAccount \\n creates Pod]\n```\n\nWIP: replace with more generic \n\n#### Generate a full Threat Tree using OSS tools such as Kubehound\nOne of the critical aspects of modelling threats is having a good threat model to start with. \nThis is why we looked at the cloud-native OSS space and are collecting the best tools and combining them\nto generate a good starting point. \nWe thank the Kubehound maintainers for a lot of inspiration!\n\n### Attack Model\n\nBased on the Threat Model, you now create a concrete AttackModel (or many).\n\nYou can use this for multiple purposes:\n- to understand if a ThreatModel can be exploited IRL . You can attack yourself or hire an offensive expert, create a bug bounty program etc.\n- to  calibrate all event-producing instrumentation in your deployment: can you see events from executing your attack-model, if not, you might need to add more TracingPolicies or change some filters.\n- to verify the pattern-matching between your events and your STIX observables: the `cti-stix-visualizer` UI can help calibrating\n- to simulate a breach: once you have at least one attack model implemented (e.g. via bash-script), you can test diverse detective/responsive processes in your deployment, e.g. if your pager starts blinking.\n\n\n\n## Getting started\n\nYou want to deploy your own HoneyCluster? We'll guide you through the setup of a local Kubernetes (`kind`) cluster, the installation of the necessary instrumentation and the collection of the baseline behaviour. Once you have set up your HoneyCluster, you can start experimenting with a standardized attack-suite, writing patterns, analysing logs, calibrating a baseline and observing the final signals. \n\n\n### 1. Create a Kubernetes Cluster\n\n\u003e [!NOTE]\n\u003e If you have already set up a Kubernetes cluster you want to utilize as a HoneyCluster, you can skip this part and head to [step 2](#2-set-up-the-honeycluster). Make sure to have [cert-manager](https://cert-manager.io/docs/) installed on it.\n\nIn order to set up your own HoneyCluster, you first need a Kubernetes cluster on which you can install the necessary instrumentation. For local explorative scenarios we provide a Kubernetes setup based on Kind, which you can create using the following command:\n\n```bash\nmake cluster-up\n```\n\n\n\n### 2. Set up the HoneyCluster\n\n\u003e [!IMPORTANT]  \n\u003e Make sure to have the correct Kubernetes config context set. From this point on we will make changes to the cluster specified as the current context, and we wouldn't want them to end up on the wrong cluster.\n\nOnce you have a running Kubernetes cluster, you can go ahead and install the instrumentation to put the honey in your cluster with the following command:\n  \n```bash\nmake honey-up\n```\n\nThe `honey-up` target installs all the necessary components to collect eBPF traces, application logs and soon audit logs. It is important that the cluster is not yet exposed to an active threat at this point because after the installation we collect the baseline behaviour of the cluster to filter out benign signals.\n\nWhile we provide you with a set of default traces and log forwarding configurations, you can adjust them to your needs in the [traces](traces/) and [honeystack/vector/values.yaml](honeystack/vector/gkevalues.yaml) files respectively. Find out more about it in the [Tailoring the instrumentation to your needs](#tailoring-the-instrumentation-to-your-needs) section.\n\nWhile in `Calibration Mode` , you ll likely want to work with a small batch of logs to get the coarse filters in place:\n\n```bash\nkubectl port-forward service/redis-headless -n storm 6379:6379\nkubectl port-forward service/stix-visualizer -n storm 80:3000\nkubectl port-forward service/lighteningrod -n storm 8000:8000\n```\n\nAfter port-forwarding, you can access the UI [http://localhost:30000](http://localhost:30000), edit your `Patterns` and `activate Patterns` to be used by the `lightening-rod` (a microservice that matches patterns and converts logs to a standard STIX 2.1 format)\nWIP: For the `Kubehound-inspired Calibration`, there is a set of sample patterns [lightening-rod/testpost.sh](lightening-rod/testpost.sh). You should add/modify: e.g. if you click on `Add Pattern`, it'll give you the minimal example of one to fill out:\n\n\n![LighteningSTIXViz_KubernetesStormcenter_downsampled](https://github.com/user-attachments/assets/55b2f311-3b98-4237-953c-f40fada18d52)\n\n\n\n### 3.  Baselining\nAt this point, you ll likey want to mark all your current logs as `benign`, this will reduce the noise considerably.\nIf you want to be sure that your `HoneyCluster` has reached equilibrium, check the rate at which logs are appended to the `redis`DB \"table\"=`tetra` . On `kind`, the rate will go to zero after the all services are booted up.\n\n\nYou achieve this, first put on all traces, then load all your current logs into a temporary table:\n```\nmake --makefile=Makefile_calibrate_kubehound calibration-traces\nhttp://localhost:3000/reload-tetra\nkubectl port-forward -n storm lightening-rod 8000:8000\nchmod +x lightening-rod/testpost.sh\n./lightening-rod/testpost.sh\n```\n\nand once all the preps are done: press the `Mark all logs as BENIGN` button in the UI.\n\nAt this point (if you inspect `redis`), you have 4 tables: \n`tetra` = list of all logs (vector-sink)\n`raw_logs` = temporary list to stage logs for processing\n`benign_logs` = hashtable for md5-hashes of bengin logs\n`tetra_pattern` = hashtable for json STIX 2.1 attack-patterns\n\n\n### 4.  Lightening\n\u003e[!IMPORTANT]\n\u003e Not all attacks are implemented, not all traces make sense ATM, this is a great contribution opportunity\n\u003e\nAttack yourself with the calibration attacks in the namespace `lightening` to trigger the tripwires (else running the analysis over the `Active Patterns` is not going to turn up any `matched STIX-bundles`). You can choose between `calibration` or `lightening` depending on \n```\nmake --makefile=Makefile_calibrate_kubehound calibrate\nor\nmake lightening\n```\nGive it a minute or two until those attacks that will succeed have succeeded (pull their images or booted up etc), then you should remove the `lightening` again (else you ll have duplicate logs that contain no new insights)\n\n```\nmake --makefile=Makefile_calibrate_kubehound wipe\nor\nmake lightening-off\n```\n\nNow, in your `redis` DB, the table `tetra` will have accumulated a few thousand logs. \n\n\u003cimg width=\"1426\" alt=\"Screenshot 2024-12-02 at 12 24 46\" src=\"https://github.com/user-attachments/assets/b1b064f4-55ec-4cb2-9370-5f5fb3d104e2\"\u003e\n\n\nNow, back in the UI, press `Select all logs for processing` \n\n### 5. Create STIX observables\n\nWith the HoneyCluster set up and the baseline behaviour filtered out, and the calibration being sucessful (i.e. your traces, patterns, filters and other `yaml` files are edited to your satisfaction), we can now attack the cluster with ever more real scenarios.\n\nIt is recommended to test the quality of your `threat-observability` by running some simple attacks that - unlike the calibration suite- are not 100% pre-configured.\n\nThis repo provides some examples/ideas YMMV:\nIn the attack makefile, we provide you with a simple simple bait to deploy and attack execute on the cluster. The bait consists of an ssh server with weak credentials, giving an attacker root access on the pod from which more damage can be done because of misconfigured RBAC. To deploy the bait: \n\n```bash\nmake --makefile=Makefile_attack bait\n```\n\nAfterwards, you can simply try connecting to the ssh server first:\n\n```bash\nmake --makefile=Makefile_attack ssh-connect\n```\n\nWhen prompted, the password is `root`. You should see the newly established ssh connection being picked up by the eBPF traces.\n\nNow that you have gotten a feel of how unusual traffic is picked up by the HoneyCluster, you can try executing an actual attack. For that, we prepared an attack path made possible if an attacker can create `/var/log` hostPath Persistent Volumes on a cluster, inspired by [this blog post](https://jackleadford.github.io/containers/2020/03/06/pvpost.html), depicted in the following attack tree.\n\n```mermaid\nflowchart TD\n    A[Access sensitive \\ninfo on node] --\u003e B{kubectl logs BAD_POD}\n    A --\u003e H[Pod uses PVC \\nwhich references a \\nhostPath PV]\n    B --\u003e I[Pod writes symlink \\nto 0.log file]\n    B --\u003e C[Container can \\nrun as root]\n    B --\u003e D[Pod with \\nwriteable hostPath \\nto /var/log]\n    D --\u003e E{Ability to create \\nK8s resources}\n    H --\u003e E\n    E --\u003e F[Misconfigured RBAC]\n    E --\u003e G[Initial access \\nto Pod]\n```\n\nClose the SSH connection, and run the full attack which will again make an SSH connection to our vulnerable server, run a malicious script which will create a HostPath type PersistentVolume, allowing a pod to access `/var/log` on the host (inspired by [this blog post](https://jackleadford.github.io/containers/2020/03/06/pvpost.html)), using the [Python Kubernetes client library](https://github.com/kubernetes-client/python). Note that you could modify the hostPath in the Python script to go directly for the data on the host that you want to compromise, however, in order to increase the number of attack steps in our scenario (and hence the number of indicators that we can look for), let's imagine that we are not able to create arbitrary hostPaths. In this scenario, perhaps a `hostPath` type `PersistentVolume` is allowed for `/var/log` so that a Pod can monitor other Pod's logs.\n\n\nWIP: TODO: check if below commands are still supported\n```bash\nmake --makefile=calibrate_kubehound calibrate\nmake --makefile=Makefile_attack attack\n```\n\nWhen prompted, the password is again `root`.\n\nIf the service account compromised by our attacker could inspect the logs of the containers it can create, running `kubectl logs bad-pv-pod --tail=-1` (or making an API call from within the bad pod) will enable an attacker to view arbitrary files (line by line) on the host. In this example, we have a single node cluster, so we can access control plane data.\n\nTOOO: insert SSH detction here.\n\n### 5. Teardown\n\nAfter you have finished experimenting with your HoneyCluster, you can remove the bait:\n\n```bash\nmake --makefile=Makefile_attack bait-delete\nmake --makefile=calibrate_kubehound wipe\n```\n\nAnd finally, you can wipe the HoneyCluster instrumentation from your Kubernetes cluster:\n\n```bash\nmake wipe\n```\n\n\n\n## Tailoring the instrumentation to your needs\n\nThis repository gives you a framework to run experiments, simulations and to make threat-modelling concrete and actionable.\nThe \"Stormcenter\" contains a blue (`honey` and `lightening-rod`) and a red (`lightening` and `storm`) half: blue is all about bait, observability, detection and response while red is about having a clear entry-point for anyone to use their red-tooling and measure the damage or attack-efficiency. \nIn the end, I m interested in understanding how humans behave in cyber-systems - on either side.\n\n\n### Tracing Policies\n\nThis paragraph is about choosing Tetragon tracing policies that work for you. Tetragon uses eBPF technology to trace kernel and system events, providing detailed insights into system behavior.\n\nHere's an example tracing policy:\n\n```yaml\napiVersion: cilium.io/v1alpha1\nkind: TracingPolicy\nmetadata:\n  name: \"monitor-network-activity-outside-cluster-cidr-range\"\nspec:\n  kprobes:\n  - call: \"tcp_connect\"\n    syscall: false\n    args:\n    - index: 0\n      type: \"sock\"\n    selectors:\n    - matchArgs:\n      - index: 0\n        operator: \"NotDAddr\"\n        values:\n        - 127.0.0.1\n        - 172.16.0.0/28\n        - 192.168.64.0/24\n```\n\nIn this example, the policy monitors tcp_connect events, filtering out connections to specific IP ranges and capturing only those to other addresses. This helps ensure that only relevant and interesting tcp information is gathered. See subfolder `/traces` for more examples.\n\n\n### Application \u0026 Audit Logs\n\nThis paragraph is about application (incl audit) and networking logs. WIP\n\nComing soon: examples and how to test it locally\n\n\n### Mapping and Matching: Stix Observables and Stix Indicators\n\nThe collected logs are transformed into [STIX observables](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_mlbmudhl16lr), which are then matched against [STIX indicators](https://docs.oasis-open.org/cti/stix/v2.1/cs01/stix-v2.1-cs01.html#_muftrcpnf89v). Observables, which match the provided indicators represent potentially malicious behavior and are persisted into a document store. More detailed information on the setup of the indicators and how the matching works is provided in the [README](https://github.com/k8sstormcenter/threatintel/blob/main/README.md) of the threatintel repository.\n\n[![Detection](./docs/log-detection.png)](https://drive.google.com/file/d/1RfPr_7RmXDlU22-l7ZFoMnWJKloP0VpG/view?usp=sharing)\n\n\n\n## Experiment: Detect Leaky Vessel on live clusters\n\nWe show a simple and unspecific detection of Leaky Vessel via Supply Chain (cf. KubeCon Europe 2024) and an elaborate breach using Leaky Vessel for `priviledge escalation` (cf. KCD Munich 2024). No additional cluster instrumentation was needed, no specific assumptions were made, etc.\n\nYou can watch a recording of the KCD Munich 2024 talk here:\n\n[![K8sStormCenter @ KCD Munich 2024](https://img.youtube.com/vi/axh7SOufh8M/0.jpg)](https://www.youtube.com/watch?v=axh7SOufh8M)\n\nThe KubeCon Europe 2024 talk can be viewed here:\n\n[![K8sStormCenter @ KubeCon Europe 2024](https://img.youtube.com/vi/RNYz86uDXLc/0.jpg)](https://www.youtube.com/watch?v=RNYz86uDXLc)\n\nThe video above shows the poisoning of a registry with an image exploiting CVE-2024-21626 \"Leaky-Vessel\" by tagging and pushing the poisoned image with identical name/tag as the original image. (This is a type of Supply Chain Attack).\n\nTwo different RKE2 clusters (intentionally running a vulnerable `runc`) are observed by streaming the `smb` topic in the Redpanda UI. When the poisoned image is pulled and started up, the traces appear on the topic. As well as we see the sensitive-file-access to the private key on the host-node, as well as the newly created file `LEAKYLEAKY` on the host node.\n\n\n### Bait\n\nWe don't share details about the bait used in our experiments here since we don't want potential attackers to know what to look for. However, if you are interested feel free to reach out to us on our [Slack channel](https://join.slack.com/t/k8sstorm/shared_invite/zt-2hulzsqh1-mnZL6fGFZiVLrOcqeTObIA)!\n\n\n### Security Considerations\n\nGiven this is an insecure and experimental setup of a honeypot-infrastructure, there are several additional measures taken that are not covered in the talk or this repo.\nThis repo is for demonstration purposes only.\n\n\n\n## Contributing\n\nContributions are always welcome!\n\n(For example in the form of testing, feedback, code, PRs, eBPF tripwires, realistic threatmodels, mappings onto the critical attack path or anything you think could be useful for others)\n\nTODO: write contributor guidelines\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fk8sstormcenter%2Fhoneycluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fk8sstormcenter%2Fhoneycluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fk8sstormcenter%2Fhoneycluster/lists"}