{"id":15642877,"url":"https://github.com/developer-guy/falco-analyze-audit-log-from-k3s-cluster","last_synced_at":"2025-04-23T05:34:03.975Z","repository":{"id":104875810,"uuid":"340747860","full_name":"developer-guy/falco-analyze-audit-log-from-k3s-cluster","owner":"developer-guy","description":"Detect intrusions that happened in your Kubernetes cluster through audit logs using Falco ","archived":false,"fork":false,"pushed_at":"2021-06-02T18:33:43.000Z","size":3396,"stargazers_count":63,"open_issues_count":0,"forks_count":9,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-02-04T23:55:20.347Z","etag":null,"topics":["audit-log","falco","falcosidekick","falcosidekickui","k3s","k8s-audit-log"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/developer-guy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-02-20T20:33:22.000Z","updated_at":"2025-01-15T14:40:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"bc6c730c-1b9a-452a-957b-6271fa24219f","html_url":"https://github.com/developer-guy/falco-analyze-audit-log-from-k3s-cluster","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developer-guy%2Ffalco-analyze-audit-log-from-k3s-cluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developer-guy%2Ffalco-analyze-audit-log-from-k3s-cluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developer-guy%2Ffalco-analyze-audit-log-from-k3s-cluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developer-guy%2Ffalco-analyze-audit-log-from-k3s-cluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/developer-guy","download_url":"https://codeload.github.com/developer-guy/falco-analyze-audit-log-from-k3s-cluster/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246251061,"owners_count":20747447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["audit-log","falco","falcosidekick","falcosidekickui","k3s","k8s-audit-log"],"created_at":"2024-10-03T11:57:58.609Z","updated_at":"2025-03-29T22:25:35.524Z","avatar_url":"https://github.com/developer-guy.png","language":null,"funding_links":[],"categories":["Tutorials \u0026 Learning"],"sub_categories":[],"readme":"![falco_audit_log](./assets/falco_audit_log2.png)\n\n# Analyze Kubernetes Audit logs using Falco\nThere is concept called \"Event Sources\" in Falco, these \"Event Sources\" defines where Falco can consume events, and apply rules to these events to detect abnormal behavior. Currently Falco supports thefollowing event sources:\n\n* System Calls (syscall) via the [drivers](https://falco.org/docs/event-sources/drivers)\n* [Kubernetes Audit Events](https://falco.org/docs/event-sources/kubernetes-audit) (k8s_audit)\n\nIn this guide, we are going to use \"Kubernetes Audit Logs\" as an event source that Falco can consume.\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n\n# Table of Contents\n- 🧰 [Prerequisites](#prerequisites)\n\n- 🎁 [What is in it for you ?](#what-is-in-it-for-you-)\n\n- \u003cimg src=\"https://cncf-branding.netlify.app/img/projects/kubernetes/icon/color/kubernetes-icon-color.svg\" height=\"16\" width=\"16\"/\u003e [What are the meanings of Audit Log in Kubernetes ?](#what-are-the-meanings-of-audit-log-in-kubernetes-)\n- \u003cimg src=\"https://cncf-branding.netlify.app/img/projects/kubernetes/icon/color/kubernetes-icon-color.svg\" height=\"16\" width=\"16\"/\u003e [How can you enable Audit Log feature in Kubernetes ?](#how-can-you-enable-audit-log-feature-in-kubernetes-)\n- \u003cimg src=\"https://cncf-branding.netlify.app/img/projects/falco/icon/color/falco-icon-color.svg\" height=\"16\" width=\"16\"/\u003e [How can you enable embedded web server available within the Falco project ?](#how-can-you-enable-embedded-web-server-available-within-the-falco-project-)\n- \u003cimg src=\"https://raw.githubusercontent.com/falcosecurity/falcosidekick/master/imgs/falcosidekick_color.svg\" height=\"16\" width=\"16\"/\u003e [What is the purpose of Falcosidekick project ?](#what-is-the-purpose-of-falcosidekick-project-)\n\n- 👨‍💻 [Hands On](#hands-on)\n- 👀 [References](#references)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n# Prerequisites\n* \u003cimg src=\"https://cncf-branding.netlify.app/img/projects/kubernetes/icon/color/kubernetes-icon-color.svg\" height=\"16\" width=\"16\"/\u003e Kubernetes Cluster v1.20.2 \n* \u003cimg src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/e/e4/Tmux_logo.svg/608px-Tmux_logo.svg.png\" height=\"16\" width=\"16\"/\u003e tmux-cssh (or another tool to connect multiple SSH servers)\n* \u003cimg src=\"https://github.com/cncf/artwork/blob/master/other/illustrations/ashley-mcnamara/kubectl/kubectl.svg\" height=\"16\" width=\"16\"/\u003e kubectl v1.20.2\n* \u003cimg src=\"https://cncf-branding.netlify.app/img/projects/helm/horizontal/color/helm-horizontal-color.svg\" height=\"16\" width=\"16\" /\u003e Helm v3.5.1\n\n# What is in it for you ?\nYou are going to learn: \n  * the meaning of audit logs in Kubernetes.\n  * how you can enable Audit log feature in Kubernetes.\n  * how you can enable embedded web server available within the Falco project.\n  * the purpose of the Falcosidekick project.\n\n# What are the meanings of Audit Log in Kubernetes ?\nKubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself.\n\nAuditing allows cluster administrators to answer the following questions:\n\n* what happened?\n* when did it happen?\n* who initiated it?\n* on what did it happen?\n* where was it observed?\n* from where was it initiated?\n* to where was it going?\n\nAudit records begin their lifecycle inside the kube-apiserver component. Each request on each stage of its execution generates an audit event, which is then pre-processed according to a certain policyand written to a backend. The policy determines what's recorded and the backends persist the records. The current backend implementations include logs files and webhooks.\n\n\u003e Credit: https://kubernetes.io/docs/tasks/debug-application-cluster/audit\n\n# How can you enable Audit Log feature in Kubernetes ?\nIn this demo, we are going to use Webhook backend mode for Audit Log to be able to send audit logs to the webserver that is available within the Falco project.\n\nThe webhook audit backend sends audit events to a remote web API, which is assumed to be a form of the Kubernetes API, including means of authentication. You can configure a webhook audit backend using the following kube-apiserver flags:\n\n* --audit-webhook-config-file specifies the path to a file with a webhook configuration. The webhook configuration is effectively a specialized kubeconfig.\n* --audit-webhook-initial-backoff specifies the amount of time to wait after the first failed request before retrying. Subsequent requests are retried with exponential backoff.\nThe webhook config file uses the kubeconfig format to specify the remote address of the service and credentials used to connect to it.\n\n\u003e Credit: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#webhook-backend\n\n# How can you enable embedded web server available within the Falco project ?\nIn this demo, we are going to using Helm to install the Falco, so, there is value that we can enableor disable to audit log feature called \"auditLog.enabled\", if we enable this, the embedded webserveris going to start within the Falco to consume audit events at port *8765* and behind the *k8s-audit* endpoint.\n\n# What is the purpose of Falcosidekick project ?\nBy default, Falco has 5 outputs for its events: stdout, file, gRPC, shell and http.Even if they're convenient, we can quickly be limited to integrating Falco with other components. Here comes falcosidekick, a little daemon that extends that number of possible outputs.\n\nThe current list of available falcosidekick outputs (version 2.13.0) is:\n\n* Slack\n* Rocketchat\n* Mattermost\n* Teams\n* Datadog\n* AlertManager\n* Elasticsearch\n* Loki\n* NATS\n* Influxdb\n* AWS Lambda\n* AWS SQS\n* SMTP (email)\n* Opsgenie\n* Webhook\n\nBeyond that, it provides metrics about the number of events and let you add custom fields in events, for example environment, region, etc\n\n# Hands On\nIn this demo, we are going provision our k3s cluster using Terraform and Ansible with HA support. To get more detail about the installation of the k3s cluster please follow the [link.](https://github.com/developer-guy/kubernetes-cluster-setup-using-terraform-and-k3s-on-digitalocean)\nFirst, we are going to install Falco and the Falcosidekick with UI enabled. In order to do that, we'll use the Helm.\n```bash\n$ helm install falco --namespace falco --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falcosidekick:2801 --set falcosidekick.enabled=true --set falcosidekick.webui.enabled=true  falcosecurity/falco --set auditLog.enabled=true --create-namespace\nNAME: falco\nLAST DEPLOYED: Sun Feb 21 23:53:05 2021\nNAMESPACE: falco\nSTATUS: deployed\nREVISION: 1\nNOTES:\nFalco agents are spinning up on each node in your cluster. After a few\nseconds, they are going to start monitoring your containers looking for\nsecurity issues.\n\n\nNo further action should be required.\n```\nLets verify the installation.\n```bash\n$ kubectl --namespace=falco get pods\nFound existing alias for \"kubectl\". You should use: \"k\"\nNAME                                      READY   STATUS    RESTARTS   AGE\nfalco-falcosidekick-ui-5d59c77944-7htbc   1/1     Running   0          44s\nfalco-ckgqp                               1/1     Running   0          45s\nfalco-falcosidekick-6c5df8d54-6rb5h       1/1     Running   0          44s\nfalco-74vkg                               1/1     Running   0          45s\nfalco-falcosidekick-6c5df8d54-7tprb       1/1     Running   0          44s\nfalco-tc78l                               1/1     Running   0          45s\nfalco-48vk2                               1/1     Running   0          45s\nfalco-blkzl                               1/1     Running   0          45s\n```\n\nThen, we should expose embedded webserver to be able send logs to it from the master nodes.\n```bash\n$ kubectl patch svc falco --namespace=falco --type='json' -p '[{\"op\":\"replace\",\"path\":\"/spec/type\",\"value\":\"NodePort\"}]'\nFound existing alias for \"kubectl\". You should use: \"k\"\nservice/falco patched\n```\n\nGet the NodePort ip of the Falco service to reach embedded server.\n```bash\n$  kubectl --namespace=falco get services                                                                                Found existing alias for \"kubectl\". You should use: \"k\"\nNAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE\nfalco-falcosidekick-ui   ClusterIP   10.43.221.136   \u003cnone\u003e        2802/TCP         2m52s\nfalco-falcosidekick      ClusterIP   10.43.179.95    \u003cnone\u003e        2801/TCP         2m52s\nfalco                    NodePort    10.43.114.83    \u003cnone\u003e        8765:30958/TCP   2m52s \n```\n\nAfter you set up the k3s cluster, lets enable Audit Log feature on the k3s master nodes. In order to do that, we'll use tmux-cssh to connect to the VMs. Copy your VM'S IP addresses first then run the following command:\n```bash\n$ tmux-cssh -o '-i ~/.ssh/digitalocean_rsa' root@161.35.209.51 root@167.71.44.217  # \u003c-- This command will open a new tmux session and divide panes for each of the VM and syncronize commnds between them.\n```\n\nAfter you run the command you should see similar screen below:\n\n\u003cimg src=\"./assets/tmux_cssh_vm.png\" widht=\"600\" height=\"600\"/\u003e\n\nAfter that, you should run the following commands to enable Audit Log, once you type command in one screen, it will appear in the other pane immediately.\n```bash\n$ sudo mkdir -p /var/lib/rancher/audit\n$ wget https://raw.githubusercontent.com/falcosecurity/evolution/master/examples/k8s_audit_config/audit-policy.yaml\n$ sudo cp audit-policy.yaml /var/lib/rancher/audit/\n$ export IP=161.35.217.182\n$ cat \u003c\u003c EOF | sudo tee /var/lib/rancher/audit/webhook-config.yaml\napiVersion: v1\nkind: Config\nclusters:\n- name: falco\n  cluster:\n    server: http://$IP:30958/k8s-audit\ncontexts:\n- context:\n    cluster: falco\n    user: \"\"\n  name: default-context\ncurrent-context: default-context\npreferences: {}\nusers: []\nEOF\n```\nOnce you prepare the the files, we should edit our k3s.service in the VMs.\n```bash\n$ vim /etc/systemd/system/k3s.service\n# add these lines to after the k3s server command.\n# --kube-apiserver-arg=audit-log-path=/var/lib/rancher/audit/audit.log\n# --kube-apiserver-arg=audit-policy-file=/var/lib/rancher/audit/audit-policy.yaml\n# --kube-apiserver-arg=audit-webhook-config-file=/var/lib/rancher/audit/webhook-config.yaml\n```\n\nThis Unit file should looks like the following above:\n```Unit\n[Unit]\nDescription=Lightweight Kubernetes\nDocumentation=https://k3s.io\nWants=network-online.target\nAfter=network-online.target\n\n[Service]\nType=notify\nExecStartPre=-/sbin/modprobe br_netfilter\nExecStartPre=-/sbin/modprobe overlay\nExecStart=/usr/local/bin/k3s server --kube-apiserver-arg=audit-log-path=/var/lib/rancher/audit/audit.log --kube-apiserver-arg=audit-policy-file=/var/lib/rancher/audit/audit-policy.yaml --kube-apiserver-arg=audit-webhook-config-file=/var/lib/rancher/audit/webhook-config.yaml --config /etc/rancher/k3s/config.yaml\nKillMode=process\nDelegate=yes\nLimitNOFILE=1048576\nLimitNPROC=infinity\nLimitCORE=infinity\nTasksMax=infinity\nTimeoutStartSec=0\nRestart=always\nRestartSec=5s\n\n[Install]\nWantedBy=multi-user.target\n```\n\nAfter you edit the Unit file, you should reload the daemon and the k3s service.\n```bash\n$ sudo systemctl daemon-reload \u0026\u0026 \\\n sudo systemctl restart k3s\n```\n\nIf everything goes fine, you should see similar output like the following image below:\n\n\u003cimg src=\"./assets/everything_should_fine.png\" width=\"850\" height=\"600\"/\u003e\n\nIn order to test this scenario, we need to create an event that Falco provides alert for it.\n```bash\n$ kubectl create configmap aws-creds \\\n  --from-literal aws_access_key_id=AKES20LNOA\n```\n\nOnce you created this, we should see the alert at the Falcosidekick UI.\n```bash\n$ kubectl --namespace=falco port-forward svc/falco-falcosidekick-ui 2802\nFound existing alias for \"kubectl\". You should use: \"k\"\nForwarding from 127.0.0.1:2802 -\u003e 2802\nForwarding from [::1]:2802 -\u003e 2802\n```\n\nOpen the browser, go to the localhost:2802, you should see similar screen following below:\n![falco_sidekick_ui](./assets/falcosidekick_ui.png)\n\n# References\n* https://blog.alexellis.io/falco-at-the-edge-arm64/\n* https://falco.org/blog/extend-falco-outputs-with-falcosidekick/\n* https://sysdig.com/blog/kubernetes-audit-log-falco\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeveloper-guy%2Ffalco-analyze-audit-log-from-k3s-cluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeveloper-guy%2Ffalco-analyze-audit-log-from-k3s-cluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeveloper-guy%2Ffalco-analyze-audit-log-from-k3s-cluster/lists"}