{"id":13600604,"url":"https://github.com/controlplaneio/collie","last_synced_at":"2025-05-12T08:31:51.232Z","repository":{"id":157996271,"uuid":"630060138","full_name":"controlplaneio/collie","owner":"controlplaneio","description":"OSCAL and Kyverno Policy Demo for AWS","archived":false,"fork":false,"pushed_at":"2023-08-16T08:37:06.000Z","size":2632,"stargazers_count":12,"open_issues_count":2,"forks_count":1,"subscribers_count":18,"default_branch":"main","last_synced_at":"2024-11-07T03:41:51.397Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/controlplaneio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-19T15:25:54.000Z","updated_at":"2024-11-01T17:47:28.000Z","dependencies_parsed_at":"2024-01-14T06:04:52.628Z","dependency_job_id":"bf0d56aa-5132-4354-a305-3e501768581a","html_url":"https://github.com/controlplaneio/collie","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/controlplaneio%2Fcollie","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/controlplaneio%2Fcollie/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/controlplaneio%2Fcollie/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/controlplaneio%2Fcollie/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/controlplaneio","download_url":"https://codeload.github.com/controlplaneio/collie/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225130752,"owners_count":17425506,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T18:00:44.719Z","updated_at":"2024-11-18T04:47:41.252Z","avatar_url":"https://github.com/controlplaneio.png","language":"Shell","funding_links":[],"categories":["Tools"],"sub_categories":[],"readme":"# Collie\n\n`Collie`: Toolkit for securing cloud controller provisioned infrastructure and demonstrating compliance\n\n\u003cdiv style=\"text-align:center\"\u003e\u003cimg src=\"docs/assets/collie_logo.png\" width=\"60%\"/\u003e\u003c/div\u003e\n\nCollie is a POC project demonstrating how infrastructure provisioned by cloud infrastructure controllers can be simultaneously secured and validated for compliance. It provides NIST 800-53 rev.5 aligned libraries of Kyverno Policy to secure Infrastructure provisioned by [Crossplane](https://www.crossplane.io/) [Community Provider](https://marketplace.upbound.io/providers/crossplane-contrib/provider-aws/), generated from OSCAL documents, and leverages Lula to use the same OSCAL documents to validate compliance.\n\n* [Collie](#collie)\n  * [Why?](#why)\n* [Prerequisites](#prerequisites)\n* [AWS Usage](#aws-usage)\n  * [Bootstrapping a cluster](#bootstrapping-a-cluster)\n\n\n## Why?\n\nOrganisations moving into cloud, often rely on using Infrastructure as Code templates with Policy enforcement for provisioning secure infrastructure, and rightly so. However, Kubernetes itself provisions infrastructure natively, through Load Balancer Services, or through third party cloud controllers, such as Crossplane. \n\nThis provides a challenge in large regulated organisations, which invest heavily in IAC and Policy patterns, forming an integral part of their enterprise security architecture. This inertia and lack of an equivalent pattern for infrastructure provisioned by cloud controllers will impede the adoption of these technologies, despite the benefits of \n* developer simplicity via consumption of a single K8s deployment pipeline for apps and infrastructure \n* gitops enablement for infrastructure    \nand\n* drift protection, as cloud controllers continuously reconcile\n\nThe intention of Collie is to create confidence in a pattern where k8s provisioned infrastructure is secured via policy engines acting as Kubernetes Admission controllers, such as Kyverno, and can be simultaneously validated for compliance using OSCAL documents and Lula.\n\nFor a more in depth discussion around the motivation for this and demonstrating it in action you can watch Andy presenting at Cloud Native SecurityCon 2023:   \n   \n[![SecurityCon Talk](https://img.youtube.com/vi/cvoWlwftbEE/0.jpg)](https://www.youtube.com/watch?v=cvoWlwftbEE)\n\n\n## Prerequisites\n\nAn installation of [lula](https://github.com/defenseunicorns/lula) is required for compliance validation and policy generation. For convenience, its installed as a submodule within [Tools](./Tools). Install lula by running `make build-lula` from the root folder.\n\nKyverno also needs to be installed (Not via Kubectl plugin). Follow [Kyverno Installation Instructions](https://kyverno.io/docs/kyverno-cli/#manual-binary-installation)\n\n## AWS Usage\n\n### Bootstrapping a Cluster\n\n\u003e ❗️Note that this cluster is not hardened and is for experimentation/test purposes only ❗️\n\n\u003e ❗ The cluster API will be exposed publicly but will be restricted so it can only be accessed from the IP address you are running the tests from. This can cause issues if your IP address changes, e.g. you move to a new network. If this happens you will have to manually add your new IP from the AWS console. ❗\n\nNavigate to [AWS/Bootstrap](https://github.com/controlplaneio/collie/tree/main/AWS/Bootstrap)\n\nTo create a cluster first run `init.sh` to pull the terraform dependencies for both stages.\n\nWithin [provider](./AWS/Bootstrap/provider), [cluster](./AWS/Bootstrap/cluster) and [demoResources](./AWS/Bootstrap/demoResources/) you need to define the variables (via tfvars) `project_name`. Optionally you can also define `aws_region` and `aws_profile`\n\nThen to bring up the cluster run `up.sh`. Once this is complete you should have a cluster with Kyverno, Crossplane and the Crossplane Community Provider ready to go.\n\n### Applying Policies\n\nPolicies are available within the [AWS/S3/Policies](https://github.com/controlplaneio/collie/tree/main/AWS/S3/Policies) and [AWS/RDS/Policies](https://github.com/controlplaneio/collie/tree/main/AWS/RDS/Policies) folders\n\nThey are set with `validationFailureAction: enforce` and can be applied to the cluster to block non-compliant resources.\n\n### Creating Resources\n\nResources are available within [AWS/S3/resourceTemplates](https://github.com/controlplaneio/collie/tree/main/AWS/S3/resourceTemplates) and [AWS/RDS/resourceTemplate](https://github.com/controlplaneio/collie/tree/main/AWS/RDS/resourceTemplates) folders\n\nThey leverage helm, set the secrets values file as required, prior to installation.\n\nTo install individual resources use\n`helm template -f values.secret.yaml -s templates/\u003cResource\u003e.yaml . | kubectl apply -f -`\n\n### Assessing for Compliance\n\nLula can be run against the cluster to assess for compliance, using the component definitions as a source of truth. run `\u003cpath/to\u003e/lula validate \u003coscal-component-definition.yaml\u003e`\n\n### Creating new Policies\n\n1. Edit the relevant OSCAL component definition file e.g [AWS/S3/S3-component-definition.yaml](https://github.com/controlplaneio/collie/blob/main/AWS/S3/S3-component-definition.yaml)\n2. Run `make generate-policies` from the root directory. Policies will be written to the Policies folder for each collection (e.g. S3, RDS, etc)\n3. Update the tests by creating the resources required within the `resourceTemplates` folder and updating `kyverno-test.yaml`\n\n### Unit Testing\n\nTests are defined within kyverno-test.yaml with resources tested defined within resourceTemplates for each service\n\nTo run the tests define `values.secret.yaml` in each resourceTemplates folder and run `make unit-test`\n\n### Linting\n\nTo ensure code quality code will be linted ot make sure all files follow a consistent style and catch any potential errors.\n\nCurrently there are linters for:\n\n* Yaml ([yamllint](https://github.com/adrienverge/yamllint))\n* Helm ([Helm CLI](https://helm.sh/))\n* Terraform ([Terraform CLI](https://www.terraform.io/), [tflint](https://github.com/terraform-linters/tflint))\n\nThese will be run as part of CI, it is recommended to install these on your system and check linting before contributing code.\n\n### End to End testing\n\nEnd to end testing involves deploying a k8s cluster with crossplane and kyverno installed then deploying the kyverno policies and then trying to create infrastructure and asserting that the resources are appropriately allowed or denied by kyverno.\n\n\u003e ❗️Note that these tests are creating real resources in AWS. This may take a while, as some resources like RDS can take 10-15 minutes to provision, this means it can appear that the tests are hanging but it may take 20 mins for them to complete. Since the resources are actually provisioned this may incur costs.❗️\n\n\u003e ❗️ These tests can sometimes be a little brittle, normally rerunning will fix it. We are targeting to make them less brittle by the 1st of May❗️\n\nThe tests are run using [bats](https://bats-core.readthedocs.io/en/stable/index.html). The bats tooling is included as submodules so make sure to run `git submodule update --init --recursive` to pull the submodules.\nTests are run concurrently which requires [GNU Parallel](https://www.gnu.org/software/parallel/) to be installed on your system as well.\n\nYou will also need to be set up to bootstrap a cluster, this requires [terraform](https://www.terraform.io/) to be installed, any tfvars described in the bootstrapping section of this readme, the terraform dependencies are installed (this can be done with the `init.sh` script), and you need credentials for connecting to the cloud provider e.g. AWS or GCP.\nAlso make sure the correct profile is configured for AWS in a `tfvars` file in *each* folder, i.e. `cluster`, `demoResources` and `provider`.\n\nSome of the tests require [yq](https://github.com/mikefarah/yq) so this should be installed.\n\nThe policies and test resources will be regenerated before running the tests. To generate the test resources you need [helm](https://helm.sh/) installed. To generate the policies you need to build [lula](https://github.com/defenseunicorns/lula) which requires [go](https://go.dev/) to be installed. The Lula repository is included as a submodule so you don't need to clone it.\n\nOnce all this is set up you can run the tests by running `make bats-test`. This will provision the cluster and also tear it down when the tests are done. If you are going to be running the tests multiple times you can set the environment variable `PRESERVE_CLUSTER` to true to skip deleting the cluster at the end as creating a cluster can take over 10 minutes. To skip refreshing the terraform state each time as well you can set `SKIP_TERRAFORM` to `true` as well.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontrolplaneio%2Fcollie","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcontrolplaneio%2Fcollie","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontrolplaneio%2Fcollie/lists"}