{"id":20516352,"url":"https://github.com/nianticlabs/modron","last_synced_at":"2025-04-14T00:35:55.980Z","repository":{"id":71191115,"uuid":"518880090","full_name":"nianticlabs/modron","owner":"nianticlabs","description":"Modron - Cloud security compliance","archived":false,"fork":false,"pushed_at":"2024-12-11T10:42:25.000Z","size":3937,"stargazers_count":33,"open_issues_count":1,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-27T14:51:53.255Z","etag":null,"topics":["cloud","cloud-security","cloud-security-audit","security","security-automation","security-tools"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nianticlabs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-07-28T14:31:52.000Z","updated_at":"2024-11-19T13:48:11.000Z","dependencies_parsed_at":"2023-06-01T12:46:13.846Z","dependency_job_id":null,"html_url":"https://github.com/nianticlabs/modron","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fmodron","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fmodron/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fmodron/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nianticlabs%2Fmodron/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nianticlabs","download_url":"https://codeload.github.com/nianticlabs/modron/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248802635,"owners_count":21163891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud","cloud-security","cloud-security-audit","security","security-automation","security-tools"],"created_at":"2024-11-15T21:28:23.661Z","updated_at":"2025-04-14T00:35:55.973Z","avatar_url":"https://github.com/nianticlabs.png","language":"Go","readme":"# Modron - Cloud security compliance\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/pics/modron.svg\" width=\"200\" alt=\"Modron Logo\"/\u003e\n\u003c/p\u003e\n\n\u003e _We are the ultimate law. All other law is tainted when compared to us.  \n\u003e We are order. All other order disappears when held to our light.  \n\u003e We are structure. All other structure crumbles when brought against us.  \n\u003e We are perfect law._  \n\u003e   \n\u003e — A spokesmodron\n \n\u003csmall\u003eMonte Cook, Colin McComb (1997-10-28). The Great Modron March. Edited by Michele Carter. (TSR, Inc.), p. 26. ISBN\n0-7869-0648-0.\u003c/small\u003e\n\n\n## Introduction\n\nModron is a cloud security compliance tool. It is designed to help organizations manage their cloud infrastructure\nand ensure that it is compliant with their security policies.\n\nUsers can navigate the Modron UI and view their resource groups, together with the respective observations.\nResource Groups that require attention are immediately visible and users can dig deeper to assess the observations.\n\n\u003cimg src=\"docs/screenshots/home.jpg\" width=\"800\" alt=\"Resource Group Observations\"\u003e\n\nA detailed explanation of why Modron was created can be read on [the original blog post](https://nianticlabs.com/news/modron).\n\n## Problem Statement\n\nThe rise of cloud computing has sharply increased the number of resources that need to be managed in a production\nenvironment. This has increased the load on security teams. At the same time, vulnerability and compliance scanning on\nthe cloud have made little progress. The process of inventory, data collection, analysis and remediation have scaled up,\nbut did not evolve to manage the scale and diversity of cloud computing assets. Numerous security tools still assume\nthat maintaining inventory, collecting data, looking at results and fixing issues is performed by the same person. This\nleads to increased pressure on teams already overwhelmed by the size of their infrastructure.\n\nMaintaining a secure cloud infrastructure is surprisingly hard. Cloud computing came with the promise of automation and\nease of use, yet there is a lot of progress to be made on both of these fronts. Infrastructure security also suffered\nfrom the explosion of assets under management and lack of security controls on new and existing assets.\n\nModron addresses the inventory and ownership issues raising with large cloud infrastructure, as well as the scalability\nof the remediation process by resolving ownership of assets and handling communication with different asset owners.\nModron still has the security practitioners and leadership teams in mind and provides organization wide statistics about\nthe reported issues.\n\nDesigned with multi cloud and scalability in mind, Modron is based on GCP today. The model allows for writing detection\nrules once and apply them across multiple platforms.\n\n## The Modron solution\n\nWith the help of Modron, organizations can:\n- Automatically collect data from their cloud infrastructure\n- Run security rules against the collected data\n- Notify the owners of the resources that are not compliant with the security rules\n- Provide a personalized dashboard to visualize the compliance status of the organization\n- Provide engineers information on how to remediate the issues\n\n### Analyzing the Resource Group observations\n\nThrough the Modron UI, users can view a list of their resource groups. By clicking on a resource group, they can\nsee a list of observations that have been made against that resource group:\n\n\u003cimg src=\"docs/screenshots/observations.jpg\" width=\"800\" alt=\"Resource Group Observations\"\u003e\n\nThis view provides a list of observations for the resource group. Each observation has an associated \"Risk Score\" that\nis computed taking into consideration the severity of the rule that generated the observation and the environment in\nwhich the resource group is running. This allows users to prioritize their remediation efforts.\n\n### Expanding a single observation\n\nBy expanding a single observation, users can see more details about it - including remediation steps.\n\n\u003cimg src=\"docs/screenshots/single-observation.jpg\" width=\"600\" alt=\"Single Observation\"\u003e\n\nWhen available, a command is provided to enable the user to quickly remediate the issue.\n\n### Risk Score\n\nEach observation has an associated Risk Score. This score is computed based on the severity of the rule that generated the\nobservation (Severity) and the environment in which the resource group is running (Impact).\n\nRisk Scores range from *INFO* to *CRITICAL* (slightly adapted from the [CVSS v3.x Ratings](https://nvd.nist.gov/vuln-metrics/cvss)).\nBy also analyzing the impact of the observation, the risk score can be used to prioritize remediation efforts: an\nobservation in an environment containing customer data (e.g: production) will be considered more critical than the same\nfinding in an environment containing only test data (e.g: dev).\n\n\u003cimg src=\"docs/screenshots/risk-score.jpg\" width=\"600\" alt=\"Risk Score computation\"\u003e\n\nThe details on how we compute and define the Risk Score are available in the [Risk Score documentation](docs/RISK_SCORE.md).\n\n### Statistics\n\nVia the statistics page, users can view a list of all the rules that have been run against their resource groups,\ntogether with the results. Each rule can be exported to a CSV file for further analysis.\n\n\u003cimg src=\"docs/screenshots/stats.jpg\" width=\"800\" alt=\"Statistics\"\u003e\n\nBy expanding the \"List of the observations\", users can see all the matching observations for that rule, regardless of\nthe resource group they belong to.  \nThis tool can be used by security teams to understand the impact of a rule across the organization, and which \nissues to tackle first.\n\n### Notifications\n\nA remediation cannot be effective if the right people are not informed. \nModron sends notifications to the owners of the resource groups that have observations via [Nagatha](https://github.com/nianticlabs/nagatha).\n\nUsers will receive periodically a notification with the list of observations that they need to address. Nagatha\nwill take care of delivering the notification via Slack and Email:\n\n\u003ctable border=\"0\"\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\n            \u003cimg src=\"./docs/screenshots/nagatha-1.jpg\" width=\"400\" alt=\"Notification\"\u003e\n        \u003c/td\u003e\n        \u003ctd\u003e\n            \u003cimg src=\"./docs/screenshots/nagatha-2.jpg\" width=\"400\" alt=\"Notification\"\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003cp align=\"center\"\u003eEmail Notification\u003c/p\u003e\u003c/td\u003e\n        \u003ctd\u003e\u003cp align=\"center\"\u003eSlack Notification\u003c/p\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n## Taxonomy\n\n| Term           | Definition                                                                                                                                                                                                                                                               |\n|:---------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| Resource       | An entity existing in the cloud platform. A resource can be a VM instance, a service account, a kubernetes clusters, etc.                                                                                                                                                |\n| Resource group | The smallest administrative grouping of resources, usually administered by the same individual or group of individuals. On GCP this corresponds to a Project, on Azure to a Resource Group.                                                                              |\n| Rule           | The implementation of a desired state for a given resource or set of resources. A rule applies only to a predefined set of resources, compares the state of the resource with the expected state. If these states differ, the rule generates one or more *observation*s. |\n| Observation    | Instance of a difference between the state of a resource and its expected state at a given timestamp.                                                                                                                                                                    |\n| Collection     | The action of fetching all data from the cloud platforms. This data is then stored in the database, ready to run the scan.                                                                                                                                               |\n| Scan           | The action of running a set of *rules* against a set of *resource groups*. *Observations* resulting of that scan are added to the database. There is no guarantee that all observations of the same scan will have the same timestamp.                                   |\n| Nagatha        | The notification system associated with Modron. It aggregates notifications going to the same recipient over a given time frame and sends a notification to that user.                                                                                                   |\n| Notification   | An instance of a message sent to an owner of a *resource group* for a given *observation*.                                                                                                                                                                               |\n| Exception      | An owner of a *resource group* opting out of *notifications* for a specific *rule*. Exceptions *must* have an expiration date and cannot be set forever. This limitation can be bypassed by accessing the Nagaatha service directly.                                     |\n\n## Process\n\nModron follows the process of any security scanning engine:\n\n![Scanning process](images/scan_process.png)\n\nExcept that in most scanning engines, the inventory and remediation parts are left as an exercise for the user.\nIn Modron, inventory is taken care of by identifying automatically the owners of a resource group based on the people\nthat have the permission to act on it, as the remediation is largely facilitated by running the communication with the\ndifferent resource group owners.\n\n* *Collector*: The collector fetches the data from the cloud platforms. This code must be implemented for each supported\n  code platform separately. It takes care of the inventory and data collection parts of the process.\n* *Rule engine*: The rule engine runs the rules against all collected resources and generates observations.\n  Notifications are sent to Nagatha for each observation.\n* *Nagatha* receives all the notifications for all observations, aggregates, deduplicates and limits the rate of\n  notification. It also applies the exceptions provided by the user.\n\n## Architecture\n\n### GCP\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./docs/pics/architecture.svg\" width=\"500\"/\u003e\n\u003c/p\u003e\n\n### Modron\n\n```mermaid\nflowchart LR\n    User --\u003e ui\n    ui[Modron UI]\n    ui --\u003e Modron\n    subgraph Modron\n        coll[Collector]\n        acl[ACL Fetcher]\n        re[Rule Engine]\n        re --\u003e|Check rule| re\n    end\n\n    re --\u003e|Create Observation| psql\n    coll ---\u003e|fetch| gcp\n    acl ---\u003e|fetch| gcp\n    coll ---\u003e|store| psql\n    gcp[GCP APIs]\n    psql[(PSQL)]\n    re ---\u003e|Fetch resources| psql\n    re ---\u003e|Create Notification| Nagatha\n    slack(Slack)\n    email(Email)\n    Nagatha --\u003e slack\n    Nagatha --\u003e email\n```\n\n## Getting started\n\nIn order to install Modron \u0026 Nagatha, you'll need to:\n\n1. Build the modron images following [Building the images](#building-the-images) below.\n1. Create a copy of [main.tf.example](terraform/dev/main.tf.example) and edit it with your own configuration\n1. Run `tf plan --out tf.plan` in the [dev folder](terraform/dev/)\n    * This could need multiple occurrences as setting up resources on GCP takes time.\n1. Create a copy of [tf.tfvars.json.example](nagatha/terraform/tf.tfvars.json.example) and edit it with your own\n   configuration\n1. Run `tf plan --out tf.plan` in the [nagatha folder](nagatha/terraform/)\n1. Assign the permissions to the Modron runner as mentioned in [permissions](#permissions)\n\n### Building the images\n\nThis assumes you have an Artifact registry in your GCP project following this format:\n```\nus-central1-docker.pkg.dev/$PROJECT_ID/modron\n```\n\n#### Modron Backend\n\nThe Modron backend image can be built using [Cloud Build](https://cloud.google.com/build/docs):\n```bash\ngcloud \\\n  --project \"$PROJECT_ID\" \\\n  builds submit \\\n  --config cloudbuild.yaml \\\n  --substitutions=_TAG_REF_1=dev-$(date +%s),_TAG_REF_2=dev\n```\n\n#### Modron UI\n\n```bash\ngcloud \\\n  --project \"$PROJECT_ID\" \\\n  builds submit \\\n  --config cloudbuild-ui.yaml \\\n  --substitutions=_TAG_REF_1=dev-$(date +%s),_TAG_REF_2=dev\n```\n\n\n## Development of Modron\n\n### Requirements\n\nTo run this project you'll need:\n\n* Docker\n* Go\n* The Google SDK\n* npm\n* terraform\n\n### Getting started\n\n#### Generate the protobuf files\n\nWe'll use a Docker image that contains [`buf`](https://buf.build/) and some protoc plugins to generate the protobuf\nfiles.\nWe'll call this image `bufbuild` - it needs to be built only once:\n\n```bash\ndocker build -t bufbuild -f docker/Dockerfile.buf .\n```\n\nNow, this image can be used to generate the protobuf files:\n\n```bash\ndocker run -v \"$PWD:/app\" -w \"/app\" bufbuild generate\n```\n\n\u003csmall\u003e\n   We don't use the buf plugins because we might encounter some \n   \u003ca href=\"https://buf.build/docs/bsr/rate-limits\"\u003erate limits\u003c/a\u003e\n\u003c/small\u003e\n\n### Formatting\n\nYou can format your code using:\n- `gofmt -w ./`\n- `terraform fmt -recursive`\n- `eslint`\n\n#### IDE\n\nYou can configure your IDE to format Terraform code by following these guides:\n- [Use Terraform formatter on IDEA-based IDEs](https://www.jetbrains.com/help/idea/terraform.html#use-terraform-formatter)\n- [Terraform extension for VSCode - Formatting](https://marketplace.visualstudio.com/items?itemName=hashicorp.terraform#formatting)\n\n## Permissions\n\nThe Modron service is meant to work at the organization level on GCP. In order to access the data it needs to run the\nanalysis, the Modron runner service account will need the following permissions at the organization level:\n\n```plain\napikeys.keys.list\ncloudasset.assets.searchAllIamPolicies\ncompute.backendServices.list\ncompute.instances.list\ncompute.regions.list\ncompute.sslCertificates.list\ncompute.sslPolicies.list\ncompute.subnetworks.list\ncompute.targetHttpsProxies.list\ncompute.targetHttpsProxies.list\ncompute.targetSslProxies.list\ncompute.urlMaps.list\ncompute.zones.list\ncontainer.clusters.list\niam.serviceAccounts.list\niam.serviceAccountKeys.list\niam.serviceAccounts.getIamPolicy\nmonitoring.metricDescriptors.get\nmonitoring.metricDescriptors.list\nmonitoring.timeSeries.list\nresourcemanager.projects.getIamPolicy\nserviceusage.services.get\nstorage.buckets.list\nstorage.buckets.getIamPolicy\n```\n\nIt is recommended to create a custom role with these permissions. For that you can use this terraform stanza:\n\n```hcl\nresource \"google_organization_iam_custom_role\" \"modron_lister\" {\n  org_id      = var.org_id\n  role_id     = \"ModronSecurityLister\"\n  title       = \"Modron Security Lister\"\n  description = \"All list permissions for Modron the security compliance scanner\"\n  permissions = [\n    \"apikeys.keys.list\",\n    \"cloudasset.assets.searchAllIamPolicies\",\n    \"compute.backendServices.list\",\n    \"compute.instances.list\",\n    \"compute.regions.list\",\n    \"compute.sslCertificates.list\",\n    \"compute.sslPolicies.list\",\n    \"compute.subnetworks.list\",\n    \"compute.targetHttpsProxies.list\",\n    \"compute.targetHttpsProxies.list\",\n    \"compute.targetSslProxies.list\",\n    \"compute.urlMaps.list\",\n    \"compute.zones.list\",\n    \"container.clusters.list\",\n    \"iam.serviceAccounts.list\",\n    \"iam.serviceAccountKeys.list\",\n    \"monitoring.metricDescriptors.get\",\n    \"monitoring.metricDescriptors.list\",\n    \"monitoring.timeSeries.list\",\n    \"resourcemanager.projects.getIamPolicy\",\n    \"serviceusage.services.get\",\n    \"storage.buckets.list\",\n    \"storage.buckets.getIamPolicy\",\n  ]\n}\n```\n\n## Debug\n\n### GoSec\n\nRun gosec as run by gitlab:\n\n```\ndocker run -i --tty --rm --volume \"$PWD\":/tmp/app --env CI_PROJECT_DIR=/tmp/app registry.gitlab.com/security-products/gosec:3 /analyzer run\n```\n\n## Testing\n\n### Unit test\n\nThere are quite a few unit test on Modron. If you want to run the go tests:\n\n```\ncd src/\ngo test ./... --short\n```\n\n### Integration test\n\nTo run the integration test, you'll need a self signed certificate for the notification service.\n\n```\nopenssl req -x509 -newkey rsa:4096 -keyout key.pem -nodes -out cert.pem -sha256 -days 365 -subj '/CN=modron_test' -addext \"subjectAltName = DNS:modron_test\"\ndocker compose up --build --exit-code-from \"modron_test\" --abort-on-container-exit\n```\n\n### UI Integration test\n\n```\ndocker compose -f docker-compose.ui.yaml up --build --exit-code-from \"modron_test\" --abort-on-container-exit\n```\n\n### Running locally\n\n#### Log in to GCP\n\nIn order to use the Google Cloud APIs, you need to log in to GCP as if you were using a service account:\n\n```bash\ngcloud auth application-default login\n```\n\nCheck out the [`gcloud` docs](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login) for more\ninformation.\nIf you don't log in using the above command, the collector might fail with an error similar to:\n\n```plain\n\"invalid_grant\" \"reauth related error (invalid_rapt)\" \"https://support.google.com/a/answer/9368756\"  \n```\n\n#### Start the docker-compose stack\n\nUse this docker command to spin up a local deployment via docker-compose (will rebuild on every run):\n\n```\ndocker compose -f docker-compose.ui.yaml up --build\n```\n\nIn case you want to clean up all the created images, services and volumes (e.g. if you suspect a caching issue or if a\nservice does not properly shut down):\n\n```\ndocker compose rm -fsv # remove all images, services and volumes if needed\n```\n\n#### Use Docker by itself\n\nAs an alternative you can use the following docker command to run Modron locally (against a dev project):\n\n```\nchmod 644 ~/.config/gcloud/application_default_credentials.json\ndocker build -f Dockerfile.db -t modron-db:latest .\ndocker run -e POSTGRES_PASSWORD=\"docker-test-password\" -e POSTGRES_USER=\"modron\" -e POSTGRES_DB=\"modron\" -e PG_DATA=\"tmp_data/\" -t postgres:latest -p 5432\nGOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/application_default_credentials.json PORT=\"8080\" RUN_AUTOMATED_SCANS=\"false\" ORG_SUFFIX=\"@nianticlabs.com\" STORAGE=\"SQL\" DB_MAX_CONNECTIONS=\"1\" SQL_BACKEND_DRIVER=\"postgres\" SQL_CONNECT_STRING=\"host=localhost port=5432 user=modron password=docker-test-password database=modron sslmode=disable\" go run .\n```\n\n## Telemetry\n\nModron supports [OpenTelemetry](https://opentelemetry.io/docs/) and expects a GRPC OTEL collector to be running\nalongside the deployment. We currently export traces and metrics through this collector.\n\nThe collector (`otel-collector`) can be configured to forward the telemetry data to other exporters - by default\nthese are Google Cloud Monitoring for the production environment and Prometheus / Jaeger for the local deployment.\n\n### Checking the telemetry data locally\n\nWhen running Modron locally, we suggest to start the auxiliary services by running:\n\n```bash\ndocker-compose -f docker-compose.dev.yaml up -d\n```\n\nThis will start everything you need to get started to develop locally for Modron:\n\n- [Jaeger](http://127.0.0.1:16686/)\n- [Prometheus](http://127.0.0.1:9090/)\n- `otel-collector` running on `127.0.0.1:4317` (GRPC)\n- PostgreSQL running on `127.0.0.1:5432`\n\n## Future developments\n\n* Provide an historical view of the reported issues.\n* Support AWS for analysis\n* Support Azure for analysis\n* Hands-off installation process\n\n## Security\n\nReport any security issue to [security@nianticlabs.com](mailto:security@nianticlabs.com). \n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnianticlabs%2Fmodron","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnianticlabs%2Fmodron","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnianticlabs%2Fmodron/lists"}