{"id":13843272,"url":"https://github.com/openraven/magpie","last_synced_at":"2026-01-11T16:59:53.608Z","repository":{"id":37566688,"uuid":"339822402","full_name":"openraven/magpie","owner":"openraven","description":"A Cloud Security Posture Manager or CSPM with a focus on security analysis for the modern cloud stack and a focus on the emerging threat landscape such as cloud ransomware and supply chain attacks. ","archived":false,"fork":false,"pushed_at":"2024-08-29T19:32:51.000Z","size":9182,"stargazers_count":188,"open_issues_count":1,"forks_count":32,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-07-04T06:06:53.572Z","etag":null,"topics":["aws","cloud","cloudsecurity","cspm","gcp","security","security-audit","security-scanner","security-testing","security-tools","security-vulnerability"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openraven.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-17T18:44:49.000Z","updated_at":"2025-06-02T19:12:03.000Z","dependencies_parsed_at":"2023-11-21T16:47:10.734Z","dependency_job_id":"a2460a1c-026c-41e5-86d0-0a0ae87504dc","html_url":"https://github.com/openraven/magpie","commit_stats":null,"previous_names":[],"tags_count":63,"template":false,"template_full_name":null,"purl":"pkg:github/openraven/magpie","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openraven%2Fmagpie","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openraven%2Fmagpie/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openraven%2Fmagpie/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openraven%2Fmagpie/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openraven","download_url":"https://codeload.github.com/openraven/magpie/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openraven%2Fmagpie/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264870188,"owners_count":23676177,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloud","cloudsecurity","cspm","gcp","security","security-audit","security-scanner","security-testing","security-tools","security-vulnerability"],"created_at":"2024-08-04T17:01:58.520Z","updated_at":"2026-01-11T16:59:53.600Z","avatar_url":"https://github.com/openraven.png","language":"Java","funding_links":[],"categories":["Cloud asset inventory","安全","Java","Multi-Cloud Security"],"sub_categories":["Threat modelling"],"readme":"# Magpie\n#### [Open Raven's](https://openraven.com) Cloud Security Framework\n- [What is Magpie](#what-is-magpie)\n- [Overview](#overview)\n  * [Magpie Architecture](#magpie-architecture)\n- [Building Magpie](#building-magpie)\n  * [Clone and build Magpie](#clone-and-build-magpie)\n- [Running Magpie](#running-magpie)\n  * [Configuration](#configuration)\n    + [Overriding config.yaml](#overriding-configyaml)\n    + [Multiple Overrides](#multiple-overrides)\n  * [Running via Docker](#running-via-docker)\n  * [Plugins](#plugins)\n  * [Community Contributed Plugins](#community-contributed-plugins)\n  * [Cloud Provider Status](#cloud-provider-status)\n    + [AWS](#aws)\n      - [Per region discovery](#per-region-discovery)\n    + [GCP](#gcp)\n    + [Azure](#azure)\n  * [Saving data to PostgreSQL](#saving-data-to-postgresql)\n\n## What is Magpie?\nMagpie is a free, open-source framework and a collection of community developed plugins that can be used to build complete end-to-end security tools such as a CSPM or Cloud Security Posture Manager. The project was originally created and is maintained by Open Raven. We build commercial cloud native data security tools and in doing so have learned a great deal about how to discover AWS assets and their security settings at scale.\n\nWe also heard that many people were frustrated with their existing security tools that couldn't be extended  and couldn't work well with their other systems, so decided to create this Magpie framework and refactor and sync our core AWS commercial discovery code as the first plugin.\n\nWe plan to actively contribute additional modules to make Magpie a credible free open source alternative to commercial CSPM’s and welcome the community to join us in adding to the framework and building plugins.\n\n## Overview\n\n### Magpie Architecture\nMagpie relies on plugins for all its integration capabilities.  They are the core of the framework and key to integration\nwith both cloud providers and downstream processing and storage.\n\n*Magpie is essentially a series of layers separated by FIFOs.*\n\nDepending on the configuration, these FIFOs are either **1) Java queues** (in the default configuration) or\n**2) Kafka queues**.  Using Kafka queues allows Magpie to run in a distributed and highly scalable fashion where\neach layer may exist on separate compute instances.\n\n![Magpie Architecture](https://raw.githubusercontent.com/openraven/magpie-api/main/media/magpie_architecture.png?token=AAHX2PKUJYSKWMDS333MPSTALXTGC)\n\n## Building Magpie\n\n### Clone and build Magpie\n```shell\ngit clone git@github.com:openraven/magpie.git\ncd magpie\nmvn clean install -DskipTests \u0026\u0026 mvn --projects magpie-cli assembly:single -DskipTests\n```\n\nThe distribution zip file will be located in `magpie-cli/target/magpie-\u003cversion\u003e.zip`\n\nAlternatively you can download the latest snapshot build by going to Action-\u003e(choose latest) and click the `magpie-cli` artifact,\nwhich will download a zip distribution.  \n\nBy default Magpie has Maven building uber jars (shaded jars) to keep plugins from bringing potentially conflicting dependencies\nonto the classpath.  Releases to Maven Central do not push uber jars. To build standard (non-shaded) jars you need to negate\nthe `uberjar` profile as such:\n```\n\u003e mvn clean package -P -uberjar\n```\n\n## Running Magpie\n\n*Java 17 is a prerequisite and must be installed to run Magpie.*\n\nOut of the box Magpie supports AWS for the cloud provider and outputs discovery data to `stdout` in JSON format. The\nAWS plugin utilizes the AWS Java SDK and will search for credentials as described in [Using Credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html).\n\nAssuming you have read credentials set up, you can start discovery by running:\n```shell\n./magpie-discovery.sh\n```\nOr from the Windows CMD line:\n```\nmagpie-discovery.bat\n```\n\nFrom Windows Powershell:\n```\n./magpie-discovery.bat\n```\n\n### Configuration\nMagpie allows for complex configurations to be enabled via the YAML-based config file.  This file has 3 primary sections:\n\n- **Layers**: each of which contain 1 or more plugins and are surrounded by at least 1 FIFO\n- **FIFOs**: which are either local (in-process Java queues) or Kafka queues\n- **Plugins**: Each running plugin must be explictly.  A plugin-specific configuration object may reside in the `config` subsection.\n\n\nThe simplest Magpie configuration is shown below. This configuration enables AWS discovery with a JSON output to `stdout`. To write to a file\nsimply redirect the output:\n```shell\n./magpie.sh \u003e output.json\n```\nLog messages are printed to `stderr` and will still show up as console output.\n\nThe simplest configuration:\n```yaml\nlayers:\n  enumerate:\n    type: origin\n    queue: default\n    plugins:\n      - magpie.aws.discovery\n  output:\n    type: terminal\n    dequeue: default\n    plugins:\n      - magpie.json\nfifos:\n  default:\n    type: local\nplugins:\n  magpie.aws.discovery:\n    enabled: true\n    config:\n  magpie.json:\n    enabled: true\n    config:\n  magpie.persist:\n    enabled: false\n    config:\n      hostname: localhost\n      port: 5432\n      databaseName: db_name\n      user: postgres\n      password: postgres\n```\n\n#### Overriding config.yaml\nIt is possible to override *most* configuration values via environmental variables. This is most useful as an easy way to\nscript a Magpie instance on a one-per-aws-service basis.  To override configuration values, set an environmental variable\nnamed `MAGPIE_CONFIG` and with a specially formed JSON object or array. For example, to perform an S3 *only* scan using\nwith the default configuration:\n\n```bash\n\u003e MAGPIE_CONFIG=\"{'/plugins/magpie.aws.discovery/config/services': ['s3']}\" ./magpie.sh\n```\nThe value of `MAGPIE_CONFIG` must be a JSON object where the key is a [JSON Pointer](https://tools.ietf.org/html/rfc6901)\nand the value is legal JSON which should be inserted into the location referenced by the pointer.\n\nIn the case where multiple overrides are required you may instead use an array of the above formatted objects as such:\n```bash\n\u003e MAGPIE_CONFIG=\"[{'/plugins/magpie.aws.discovery/enabled', false }, {'/plugins/magpie.aws.discovery/config/services': ['s3']}]\" ./magpie.sh\n```\n\n#### Multiple Overrides\nIf you have multiple values to set it may be easier to set multiple override variables instead of attempting to fit it\nin a single env var.  Magpie will accept any and all environmental variables that match the regex `MAGPIE_CONFIG.*`. They\nwill be applied in Java's natural String ordering (lexicographic).  For example:\n```bash\n\u003e export MAGPIE_CONFIG_1=\"[...]\"\n\u003e export MAGPIE_CONFIG_2=\"[...]\"\n\u003e ./magpie.sh\n```\nBoth variables will be applied, if any duplicate JSON Pointers are provided the last one applied will win.\n\n\n### Running via Docker\nBuilds on the `main` branch are automatically uploaded to quay.io under the `latest` tag.  See https://quay.io/repository/openraven/magpie for\nall available tags. Once a stable public release is made this will also be available there.  The Docker image uses the\ndefault config and provides no AWS credentials.\n\n- Credentials can be added at runtime via volume mapping or passing environmental variables into the container.\n- Configuration overrides can be done via environmental variable.\n\nFor example: to pass environmental variables for both credentials and configuration:\n```bash\ndocker run -a stdout -a stderr \\\n--env MAGPIE_CONFIG=\"{'/plugins/magpie.aws.discovery/config/services': ['s3']}\" \\\n-e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \\\nquay.io/openraven/magpie:latest\n```\n\nThe two `-a` arguments map both `stdout` and `stderr` to your local terminals, so you can redirect output as you would\nwith a local Mapgie execution.\n\n### Plugins\n\n### Community Contributed Plugins\n\nIf you've written a plugin you'd like listed please create a Pull Request with it listed here.\n\n### Cloud Provider Status\n\n#### AWS\nMagpie supports AWS as a core plugin out of the box. Checked boxes are complete and available today, the unchecked are on the roadmap for completion. We have already built the code for all services in the list, but need to port them over from a previous framework.\n\n- [x] EC2\n- [x] S3\n- [x] Athena\n- [x] Batch\n- [x] Backup\n- [x] Cassandra\n- [x] Cloudfront\n- [x] Cloudsearch\n- [x] Cloudtrail\n- [x] CloudWatch\n- [x] DynamoDB\n- [x] EB\n- [x] ECS\n- [x] EFS\n- [x] EKS\n- [x] Elastic Cache\n- [x] ELB\n- [x] ELBv2\n- [x] EMR\n- [x] ESS\n- [x] FSX\n- [x] Glacier\n- [x] GuardDuty  \n- [x] IAM\n- [x] KMS\n- [x] Lakeformation\n- [x] Lambda\n- [x] Lightsail\n- [x] QLDB\n- [x] RDS\n- [x] Redshift\n- [x] Route 53\n- [x] Secrets Manager\n- [x] SecurityHub  \n- [x] SNS\n- [x] SSM\n- [x] Storage Gateway\n- [x] VPC\n\n##### Per region discovery\nBy default the Magpie AWS Plugin will run discovery in all regions.  To narrow down discovery to a subset\nof regions edit the plugins.magpie.aws.discovery.config.regions value to an array of desired region names, for example:\n\n```yaml\nplugins:\n  magpie.aws.discovery:\n    enabled: true\n    config:\n      regions:\n        - us-east-2\n        - us-east-1\n```\n\n#### GCP\nMagpie also supports GCP as a core plugin out of the box. Checked boxes are complete and available today, the unchecked are on the roadmap for completion.\n\n- [x]  AI Platform Data Labeling Service\n- [x]  Access Approval\n- [x]  AutoML\n- [x]  BigQuery\n- [ ]  BigQuery Connection API\n- [x]  BigQuery Data Transfer Service\n- [x]  BigQuery Reservation API\n- [ ]  BigQuery Storage\n- [x]  Cloud Asset Inventory\n- [x]  Cloud Bigtable\n- [x]  Cloud Billing\n- [x]  Cloud Build\n- [x]  Cloud Data Loss Prevention\n- [x]  Cloud DNS\n- [x]  Cloud Functions\n- [x]  Cloud Key Management Service\n- [x]  Cloud Logging\n- [x]  Cloud Monitoring\n- [x]  Cloud Monitoring Dashboards\n- [ ]  Cloud Natural Language API\n- [x]  Cloud OS Config\n- [ ]  Cloud OS Login API\n- [x]  Cloud Spanner\n- [x]  Cloud Scheduler\n- [x]  Cloud Storage\n- [x]  Cloud Talent Solution Job Search\n- [x]  Cloud Tasks\n- [x]  Cloud Trace\n- [x]  Cloud Translation\n- [x]  Cloud Vision\n- [x]  Compute Engine\n- [x]  Container Analysis\n- [x]  Data Catalog\n- [x]  Dataproc\n- [ ]  Datastore\n- [x]  Dialogflow\n- [x]  Error Reporting\n- [ ]  Firestore\n- [x]  Game Servers\n- [x]  Google Kubernetes Engine\n- [x]  Identity and Access Management\n- [x]  Iam\n- [x]  IoT Core\n- [ ]  Media Translation\n- [x]  Memorystore for Memcached\n- [x]  Memorystore for Redis\n- [ ]  Phishing Protection\n- [x]  Pub/Sub\n- [x]  Pub/Sub Lite\n- [ ]  Recommendations AI\n- [ ]  Recommender\n- [x]  Resource Manager\n- [x]  Secret Manager\n- [ ]  Security Command Center\n- [x]  Service Directory\n- [ ]  Speech-to-Text\n- [ ]  Text-to-Speech\n- [ ]  Video Intelligence API\n- [x]  VPC\n- [ ]  Web Risk\n- [x]  Web Security Scanner\n- [x]  reCAPTCHA Enterprise\n\n##### Authentication\nTo use this plugin save .json file with authentication key to file then set environment variable:\n```\nexport GOOGLE_APPLICATION_CREDENTIALS=PATH_TO_CREDENTIALS_JSON_FILE\n```\n\n#### Azure\nMagpie now contains experimental Azure support. The services supported are limited to:\n\n- [x] SQL Servers\n- [x] SQL Databases\n- [x] Subscriptions\n- [x] Storage Accounts\n- [x] Storage Containers\n\nWhile PG persistence works for Azure, there are no configuration rules written at this time and there is no estimate on when\nsuch CSPM rules will exist for Azure.\n\n##### Authentication\nAs with AWS and GCP, Magpie utilizes the Azure SDK to find and use credentials on the local system. Utilizing `az login` will\nsave credentials so that Magpie can use them.\n\n\n### Saving data to PostgreSQL\nBy default, Magpie prints all discovered resources to standard out in JSON format. It's possible to modify this behaviour by changing\nthe default configuration.  The Magpie Peristence plugin comes bundled with Magpie but is in the disabled state by default.  To \nenable it  look at the following lines in the config file (config.yaml). set `enabled: true` and then modify the login\ncredentials as needed. Magpie will create all required tables at startup.\n\n```yaml\nmagpie.persist:\n    enabled: false\n    config:\n        hostname: localhost\n        port: 5432\n        databaseName: db_name\n        user: postgres\n        password: postgres\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenraven%2Fmagpie","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenraven%2Fmagpie","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenraven%2Fmagpie/lists"}