{"id":19883668,"url":"https://github.com/muchdogesec/cpe2stix","last_synced_at":"2025-09-18T20:31:15.359Z","repository":{"id":252807466,"uuid":"809640625","full_name":"muchdogesec/cpe2stix","owner":"muchdogesec","description":"A command line tool that turns NVD CPE records into STIX 2.1 Objects.","archived":false,"fork":false,"pushed_at":"2024-11-26T15:25:08.000Z","size":121,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-26T16:29:18.518Z","etag":null,"topics":["cpe","cve","nvd","stix2"],"latest_commit_sha":null,"homepage":"https://www.dogesec.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/muchdogesec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-03T07:01:47.000Z","updated_at":"2024-10-28T09:53:49.000Z","dependencies_parsed_at":"2024-08-17T08:29:59.076Z","dependency_job_id":"d83b6647-bff0-45cb-8532-7338475ca3be","html_url":"https://github.com/muchdogesec/cpe2stix","commit_stats":null,"previous_names":["muchdogesec/cpe2stix"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muchdogesec%2Fcpe2stix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muchdogesec%2Fcpe2stix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muchdogesec%2Fcpe2stix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muchdogesec%2Fcpe2stix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/muchdogesec","download_url":"https://codeload.github.com/muchdogesec/cpe2stix/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233518782,"owners_count":18688320,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpe","cve","nvd","stix2"],"created_at":"2024-11-12T17:21:55.346Z","updated_at":"2025-09-18T20:31:09.942Z","avatar_url":"https://github.com/muchdogesec.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cpe2stix\n\n## Before you begin\n\nWe host a full web API that includes all objects created by cpe2stix, [Vulmatch](https://www.vulmatch.com/).\n\n## Overview\n\n![](docs/cpe2stix.png)\n\nA command line tool that turns NVD CPE records into STIX 2.1 Objects.\n\nHaving a standardised way of to describe CPEs becomes very useful when managing software tools you're using. That is where Common Platform Enumerations (CPEs) come in;\n\n\u003e CPE is a structured naming scheme for information technology systems, software, and packages. Based upon the generic syntax for Uniform Resource Identifiers (URI), CPE includes a formal name format, a method for checking names against a system, and a description format for binding text and tests to a name.\n\nWe had a requirement to have an up-to-date copy of NVD CPEs in STIX 2.1 format.\n\nThe code in this repository turns CPEs into STIX 2.1 objects, and keeps them updated to match the official CPE dictionary;\n\n1. Downloads the current CPEs (that match a users filters) from the NVD API\n2. Converts them to STIX 2.1 Objects\n3. Stores the STIX 2.1 Objects in the file store\n4. Creates STIX Bundles of generated objects for each update run\n\n## tl;dr\n\n[![cpe2stix](https://img.youtube.com/vi/ZIj7Wo0iELc/0.jpg)](https://www.youtube.com/watch?v=ZIj7Wo0iELc)\n\n[Watch the demo](https://www.youtube.com/watch?v=ZIj7Wo0iELc).\n\n## Install the script\n\n```shell\n# clone the latest code\ngit clone https://github.com/muchdogesec/cpe2stix\n# create a venv\ncd cpe2stix\npython3 -m venv cpe2stix-venv\nsource cpe2stix-venv/bin/activate\n# install requirements\npip3 install -r requirements.txt\n```\n\nYou will also need to have redis installed on your machine. [Instructions to do this are here](https://redis.io/docs/getting-started/installation/).\n\nIf you're on Mac, like me, the easiest way to do this is;\n\n```shell\nbrew install redis\n```\n\n### Configuration options\n\ncpe2stix has various settings that are defined in an `.env` file.\n\nTo create a template for the file:\n\n```shell\ncp .env.example .env\n```\n\nTo see more information about how to set the variables, and what they do, read the `.env.markdown` file.\n\n## Running the script\n\nThe script runs Redis and Celery jobs to download the data, you must start this first.\n\nGenerally you want to run these in a seperate terminal window but still in the a `cpe2stix-venv`.\n\n```shell\n# navigate to the root of cpe2stix install\ncd cpe2stix\n# activate venv\nsource cpe2stix-venv/bin/activate\n# restart redis\nbrew services restart redis\n# start celery\ncelery -A cpe2stix.celery worker --loglevel=info --purge\n```\n\nIf you continually run into issues, you can also use flower to monitor Celery workers for debugging. In a new terminal run;\n\n```shell \ncelery -A cpe2stix.celery flower\n```\n\nTo open the application. You can also use Docker to run flower, [as detailed here](https://flower.readthedocs.io/en/latest/install.html#usage).\n\nThe script to get CPEs can now be executed (in the second terminal window) using;\n\n```shell\npython3 cpe2stix.py\n```\n\nIt will also filter the data created using any values entered in the `.env` file on each run.\n\nOn each run, the old `stix2_objects/cpe-bundle.json` will be overwritten.\n\nWhen the data conversion is complete you must kill the celery worker before running the script again. Failure to do so will lead to issues with the bundle IDs.\n\n```shell\n^C\nworker: Hitting Ctrl+C again will terminate all running tasks!\n\nworker: Warm shutdown (MainProcess)\n```\n\nDon't forget to restart the workers again, as follows;\n\n```shell\n# start celery\ncelery -A cpe2stix.celery worker --loglevel=info --purge\n```\n\n## Mapping information\n\n### Marking Definition / Extension Definition\n\nThese are hardcoded and imported:\n\n* Marking Definition: https://raw.githubusercontent.com/muchdogesec/stix4doge/main/objects/marking-definition/cpe2stix.json\n* Extension Definition: https://raw.githubusercontent.com/muchdogesec/stix2extensions/refs/heads/main/extension-definitions/properties/software-cpe-properties.json\n\n### Software\n\ncpe2stix creates Software SCOs for CPEs as follows;\n\n```json\n{\n    \"type\": \"software\",\n    \"spec_version\": \"2.1\",\n    \"id\": \"software--\u003cGENERATED BY STIX2 LIBRARY\u003e\",\n    \"name\": \"\u003cproducts.cpe.titles.title\u003e (if multiple, where lan = en, else first result)\",\n    \"cpe\": \"\u003cproducts.cpe.cpeName\u003e\",\n    \"swid\": \"\u003cproducts.cpe.cpeNameId\u003e\",\n    \"version\": \"\u003cproducts.cpe.cpeName[version_section]\u003e\",\n    \"vendor\": \"\u003cproducts.cpe.cpeName[vendor_section]\u003e\",\n    \"languages\": [\n        \"\u003cproducts.cpe.titles.lang\u003e\"\n    ],\n    \"object_marking_refs\": [\n        \"marking-definition--94868c89-83c2-464b-929b-a1a8aa3c8487\",\n        \"\u003cIMPORTED MARKING DEFINTION OBJECT\u003e\"\n    ],\n    \"extensions\": {\n        \"extension-definition--82cad0bb-0906-5885-95cc-cafe5ee0a500\": {\n            \"extension_type\": \"toplevel-property-extension\"\n        }\n    },\n    \"x_cpe_struct\": {\n        \"cpe_version\": \"\u003cCPE_VERSION\u003e\",\n        \"part\": \"\u003cPART\u003e\",\n        \"vendor\": \"\u003cVENDOR\u003e\",\n        \"product\": \"\u003cPRODUCT\u003e\",\n        \"version\": \"\u003cVERSION\u003e\",\n        \"update\": \"\u003cUPDATE\u003e\",\n        \"edition\": \"\u003cEDITION\u003e\",\n        \"language\": \"\u003cLANGUAGE\u003e\",\n        \"sw_edition\": \"\u003cSW_EDITION\u003e\",\n        \"target_sw\": \"\u003cTARGET_SW\u003e\",\n        \"target_hw\": \"\u003cTARGET_HW\u003e\",\n        \"other\": \"\u003cOTHER\u003e\"\n    }\n}\n```\n\nNote, if the NVD API record contains the property `products.cpe.deprecated` then `[DEPRECATED]` is added to the `name` property.\n\n### Bundle\n\nAll objects will be packed into a bundle file in `stix2_objects` names `cpe-bundle.json` which has the following structure.\n\n```json\n{\n    \"type\": \"bundle\",\n    \"id\": \"bundle--\u003cUUIDV5 GENERATION LOGIC\u003e\",\n    \"objects\": [\n        \"\u003cALL STIX JSON OBJECTS\u003e\"\n    ]\n}\n```\n\nTo generate the id of the SRO, a UUIDv5 is generated using the namespace `5e6fc5ec-e507-52e7-8465-cf5ffc47138a` and an md5 hash of all the sorted objects in the bundle.\n\n### Updating STIX Objects\n\nNew CPEs are added weekly. Existing CPEs are also updated.\n\nTherefore the script can be used to keep an up-to-date copy of objects.\n\nGenerally it is assumed the script will be used like so;\n\n1. on install, a user will create a backfill of all CPEs (almost 1.2 million at the time of writing, depending on `CPE_LAST_MODIFIED_EARLIEST`/`CPE_LAST_MODIFIED_LATEST` date used)\n    * note, generally this job will be split into multiple parts, downloading one year of data at a time.\n2. said bundle(s) will be imported to some downstream tool (e.g. a threat intelligence platform)\n3. the user runs the script again, this time updating the `CPE_LAST_MODIFIED_EARLIEST` variable to match the last time script is run (so that updated bundle only captures new and update objects)\n\nThe script will store the STIX objects created in the `stix2_objects` directory. All old objects will be purged with each run.\n\n## Recommendations for backfill\n\nI STRONGLY recommend you [use cxe2stix_helper to perform the backfill](https://github.com/muchdogesec/cxe2stix_helper). cxe2stix_helper will handle the splitting of the bundle files into your desired time ranges.\n\n## Useful supporting tools\n\n* To generate STIX 2.1 Objects: [stix2 Python Lib](https://stix2.readthedocs.io/en/latest/)\n* The STIX 2.1 specification: [STIX 2.1 docs](https://docs.oasis-open.org/cti/stix/v2.1/stix-v2.1.html)\n* [NVD CPE Overview](https://nvd.nist.gov/products)\n* [NVD CVE API](https://nvd.nist.gov/developers/products)\n\n## Support\n\n[Minimal support provided via the DOGESEC community](https://community.dogesec.com/).\n\n## License\n\n[Apache 2.0](/LICENSE).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuchdogesec%2Fcpe2stix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmuchdogesec%2Fcpe2stix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuchdogesec%2Fcpe2stix/lists"}