{"id":25506480,"url":"https://github.com/blackducksoftware/bd_sig_filter","last_synced_at":"2025-12-14T18:40:40.579Z","repository":{"id":241403363,"uuid":"798746966","full_name":"blackducksoftware/bd_sig_filter","owner":"blackducksoftware","description":"BD Script to ignore components matched from Signature scan likely to be partial or invalid matches","archived":false,"fork":false,"pushed_at":"2024-06-17T13:12:50.000Z","size":82,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-06T00:35:08.499Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/blackducksoftware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-10T11:38:17.000Z","updated_at":"2024-06-17T13:12:18.000Z","dependencies_parsed_at":"2024-06-01T23:06:38.084Z","dependency_job_id":null,"html_url":"https://github.com/blackducksoftware/bd_sig_filter","commit_stats":null,"previous_names":["matthewb66/bd_sig_filter"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackducksoftware%2Fbd_sig_filter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackducksoftware%2Fbd_sig_filter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackducksoftware%2Fbd_sig_filter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/blackducksoftware%2Fbd_sig_filter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/blackducksoftware","download_url":"https://codeload.github.com/blackducksoftware/bd_sig_filter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239612003,"owners_count":19668275,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-19T06:57:59.340Z","updated_at":"2025-11-17T09:30:14.733Z","avatar_url":"https://github.com/blackducksoftware.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bd_sig_filter - v1.7\nBD Script to ignore components matched from Signature scan likely to be partial or invalid matches, and \nmark components reviewed which are definitive matches (dependency or component name and version in matched path for\nsignature matches).\n\n## PROVISION OF THIS SCRIPT\nThis script is provided under the MIT OSS license (see LICENSE file).\nIt does not represent any extension of licensed functionality of Synopsys software itself and is provided as-is, without warranty or liability.\nIf you have comments or issues, please raise a GitHub issue here. Synopsys support is not able to respond to support tickets for this OSS utility. Users of this pilot project commit to engage properly with the authors to address any identified issues.\n\n## INTRODUCTION\nBlack Duck Signature matching is a unique and powerful way to find OSS and 3rd party code within your applications and\nenvironments.\n\nSignature matching uses hierarchical folder analysis to find matches with depth, identifying the most likely components\nmatching the project by examining all files in all folders as a whole.\nMany competitive SCA solutions use individual file matching for files in the project, but this is not absolutely suitable \nto identify component matches because the majority of files in components do not change between versions, \nso multiple version matches will be identified for every file. It is therefore impossible to infer an overall component\nversion by looking at the individual files.\n\nHowever, Signature matching can still produce false positive matches, especially where template code hierarchies \nexist in custom and OSS code.\n\nFurthermore, Signature matches can be identified in folders created by Synopsys tools, or in cache/config\nlocations or test folders; these folders can be ignored at scan time, but can exist in the Black Duck project and need to \nbe removed after scan completion. Additionally, when scanning\nmodified OSS, Signature scanning can identify the same component with multiple versions from a single project\nlocation, with the need to curate the BOM to ignore duplicate components.\n\nThis script uses several techniques to examine the Signature match paths for components, searching for the component\nname and version in the path to determine matches which are likely correct and optionally marking them as reviewed.\n\nIt can also ignore components only matched from paths which should be excluded (Synopsys tools, cache/config folders \nand test folders), and components which are duplicates across versions where the version string is not found\nin the signature match path, one match is a dependency or where they are simply duplicates (same component name and version\nbut shown in the BOM as separate entries).\n\nOptions are available to enable ignore and review actions, and other features.\n\n## PREREQUISITES\nPython 3.8+ must be installed prior to using this script.\n\n## INSTALLATION\nThe package can be installed using the command:\n\n    python3 -m pip install bd-sig-filter\n\nUpgrade from a previous version using:\n\n    python3 -m pip install bd-sig-filter --upgrade\n\nAlternatively, the repository can be cloned and the script run directly using the command:\n\n    python3 bd_sig_filter/bd_sig_filter.py OPTIONS\n\n## USAGE\nIf installed as a package, run the utility using the command `bd-sig-filter`.\n\nAlternatively if you have cloned the repo, use a command similar to:\n\n    python3 bd_sig_filter/bd_sig_filter.py OPTIONS\n\nThe package can be invoked as follows:\n\n    usage: bd-sig-filter [-h] [--blackduck_url BLACKDUCK_URL] [--blackduck_api_token BLACKDUCK_API_TOKEN] [--blackduck_trust_cert] [-p PROJECT] [-v VERSION] [--debug] [--logfile LOGFILE]\n                         [--report_file REPORT_FILE] [--version_match_reqd] [--ignore] [--review] [--no_ignore_test] [--no_ignore_synopsys] [--no_ignore_defaults]\n                         [--ignore_no_path_matches]\n\n    options:\n      -h, --help            show this help message and exit \n      --blackduck_url BLACKDUCK_URL\n                            Black Duck server URL (REQUIRED)\n      --blackduck_api_token BLACKDUCK_API_TOKEN\n                            Black Duck API token (REQUIRED)\n      --blackduck_trust_cert\n                            Black Duck trust server cert\n      -p PROJECT, --project PROJECT\n                            Black Duck project to create (REQUIRED)\n      -v VERSION, --version VERSION\n                            Black Duck project version to create (REQUIRED)\n      --debug               Debug logging mode\n      --logfile LOGFILE     Logging output file\n      --report_file REPORT_FILE\n                            Report output file\n      --version_match_reqd  Component matches require version string in path\n      --ignore              Ignore components in synopsys, default or test folders and duplicates with wrong version\n      --review              Mark components reviewed\n      --no_ignore_test      Do not ignore components in test folders\n      --no_ignore_synopsys  Do not ignore components in synopsys tool folders\n      --no_ignore_defaults  Do not ignore components in default folders\n      --ignore_no_path_matches\n                            Also ignore components with no component/version match in signature path\n                            (Use with caution)\n      --report_unmatched    Report the list of components which will be left Unreviewed and why - these may need\n                            to be manually reviewed.\n\nThe minimum required options are:\n    \n    --blackduck_url https://BLACKDUCK_SERVER_URL\n    --blackduck_api_token BLACKDUCK_API_TOKEN\n    --project PROJECT\n    --version VERSION\n\nEnvironment variables BLACKDUCK_URL, BLACKDUCK_API_TOKEN and BLACKDUCK_TRUST_CERT may also be used.\n\n## SCRIPT BEHAVIOUR\nThe default behaviour of the script is to create a table of BOM components with details about what actions can be taken.\nBy default, no actions will be taken, with only the tables being created to explain what would happen if `--ignore` and `--review`\noptions were specified.\n\nAn example of the output table is shown below:\n\n    SUMMARY:\n              Components    Ignored    Reviewed    Neither\n    ------  ------------  ---------  ----------  ---------\n    Before           641          0           0        641\n    After            641         24         615          2\n    \n    Component                             Match Type    Ignored    Reviewed    To be Ignored    To be Reviewed    Action\n    ------------------------------------  ------------  ---------  ----------  ---------------  ----------------  ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n    aggs-matrix-stats/1.3.14              Dep+Sig       False      False       False            True              Mark REVIEWED - Dependency\n    aggs-matrix-stats/2.11.1              Sig           False      False       False            True              Mark REVIEWED - Compname \u0026 version in path '/Plugins/ActOnePluginInstaller/image/actone-plugins-installer 10.0.0.67/RCM_Plugins/actOne-opensearch-2.x-connector/lib/aggs-matrix-stats-client-2.11.1.jar', Match result 200\n    aircompressor/0.10                    Dep+Sig       False      False       False            True              Mark REVIEWED - Dependency\n    Amazon MSK Library for AW/2.0.2       Dep+Sig       False      False       False            True              Mark REVIEWED - Dependency\n    Apache HttpComponents Cor/5.2.4       Sig           False      False       True             False             Mark IGNORED - compname or version not found in paths \u0026 --ignore_no_path_matches set\n    WSDL4J/1.5.1                          Sig           False      False       False            False             No Action\n    Xalan Java Serializer/2.7.2           Sig           False      False       False            False             No Action - Is a duplicate of dependency 'Xalan Java Serializer/2.7.3', has different component id/version but version found in sigpaths\n    Xalan Java Serializer/2.7.3           Dep           False      False       False            True              Mark REVIEWED - Dependency\n\nNote component names are truncated at 25 characters.\n\nThe `Before` and `After` rows in the SUMMARY list the total number of components, and how many components would be ignored or\nmarked reviewed by the script (if the `--ignore` and `--review` options are supplied).\n\nThe list of components shows the name, matchtypes and current ignore/review statuses, with the future status\n(after running the script with the `--ignore` and `--review` options) in the `To Be Ignored` and `To Be Reviewed` \ncolumns with an explanation in the `Action` column.\n\nThe `Match Score` value shows the result of fuzzy match searching for component name and version strings (note that\norigin component ID is used where available as opposed to the textual name of the component). A score of 200 shows\nan exact match of both component name and version in Signature paths; a lower value shows the possibility of less\naccurate matching.\n\nOptions can be used to modify the behaviour of the script as follows:\n\n`--no_ignore_test`:\n        Stops components matched only by Signature scanning and containing test folders (test, tests,\n        testsuite or testsuites - case insensitive) being marked for ignore (which happens by default).\n\n`--no_ignore_synopsys`:\n        Stops components matched only by Signature scanning and containing Synopsys tools folders (.synopsys,\n        synopsys-detect, .coverity, synopsys-detect.\\*.jar, scan.cli.impl-standalone.jar, seeker-agent.\\*,\n        Black_Duck_Scan_Installation - case insensitive) being marked for ignore (which happens by default).\n\n`--no_ignore_defaults`:\n        Stops components matched only by Signature scanning and containing default folders (.cache, \n        .m2, .local, .config, .docker, .npm, .npmrc, .pyenv, .Trash, .git, node_modules - case insensitive)\n        being marked for ignore (which happens by default).\n\n`--version_match_required`:\n        Enforce search for component version string in signature paths for marking components reviewed\n        (Paths containing only the component name will be used for matching otherwise)\n\n`--ignore_no_path_matches`:\n        Components with no match in the signature path are left unreviewed by default, allowing\n        manual review. Use this option to ignore these components instead but use with caution\n        as it may exclude components which are legitimate (the Signature match path does not\n        have to include the component name or version).\n\n`--report_unmatched`:\n        Create a list of Signature components which will be left UNreviewed \n\nThe options `--report_file` and `--logfile` can be used to output the tabular report and logging data to\nspecified files.\n\n## PROPOSED WORKFLOW\nThe script can be used to classify Signature scan results.\n\nIt can mark components as reviewed which are either Dependencies, or which have Signature match paths containing\nthe component name (and optionally component version) and which are therefore highly likely to be correctly identified\nby Signature matching.\n\nIt can also ignore components only Signature matched within extraneous folders (for example created by Synopsys tools, \nconfig/cache folders or test folders).\n\nComponents shown with `No action` are Signature matches where the component name or version \ncould not be identified in the signature paths, so they are potential false matches and require manual review.\n\nAfter running the script and ignoring/reviewing components (using options `--ignore --review`), review the reported\nlist of components from the script focussing on those marked with `No Action`. Optionally use the option `--report_unmatched`\nto list the `No Action` components with the full list of Signature match paths to enable assessment whether they should\nbe included in the BOM.\n\nIf, after inspection, all `No Action` components can be removed from the BOM, the `--ignore_no_path_matches` option can be used to\nignore these components automatically, however this should be used with caution as these components may be valid \nand should be manually reviewed.\n\n## PROCESSING DUPLICATE COMPONENTS\nThe script processes multiple versions of the same component in the BOM in several ways as described below:\n\n### SCENARIO 1\n- Comp1 and Comp2 are different versions of the same component\n- Comp1 and Comp2 are BOTH dependencies\n\nOutcome:\n- Comp1 will be marked REVIEWED\n- Comp2 will be marked REVIEWED\n\n### SCENARIO 2\n- Comp1 and Comp2 are different versions of the same component\n- Comp1 is a dependency and Comp2 is a signature match\n- Comp2 name IS found but version string is NOT found in the Signature match paths\n\nOutcome:\n- Comp1 will be marked REVIEWED\n- Comp2 will be marked IGNORED\n\n### SCENARIO 3\n- Comp1 and Comp2 are different versions of the same component\n- Comp1 is a dependency and Comp2 is a signature match\n- Comp2 name and version strings ARE found in the Signature match paths\n\nOutcome:\n- Comp1 will be marked REVIEWED\n- Comp2 will be marked REVIEWED\n\n### SCENARIO 4\n- Comp1 and Comp2 are different versions of the same component\n- Comp1 and Comp2 are both signature matches\n- Comp1 name and version strings ARE both found in the Signature match paths\n- Comp2 name IS found but version string is NOT found in the Signature match paths\n\nOutcome:\n- Comp1 will be marked REVIEWED\n- Comp2 will be IGNORED\n\n### SCENARIO 5\n- Comp1 and Comp2 are different versions of the same component\n- Comp1 and Comp2 are both signature matches\n- Comp1 name string IS found but version string is NOT found in the Signature match paths\n- Comp2 name string IS found but version string is NOT found in the Signature match paths\n\nOutcome:\n- Comp1 will be marked REVIEWED\n- Comp2 will be left unignored and not reviewed - for manual review\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackducksoftware%2Fbd_sig_filter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblackducksoftware%2Fbd_sig_filter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackducksoftware%2Fbd_sig_filter/lists"}