{"id":36824530,"url":"https://github.com/scanoss/minr","last_synced_at":"2026-01-12T14:03:25.338Z","repository":{"id":41845369,"uuid":"279114441","full_name":"scanoss/minr","owner":"scanoss","description":"SCANOSS Mining tool","archived":false,"fork":false,"pushed_at":"2024-12-03T11:48:31.000Z","size":23470,"stargazers_count":23,"open_issues_count":0,"forks_count":11,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-08-27T17:08:19.735Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scanoss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"licenses/AFL-2.0/01.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-12T17:29:45.000Z","updated_at":"2025-04-18T07:36:08.000Z","dependencies_parsed_at":"2023-12-03T21:24:16.664Z","dependency_job_id":"60de6e2d-6c5c-4021-b169-d8be73a95f57","html_url":"https://github.com/scanoss/minr","commit_stats":null,"previous_names":[],"tags_count":55,"template":false,"template_full_name":null,"purl":"pkg:github/scanoss/minr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scanoss%2Fminr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scanoss%2Fminr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scanoss%2Fminr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scanoss%2Fminr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scanoss","download_url":"https://codeload.github.com/scanoss/minr/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scanoss%2Fminr/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28340243,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T12:22:26.515Z","status":"ssl_error","status_checked_at":"2026-01-12T12:22:10.856Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-12T14:03:25.212Z","updated_at":"2026-01-12T14:03:25.308Z","avatar_url":"https://github.com/scanoss.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# minr\n\nMinr is a multi-purpose command line tool used for data mining. Minr downloads, indexes and imports source code metadata into the knowledge base. Minr can be also used locally for detecting licenses and cryptographic algorithms usage.\n\n# Setup \nComponent mining requires a Knowledge database installed. Scanoss use the SCANOSS LDB (Linked-list database) as a shared library. LDB Source code and installation guide can be found on https://github.com/scanoss/ldb \n\n# System dependencies\n- coreutils\n- gzip\n- tar\n- 7z\n- unrar\n- unzip\n- xz-utils\n- ruby\n- curl\n\nOnly for scancode mode:\n- jq\n- scancode\n\n# Build and Install\n\n```\nwget -O minr.zip https://github.com/scanoss/minr/archive/refs/heads/master.zip\nunzip minr.zip\ncd minr-master\nmake all\nsudo make install\ncd ..\n```\nYou also need to create the required ldb tables/dirs:\n\n```\nsudo mkdir -p /var/lib/ldb/oss/{purl,url,file,wfp}\nsudo chown -R user.user /var/lib/ldb/  # replace 'user' with your current user\nsudo chmod -R 755 /var/lib/ldb/\n```\n\n# Usage (downloading)\n\nMinr takes two arguments: \n\n* `-d`: Target metadata (CSV) \n* `-u`: Target URL (pointing to a downloadable archive or a local folder)\n\nUsage example: \n\n```\n$ minr -d madler,pigz,2.4,20171227,zlib,pkg:github/madler/pigz -u https://github.com/madler/pigz/archive/v2.4.zip \n```\n\nThe target metadata is a comma delimited list of the following fields: \n\n* Vendor \n* Component \n* Version \n* Release date\n\n* License\n* Purl (see https://github.com/package-url/purl-spec)\n\n\nNote: Commas are not admitted in any of these fields.  \n\n# Local mining\nMinr can also be used to search for licenses, quality code, copyrights and cryptographic algorithms from a local directory.\nLicense mining is available using -L option\n\nUsage example: \n\n```\n$ minr -L /home/johndoe/os-projects\n```\n\nMines the **os-projects** directory (and recursively, its subfolders) searching for licenses declaration. The output is presented at the standard output.\nAvailable options are\n* **-L** Scans for licenses\n* **-C** Scans for copyrights\n* **-Q** Scans for code quality\n* **-Y** Scans for cryptographic algorithms\n\nFor the given example using the **-Y** option, the output is like:\n```\n/home/johndoe/os-projects/myPersonalWallet/main.c,SHA,128\n/home/johndoe/os-projects/fileMgmt/fileordering/unique.c,MD5,128\n/home/johndoe/os-projects/fileMgmt/fileochecking/mycrc32.c,CRC32,128\n```\n\nWhere each algorithm is reported once for each file. The first data of each row is the file path and its name, the second is the algorithm name and the last is the coding strenght for that algorithm.\n\n\n## File Discrimination \n\nThe following criteria will be use to discriminate files: \n\n* File is not a physical file (no symlinks, etc) \n* File extension is ignored\n* File size is below a predefined limit\n* File header has unwanted contents \n\n# Creating an Open Source Knowledge Base\n\nThe following examples show the entire process for downloading an OSS component, importing it into the Knowledge Base and performing a scan against it using the SCANOSS Open Source Inventory Engine:\n\n\n## URL mining\n\nURL mining is the process of downloading a component, expanding the files and saving component, metadata and original sources for snippet mining.\n\n```\n$ minr -d scanoss,webhook,1.0,20200320,BSD-3-Clause,pkg:github/scanoss/webhook -u https://github.com/scanoss/webhook/archive/1.0.tar.gz\nDownloading https://github.com/scanoss/webhook/archive/1.0.tar.gz\n$\n```\n\nA `mined/` directory is created with the downloaded metadata. This includes component and file metadata and source code archives which are kept using the `.mz` archives, specifically designed for this purpose. \n\n## Snippet mining\n\nSnippet mining is a second step, where snippet information is mined from the `.mz` archives. This is achieved with the `-z` parameter.\n\n```\n$ minr -z mined\nmined/sources/146a.mz\n...\n$\n```\n\nA `mined/snippets` directory is created with the snippet wfp fingerprints extracted from the `.mz` files located in `mined/sources`\n\n## Data importation into the LDB\nTo be able to import data into the LDB the version.json file must be present inside the mined directory. This file provide the last update date and will be imported join to the mined tables.\nThe \"version.json file must have the following format:\n\n```\n{\"monthly\":\"YY.MM\", \"daily\":\"YY.MM.DD\"}\n```\n\nThis file can be generated with this command in bash:\n```\necho '{\"monthly\":\"22.01\", \"daily\":\"22.01.24\"}' \u003e mined/version.json\n```\nThen the importation can be performed:\n```\n$ minr -i mined/\n$\n```\n\nThe LDB is now loaded with the component information and a scan can be performed.\n\n## Scanning against the LDB Knowledge Base\n\nThe following example shows an entire component match:\n\n```\n$ scanoss 0.8.2.zip\n{\n  \"0.8.2.zip\": [\n    {\n      \"id\": \"file\",\n      \"status\": \"pending\",\n      \"lines\": \"all\",\n      \"oss_lines\": \"all\",\n      \"matched\": \"100%\",\n      \"purl\": [\n        \"pkg:github/scanoss/webhook\"\n      ],\n      \"vendor\": \"scanoss\",\n      \"component\": \"webhook\",\n      \"version\": \"1.0\",\n      \"latest\": \"1.0\",\n      \"url\": \"https://github.com/scanoss/webhook\",\n      \"release_date\": \"20200320\",\n      \"file\": \"1.0.tar.gz\",\n      \"url_hash\": \"611e5c3a58a3c2b78385556368c5230e\",\n      \"file_hash\": \"611e5c3a58a3c2b78385556368c5230e\",\n      \"file_url\": \"https://github.com/scanoss/webhook/archive/1.0.tar.gz\",\n      \"dependencies\": [],\n      \"licenses\": [\n        {\n          \"name\": \"BSD-3-Clause\",\n          \"obligations\": \"https://www.osadl.org/fileadmin/checklists/unreflicenses/BSD-3-Clause.txt\",\n          \"copyleft\": \"no\",\n          \"patent_hints\": \"no\",\n          \"source\": \"component_declared\"\n        }\n      ],\n      \"copyrights\": [\n        {\n          \"name\": \"Copyright (C) 2017-2020; SCANOSS Ltd. All rights reserved.\",\n          \"source\": \"license_file\"\n        }\n      ],\n      \"vulnerabilities\": [],\n      \"quality\": [],\n      \"cryptography\": [],\n      \"server\": {\n        \"hostname\": \"localhost\",\n        \"version\": \"4.2.4\",\n        \"flags\": \"0\",\n        \"elapsed\": \"0.041169s\"\n      }\n    }\n  ]\n}\n\n$\n```\n\nThe following example shows an entire file match:\n\n```\n$ scanoss test.c\n{\n  \"test.c\": [\n    {\n      \"id\": \"file\",\n      \"status\": \"pending\",\n      \"lines\": \"all\",\n      \"oss_lines\": \"all\",\n      \"matched\": \"100%\",\n      \"purl\": [\n\n        \"pkg:github/scanoss/webhook\"\n      ],\n      \"vendor\": \"scanoss\",\n      \"component\": \"webhook\",\n      \"version\": \"1.0\",\n      \"latest\": \"1.0\",\n      \"url\": \"https://github.com/scanoss/webhook\",\n      \"release_date\": \"20200320\",\n      \"file\": \"webhook-1.0/scanoss/github.py\",\n      \"url_hash\": \"611e5c3a58a3c2b78385556368c5230e\",\n      \"file_hash\": \"8c2fa3f24a09137f9bb3860fa21c677e\",\n      \"file_url\": \"https://osskb.org/api/file_contents/8c2fa3f24a09137f9bb3860fa21c677e\",\n      \"dependencies\": [],\n      \"licenses\": [\n        {\n          \"name\": \"BSD-3-Clause\",\n          \"obligations\": \"https://www.osadl.org/fileadmin/checklists/unreflicenses/BSD-3-Clause.txt\",\n          \"copyleft\": \"no\",\n          \"patent_hints\": \"no\",\n          \"source\": \"component_declared\"\n        }\n      ],\n      \"copyrights\": [\n        {\n          \"name\": \"Copyright (C) 2017-2020; SCANOSS Ltd. All rights reserved.\",\n          \"source\": \"license_file\"\n        }\n      ],\n      \"vulnerabilities\": [],\n      \"quality\": [],\n      \"cryptography\": [],\n      \"server\": {\n        \"hostname\": \"localhost\",\n        \"version\": \"4.2.4\",\n        \"flags\": \"0\",\n        \"elapsed\": \"0.129558s\"\n      }\n    }\n  ]\n}\n$\n```\n\nThe following example adds a LF to the end of a file. The modified file does not match entirely anymore, but instead the snippet detection comes into effect: \n\n```\n$ echo -e \"\\n\" \u003e\u003e test.c\n$ scanoss test.c\n{\n  \"test.c\": [\n    {\n      \"id\": \"snippet\",\n      \"status\": \"pending\",\n      \"lines\": \"1-192\",\n      \"oss_lines\": \"3-194\",\n      \"matched\": \"99%\",\n      \"purl\": [\n        \"pkg:github/scanoss/webhook\"\n      ],\n      \"vendor\": \"scanoss\",\n      \"component\": \"webhook\",\n      \"version\": \"1.0\",\n      \"latest\": \"1.0\",\n      \"url\": \"https://github.com/scanoss/webhook\",\n      \"release_date\": \"20200320\",\n      \"file\": \"webhook-1.0/scanoss/github.py\",\n      \"url_hash\": \"611e5c3a58a3c2b78385556368c5230e\",\n      \"file_hash\": \"8c2fa3f24a09137f9bb3860fa21c677e\",\n      \"file_url\": \"https://osskb.org/api/file_contents/8c2fa3f24a09137f9bb3860fa21c677e\",\n      \"dependencies\": [],\n      \"licenses\": [\n        {\n          \"name\": \"BSD-3-Clause\",\n          \"obligations\": \"https://www.osadl.org/fileadmin/checklists/unreflicenses/BSD-3-Clause.txt\",\n          \"copyleft\": \"no\",\n          \"patent_hints\": \"no\",\n          \"source\": \"component_declared\"\n        }\n      ],\n      \"copyrights\": [\n        {\n          \"name\": \"Copyright (C) 2017-2020; SCANOSS Ltd. All rights reserved.\",\n          \"source\": \"license_file\"\n        }\n      ],\n      \"vulnerabilities\": [],\n      \"quality\": [],\n      \"cryptography\": [],\n      \"server\": {\n        \"hostname\": \"localhost\",\n        \"version\": \"4.2.4\",\n        \"flags\": \"0\",\n        \"elapsed\": \"0.067090s\"\n\n      }\n    }\n  ]\n}\n$\n```\n \n# Mined/ output structure \n\nThe minr command produces a directory called mined/ containing the mined metadata and original OSS files.  \n\n## Url table csv (mined/url.csv) \n\nOSS components are saved in a single CSV file called `mined/url.csv` which contains the original archive md5 hash, vendor, name, version, release date, license, purl and url as in the following example: \n\n```\n611e5c3a58a3c2b78385556368c5230e,scanoss,webhook,1.0,20200320,BSD-3-Clause,pkg:github/scanoss/webhook,https://github.com/scanoss/webhook/archive/1.0.tar.gz \n```\n\nOther metadata are saved in `mined/attribution.csv`, `mined/copyrights.csv`, `mined/cryptography.csv`, `mined/licenses.csv` and `mined/quality.csv` files. \u003c!--FIXME: add more details here --\u003e\n\n## File table csv (mined/file)\n\nOSS files are split in 256 files named `00.csv` to `ff.csv` containing the component archive md5, the file md5 and the file name, as follows: \n\n```\n6c067f97266c817b339f0e989499c8e4,00fc6e8b3ae062fbcfbe8d2e40d36e68,bison-3.5/src/ielr.c\n6c067f97266c817b339f0e989499c8e4,00ac96f26eed6fab33d7451e8f697939,bison-3.5/lib/stat-w32.c \n```\n## Pivot table csv (mined/pivot.csv)\n\nThis table is used to mantain the relation between a file and an url, the md5 of the files are saved in a single CSV file called `mined/pivot.csv` which contains the url hash and the original archive md5 hash as in the following example: \n\n```\n00fc6e8b3ae062fbcfbe8d2e40d36e68,6c067f97266c817b339f0e989499c8e4\n```\n## Wfp table csv (mined/wfp) \n\nOSS snippets are 32-bit identifiers calculated with the winnowing algorithm. They are saved in binary format split in 256 files called `00.bin` - `ff.bin` containing sequential records composed of:\n\n* 3 bytes for the snippet wfp identifier (the first byte is the file name) \n* 16 bytes for the file md5 where the snippet is found \n* 2 bytes for the line number where the wfp starts \n\nThese `.bin` files can be joined by simple concatenation.\n\n## Sources table mz file (mined/sources)\n\nThe `sources/` directory is where the original downloaded source files are stored. They are kept in 65536 `.mz` archive files. These files can be listed and extracted using the `unmz` command, which is part of minr. Unlike ZIP files, MZ files can be joined by simple concatenation.\n\n## Dependency table csv file (mined/dependency.csv)\nThe `dependency.csv` file contains mined metadata on declared dependencies.\n\n## PURLs table csv file (mined/purl.csv) \nThe `purl.csv` file contains component-level information as well as relations between purls.\nComponent-level information is presented with seven CSV fields:\n\n* Creation date,\n* Latest date,\n* Updated date,\n* Stars,\n* Watchers,\n* Is_a_fork\n\nAlternatively, purl relations are declared here with a single field:\n* related PURL\n\n# Minr join \n\nMinr can also be used to join two `mined/` structures as follows: \n\n```\n$ minr -f dir1/mined -t dir2/mined\n\n```\n\nAll files within a `mined/` structure can be joined by simple concatenation. This applies to `.csv`, `.bin` (snippet records) and `.mz` (compressed source code archives). The command above will concatenate all `.csv`, `.bin` and `.mz` files from `dir1/mined` into `dir2/mined`. Files in `dir1/mined` will be erased after concatenation is done.\n\nMinr join also performs a pre-validation on .bin and .mz file integrity (otherwise an error is generated and concatenation is aborted). \n\n# License\n\nMinr is released under the GPL 2.0 license. Please check the LICENSE file for further details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscanoss%2Fminr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscanoss%2Fminr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscanoss%2Fminr/lists"}