{"id":13846574,"url":"https://github.com/src-d/go-license-detector","last_synced_at":"2025-05-05T05:05:41.416Z","repository":{"id":57480730,"uuid":"119512564","full_name":"src-d/go-license-detector","owner":"src-d","description":"Reliable project licenses detector.","archived":false,"fork":false,"pushed_at":"2023-06-09T11:47:00.000Z","size":23649,"stargazers_count":236,"open_issues_count":11,"forks_count":39,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-05-05T05:05:33.523Z","etag":null,"topics":["license-management","license-scan","spdx","spdx-license","spdx-licenses"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/src-d.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-01-30T09:21:57.000Z","updated_at":"2024-12-09T12:00:02.000Z","dependencies_parsed_at":"2024-01-13T17:10:43.483Z","dependency_job_id":"b678f820-879d-47bd-8e43-64f2b2c9e035","html_url":"https://github.com/src-d/go-license-detector","commit_stats":{"total_commits":118,"total_committers":14,"mean_commits":8.428571428571429,"dds":0.2627118644067796,"last_synced_commit":"ee888f4c5e68b94f802b15d7194873997987ff81"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fgo-license-detector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fgo-license-detector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fgo-license-detector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/src-d%2Fgo-license-detector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/src-d","download_url":"https://codeload.github.com/src-d/go-license-detector/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252442486,"owners_count":21748451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["license-management","license-scan","spdx","spdx-license","spdx-licenses"],"created_at":"2024-08-04T18:00:40.815Z","updated_at":"2025-05-05T05:05:41.399Z","avatar_url":"https://github.com/src-d.png","language":"Go","funding_links":[],"categories":["Static Application Security Testing"],"sub_categories":[],"readme":"## WE CONTINUE THE DEVELOPMENT AT [go-enry/go-license-detector](https://github.com/go-enry/go-license-detector). This repository is abandoned, and no further updates will be done on the code base, nor issue/prs will be answered or attended.\n\n# go-license-detector [![GoDoc](https://godoc.org/gopkg.in/src-d/go-license-detector.v3?status.svg)](http://godoc.org/gopkg.in/src-d/go-license-detector.v3) [![Build Status](https://travis-ci.com/src-d/go-license-detector.svg?branch=master)](https://travis-ci.com/src-d/go-license-detector) [![Build status](https://ci.appveyor.com/api/projects/status/bxixhg99ewcgc5i5?svg=true)](https://ci.appveyor.com/project/vmarkovtsev/go-license-detector) [![codecov](https://codecov.io/github/src-d/go-license-detector/coverage.svg)](https://codecov.io/gh/src-d/go-license-detector) [![Go Report Card](https://goreportcard.com/badge/github.com/src-d/go-license-detector)](https://goreportcard.com/report/github.com/src-d/go-license-detector)\n\nProject license detector - a command line application and a library, written in Go.\nIt scans the given directory for license files, normalizes and hashes them and outputs\nall the fuzzy matches with the list of reference texts.\nThe returned names follow [SPDX](https://spdx.org/licenses/) standard.\nRead the [blog post](https://blog.sourced.tech/post/gld/).\n\nWhy? There are no similar projects which can be compiled into a native binary without\ndependencies and also support the whole SPDX license database (≈400 items).\nThis implementation is also fast, requires little memory, and the API is easy to use.\n\nThe license texts are taken directly from [license-list-data](https://github.com/spdx/license-list-data)\nrepository. The detection algorithm is **not template matching**;\nthis directly implies that go-license-detector does not provide any legal guarantees.\nThe intended area of it's usage is data mining.\n\n## Installation\n\n```\nexport GO111MODULE=on\ngo mod download\ngo build -v gopkg.in/src-d/go-license-detector.v3/cmd/license-detector\n```\n\n## Contributions\n\n...are welcome, see [CONTRIBUTING.md](CONTRIBUTING.md) and [code of conduct](CODE_OF_CONDUCT.md).\n\n## License\n\nApache 2.0, see [LICENSE.md](LICENSE.md).\n\n## Algorithm\n\n1. Find files in the root directory which may represent a license. E.g. `LICENSE` or `license.md`.\n2. If the file is Markdown or reStructuredText, render to HTML and then convert to plain text. Original HTML files are also converted.\n3. Normalize the text according to [SPDX recommendations](https://spdx.org/spdx-license-list/matching-guidelines).\n4. Split the text into unigrams and build the weighted bag of words.\n5. Calculate [Weighted MinHash](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36928.pdf).\n6. Apply Locality Sensitive Hashing and pick the reference licenses which are close.\n7. For each of the candidate, calculate the [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance) - `D`.\nthe corresponding text is the single line with each unigram represented by a single rune (character).\n8. Set the similarity as `1 - D / L` where `L` is the number of unigrams in the quieried license.\n\nThis pipeline guarantees constant time queries, though requires some initialization to preprocess\nthe reference licenses.\n\nIf there are not license files found:\n\n1. Look for README files.\n2. If the file is Markdown or reStructuredText, render to HTML and then convert to plain text. Original HTML files are also converted.\n3. Scan for words like \"copyright\", \"license\" and \"released under\". Take the neighborhood.\n4. Run Named Entity Recognition (NER) over that surrounding context and extract the possible license name.\n5. Match it against the list of license names from SPDX.\n\n## Usage\n\nCommand line:\n\n```bash\nlicense-detector /path/to/project\nlicense-detector https://github.com/src-d/go-git\n```\n\nLibrary (for a single license detection):\n\n```go\nimport (\n    \"gopkg.in/src-d/go-license-detector.v3/licensedb\"\n    \"gopkg.in/src-d/go-license-detector.v3/licensedb/filer\"\n)\n\nfunc main() {\n\tlicenses, err := licensedb.Detect(filer.FromDirectory(\"/path/to/project\"))\n}\n```\n\nLibrary (for a convenient data structure that can be formatted as JSON):\n\n```go\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\n\t\"gopkg.in/src-d/go-license-detector.v3/licensedb\"\n)\n\nfunc main() {\n\tresults := licensedb.Analyse(\"/path/to/project1\", \"/path/to/project2\")\n\tbytes, err := json.MarshalIndent(results, \"\", \"\\t\")\n\tif err != nil {\n\t\tfmt.Printf(\"could not encode result to JSON: %v\\n\", err)\n\t}\n\tfmt.Println(string(bytes))\n}\n```\n\n\n## Quality\n\nOn the [dataset](dataset.zip) of ~1000 most starred repositories on GitHub as of early February 2018\n([list](dataset.projects.gz)), **99%** of the licenses are detected.\nThe analysis of detection failures is going in [FAILURES.md](FAILURES.md).\n\nComparison to other projects on that dataset:\n\n|Detector|Detection rate|Time to scan, sec|\n|:-------|:----------------------------------------:|:-----------------------------------------|\n|[go-license-detector](https://github.com/src-d/go-license-detector)| 99%  (897/902) | 13.5 |\n|[benbalter/licensee](https://github.com/benbalter/licensee)| 75%  (673/902) | 111 |\n|[google/licenseclassifier](https://github.com/google/licenseclassifier)| 76%  (682/902) | 907 |\n|[boyter/lc](https://github.com/boyter/lc)| 88%  (797/902) | 548 |\n|[amzn/askalono](https://github.com/amzn/askalono)| 87%  (785/902) | 165 |\n|[LiD](https://source.codeaurora.org/external/qostg/lid)| 94%  (847/902) | 3660 |\n\n\u003cdetails\u003e\u003csummary\u003eHow this was measured\u003c/summary\u003e\n\u003cpre\u003e\u003ccode\u003e$ cd $(go env GOPATH)/src/gopkg.in/src-d/go-license-detector.v3/licensedb\n$ mkdir dataset \u0026\u0026 cd dataset\n$ unzip ../dataset.zip\n$ # src-d/go-license-detector\n$ time license-detector * \\\n  | grep -Pzo '\\n[-0-9a-zA-Z]+\\n\\tno license' | grep -Pa '\\tno ' | wc -l\n$ # benbalter/licensee\n$ time ls -1 | xargs -n1 -P4 licensee \\\n  | grep -E \"^License: Other\" | wc -l\n$ # google/licenseclassifier\n$ time find -type f -print | xargs -n1 -P4 identify_license \\\n  | cut -d/ -f2 | sort | uniq | wc -l\n$ # boyter/lc\n$ time lc . \\\n  | grep -vE 'NOASSERTION|----|Directory' | cut -d\" \" -f1 | sort | uniq | wc -l\n$ # amzn/askalono\n$ echo '#!/bin/sh\nresult=$(askalono id \"$1\")\necho \"$1\n$result\"' \u003e ../askalono.wrapper\n$ time find -type f -print | xargs -n1 -P4 sh ../askalono.wrapper | grep -Pzo '.*\\nLicense: .*\\n' askalono.txt | grep -av \"License: \" | cut -d/ -f 2 | sort | uniq | wc -l\n$ # LiD\n$ time license-identifier -I dataset -F csv -O lid\n$ cat lid_*.csv | cut -d, -f1 | cut -d\"'\" -f 2 | grep / | cut -d/ -f2 | sort | uniq | wc -l\n\u003c/code\u003e\u003c/pre\u003e\n\u003c/details\u003e\n\n## Regenerate binary data\n\nThe SPDX licenses are included into the binary. To update them, run\n```\nmake bindata.go\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrc-d%2Fgo-license-detector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsrc-d%2Fgo-license-detector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsrc-d%2Fgo-license-detector/lists"}