{"id":37078891,"url":"https://github.com/nogibjj/pyli","last_synced_at":"2026-01-14T09:33:04.071Z","repository":{"id":62576410,"uuid":"69824242","full_name":"nogibjj/pyli","owner":"nogibjj","description":"Deduper","archived":false,"fork":false,"pushed_at":"2017-06-26T23:21:26.000Z","size":271,"stargazers_count":5,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-28T03:52:06.247Z","etag":null,"topics":["cli","deduplication-command","duplicate-files","liten","stdout"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nogibjj.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.txt","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-02T21:52:31.000Z","updated_at":"2024-01-30T01:18:17.000Z","dependencies_parsed_at":"2022-11-03T18:44:14.488Z","dependency_job_id":null,"html_url":"https://github.com/nogibjj/pyli","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nogibjj/pyli","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nogibjj%2Fpyli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nogibjj%2Fpyli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nogibjj%2Fpyli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nogibjj%2Fpyli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nogibjj","download_url":"https://codeload.github.com/nogibjj/pyli/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nogibjj%2Fpyli/sbom","scorecard":{"id":693169,"data":{"date":"2025-08-11","repo":{"name":"github.com/nogibjj/pyli","commit":"18be77534eb6c4da4d1678f2e3af358516c61371"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.6,"checks":[{"name":"Code-Review","score":0,"reason":"Found 1/29 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":0,"reason":"license file not detected","details":["Warn: project does not have a license file"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 2 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-22T02:49:54.825Z","repository_id":62576410,"created_at":"2025-08-22T02:49:54.825Z","updated_at":"2025-08-22T02:49:54.825Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28416096,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T08:38:59.149Z","status":"ssl_error","status_checked_at":"2026-01-14T08:38:43.588Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","deduplication-command","duplicate-files","liten","stdout"],"created_at":"2026-01-14T09:33:03.275Z","updated_at":"2026-01-14T09:33:04.063Z","avatar_url":"https://github.com/nogibjj.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build\nStatus](https://travis-ci.org/nogibjj/pyli.svg?branch=master)](https://travis-ci.org/nogibjj/pyli)\n# Liten:  A deduplication command line tool and library\n\n## Summary\n\n\nA deduplication command line tool and library.  A relatively efficient\nalgorithm based on searching like sized files, and then performing a full md5\nchecksum, is used to determine duplicate files/file objects.  Files can be\ndeleted upon discovery, and pattern matching can be used to limit search\nresults. Finally, configuration file use is supported, and there is a\ndeveloping API that lends itself to customization via an ActionsMixin class.\n\n\n\n## contents\n\n### Example CLI Usage:\n\n\n\n#### Size:\n\n##### Search by size using --size or -s option:\n\n    liten.py -s 1 /mnt/raid         is equal to liten.py -s 1MB /mnt/raid\n    liten.py -s 1bytes /mnt/raid\n    liten.py -s 1KB /mnt/raid\n    liten.py -s 1MB /mnt/raid\n    liten.py -s 1GB /mnt/raid\n    liten.py c:\\in d:\\              is equal to liten.py -s 1MB c:\\in d:\\\n\n#### Report Location:\n\n\n#####Generate custom report path using -r or --report=/tmp/report.txt::\n\n    ./liten.py --report=/tmp/test.txt /Users/ngift/Documents\n\n##### By default a report will be created in CWD, called LitenDuplicateReport.csv\n\n#### Config File:\n\n\n##### You can use a config file in the following format::\n\n    [Options]\n    path=/tmp\n    size=1MB\n    pattern=*.m4v\n    delete=True\n\n\n##### You can call the config file anything and place it anywhere. \n##### Here is an example usage::\n\n    ./liten.py --config=myconfig.ini\n\n#### Verbosity:\n\nAll stdout can be suppressed by using --quiet or -q.\n\n#### Delete:\n\n\nBy using --delete the duplicate files will be automatically deleted.  The API\nhas support for an interactive mode and a dry-run mode, they have not been\nimplemented in the CLI as of yet.\n\n##### Example Library/API Usage:\n\n    \u003e\u003e\u003e Liten = Liten(spath='testData')\n    \u003e\u003e\u003e dupeFileOne = 'testData/testDocOne.txt'\n    \u003e\u003e\u003e checksumOne = Liten.createChecksum(dupeFileOne)\n    \u003e\u003e\u003e dupeFileTwo = 'testData/testDocTwo.txt'\n    \u003e\u003e\u003e checksumTwo = Liten.createChecksum(dupeFileTwo)\n    \u003e\u003e\u003e nonDupeFile = 'testData/testDocThree_wrong_match.txt'\n    \u003e\u003e\u003e checksumThree = Liten.createChecksum(nonDupeFile)\n    \u003e\u003e\u003e checksumOne == checksumTwo\n    True\n    \u003e\u003e\u003e checksumOne == checksumThree\n    False\n\nThere is also the concept of an Action, which can be implemented later, that\nwill allow customizable actions to occur upon an a condition that gets defined\nas you walk down a tree of files.\n\n#### Tests:\n\n * make all\n * Run test_create_file.py then delete those test files using liten::\n    python liten.py --delete /tmp\n\n### Display Options:\n\n\n#### Stdout:\n\n##### stdout will show you duplicate file paths and sizes such as::\n\n    Printing dups over 1 MB using md5 checksum: [SIZE] [ORIG] [DUP]\n    7 MB  Orig:  /Users/ngift/Downloads/bzr-0-2.17.tar\n    Dupe:  /Users/ngift/Downloads/bzr-0-4.17.tar\n\n##### Report:\n\n##### A report named LitenDuplicateReport.csv will be created in your current working\ndirectory::\n\n    Duplicate Version,     Path,       Size,       ModDate\n    Original, /Users/ngift/Downloads/bzr-0-2.17.tar, 7 MB, 07/10/2007 01:43:12 AM\n    Duplicate, /Users/ngift/Downloads/bzr-0-3.17.tar, 7 MB, 07/10/2007 01:43:27 AM\n\n\n#### Debug Mode Environmental Variables:\n\n* To enable print statement debugging set LITEN_DEBUG to 1\n* To enable pdb break point debugging set LITEN_DEBUG to 2\n* LITEN_DEBUG_MODE = int(os.environ.get('LITEN_DEBUG', 0))\n* Note:  When DEBUG MODE is enabled, a message will appear to standard out\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnogibjj%2Fpyli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnogibjj%2Fpyli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnogibjj%2Fpyli/lists"}