{"id":19832054,"url":"https://github.com/ail-project/similarius","last_synced_at":"2025-05-01T16:32:54.859Z","repository":{"id":65339574,"uuid":"587764028","full_name":"ail-project/Similarius","owner":"ail-project","description":"Similarius is a Python library to compare web page and evaluate the level of similarity.","archived":false,"fork":false,"pushed_at":"2025-04-29T07:58:12.000Z","size":541,"stargazers_count":18,"open_issues_count":0,"forks_count":1,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-29T08:42:28.293Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ail-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-01-11T14:31:59.000Z","updated_at":"2025-04-29T07:58:10.000Z","dependencies_parsed_at":"2023-12-22T12:17:08.710Z","dependency_job_id":"76bf56b8-6121-4d74-92c8-c1776c7cdbf9","html_url":"https://github.com/ail-project/Similarius","commit_stats":{"total_commits":25,"total_committers":4,"mean_commits":6.25,"dds":0.4,"last_synced_commit":"03e2c111b5e542dbb14e04d4fe53fba460e53990"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2FSimilarius","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2FSimilarius/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2FSimilarius/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ail-project%2FSimilarius/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ail-project","download_url":"https://codeload.github.com/ail-project/Similarius/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251907060,"owners_count":21663210,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T11:36:13.361Z","updated_at":"2025-05-01T16:32:54.434Z","avatar_url":"https://github.com/ail-project.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Similarius\n\nSimilarius is a Python library to compare web page and evaluate the level of similarity.\n\nThe tool can be used as a stand-alone tool or to feed other systems.\n\n\n\n# Requirements\n\n- Python 3.8+\n- [Requests](https://github.com/psf/requests)\n- [Scikit-learn](https://github.com/scikit-learn/scikit-learn)\n- [Beautifulsoup4](https://pypi.org/project/beautifulsoup4/)\n- [nltk](https://github.com/nltk/nltk)\n\n\n\n# Installation\n\n## Source install\n\n**Similarius** can be install with poetry. If you don't have poetry installed, you can do the following `curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python`.\n\n~~~bash\n$ poetry install\n$ poetry shell\n$ similarius -h\n~~~\n\n## pip installation\n\n~~~bash\n$ pip3 install similarius\n~~~\n\n\n\n# Usage\n\n~~~bash\ndacru@dacru:~/git/Similarius/similarius$ similarius --help\nusage: similarius.py [-h] [-o ORIGINAL] [-w WEBSITE [WEBSITE ...]]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -o ORIGINAL, --original ORIGINAL\n                        Website to compare\n  -w WEBSITE [WEBSITE ...], --website WEBSITE [WEBSITE ...]\n                        Website to compare\n~~~\n\n\n\n# Usage example\n\n~~~bash\ndacru@dacru:~/git/Similarius/similarius$ similarius -o circl.lu -w europa.eu circl.eu circl.lu\n~~~\n\n\n\n# Used as a library\n\n~~~python\nimport argparse\nfrom similarius import get_website, extract_text_ressource, sk_similarity, ressource_difference, ratio\n\nparser = argparse.ArgumentParser()\nparser.add_argument(\"-w\", \"--website\", nargs=\"+\", help=\"Website to compare\")\nparser.add_argument(\"-o\", \"--original\", help=\"Website to compare\")\nargs = parser.parse_args()\n\n# Original\noriginal = get_website(args.original)\n\nif not original:\n    print(\"[-] The original website is unreachable...\")\n    exit(1)\n\noriginal_text, original_ressource = extract_text_ressource(original.text)\n\nfor website in args.website:\n    print(f\"\\n********** {args.original} \u003c-\u003e {website} **********\")\n\n    # Compare\n    compare = get_website(website)\n\n    if not compare:\n        print(f\"[-] {website} is unreachable...\")\n        continue\n\n    compare_text, compare_ressource = extract_text_ressource(compare.text)\n\n    # Calculate\n    sim = str(sk_similarity(compare_text, original_text))\n    print(f\"\\nSimilarity: {sim}\")\n\n    ressource_diff = ressource_difference(original_ressource, compare_ressource)\n    print(f\"Ressource Difference: {ressource_diff}\")\n\n    ratio_compare = ratio(ressource_diff, sim)\n    print(f\"Ratio: {ratio_compare}\")\n~~~\n\n\n\n# Acknowledgment\n\n![](./img/cef.png)\n\nThe project has been co-funded by CEF-TC-2020-2 - 2020-EU-IA-0260 - JTAN - Joint Threat Analysis Network.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fail-project%2Fsimilarius","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fail-project%2Fsimilarius","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fail-project%2Fsimilarius/lists"}