https://github.com/ail-project/similarius
Similarius is a Python library to compare web page and evaluate the level of similarity.
https://github.com/ail-project/similarius
Last synced: about 1 year ago
JSON representation
Similarius is a Python library to compare web page and evaluate the level of similarity.
- Host: GitHub
- URL: https://github.com/ail-project/similarius
- Owner: ail-project
- License: bsd-2-clause
- Created: 2023-01-11T14:31:59.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2025-04-29T07:58:12.000Z (over 1 year ago)
- Last Synced: 2025-04-29T08:42:28.293Z (over 1 year ago)
- Language: Python
- Size: 528 KB
- Stars: 18
- Watchers: 6
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Similarius
Similarius is a Python library to compare web page and evaluate the level of similarity.
The tool can be used as a stand-alone tool or to feed other systems.
# Requirements
- Python 3.8+
- [Requests](https://github.com/psf/requests)
- [Scikit-learn](https://github.com/scikit-learn/scikit-learn)
- [Beautifulsoup4](https://pypi.org/project/beautifulsoup4/)
- [nltk](https://github.com/nltk/nltk)
# Installation
## Source install
**Similarius** can be install with poetry. If you don't have poetry installed, you can do the following `curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python`.
~~~bash
$ poetry install
$ poetry shell
$ similarius -h
~~~
## pip installation
~~~bash
$ pip3 install similarius
~~~
# Usage
~~~bash
dacru@dacru:~/git/Similarius/similarius$ similarius --help
usage: similarius.py [-h] [-o ORIGINAL] [-w WEBSITE [WEBSITE ...]]
optional arguments:
-h, --help show this help message and exit
-o ORIGINAL, --original ORIGINAL
Website to compare
-w WEBSITE [WEBSITE ...], --website WEBSITE [WEBSITE ...]
Website to compare
~~~
# Usage example
~~~bash
dacru@dacru:~/git/Similarius/similarius$ similarius -o circl.lu -w europa.eu circl.eu circl.lu
~~~
# Used as a library
~~~python
import argparse
from similarius import get_website, extract_text_ressource, sk_similarity, ressource_difference, ratio
parser = argparse.ArgumentParser()
parser.add_argument("-w", "--website", nargs="+", help="Website to compare")
parser.add_argument("-o", "--original", help="Website to compare")
args = parser.parse_args()
# Original
original = get_website(args.original)
if not original:
print("[-] The original website is unreachable...")
exit(1)
original_text, original_ressource = extract_text_ressource(original.text)
for website in args.website:
print(f"\n********** {args.original} <-> {website} **********")
# Compare
compare = get_website(website)
if not compare:
print(f"[-] {website} is unreachable...")
continue
compare_text, compare_ressource = extract_text_ressource(compare.text)
# Calculate
sim = str(sk_similarity(compare_text, original_text))
print(f"\nSimilarity: {sim}")
ressource_diff = ressource_difference(original_ressource, compare_ressource)
print(f"Ressource Difference: {ressource_diff}")
ratio_compare = ratio(ressource_diff, sim)
print(f"Ratio: {ratio_compare}")
~~~
# Acknowledgment
The project has been co-funded by CEF-TC-2020-2 - 2020-EU-IA-0260 - JTAN - Joint Threat Analysis Network.