https://github.com/ersinaksar/network-of-websites

Python command-line script that will be able to get all website page links from starting website ​START_URL with a max depth of ​DEPTH.​ The network of website URLs stored in the Memgraph database. The script also able to find the shortest path from S​TART_URL​to E​ ND_URL​from a scraped network of websites in the Memgraph database.
https://github.com/ersinaksar/network-of-websites

beatifulsoup command-line memgraph python3 scraper scraping-websites web-page-parsing

Last synced: 11 days ago
JSON representation

Python command-line script that will be able to get all website page links from starting website ​START_URL with a max depth of ​DEPTH.​ The network of website URLs stored in the Memgraph database. The script also able to find the shortest path from S​TART_URL​to E​ ND_URL​from a scraped network of websites in the Memgraph database.

• Host: GitHub
• URL: https://github.com/ersinaksar/network-of-websites
• Owner: ersinaksar
• License: gpl-3.0
• Created: 2021-09-27T23:03:27.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2021-09-27T23:34:10.000Z (about 3 years ago)
• Last Synced: 2023-11-28T20:36:11.287Z (about 1 year ago)
• Topics: beatifulsoup, command-line, memgraph, python3, scraper, scraping-websites, web-page-parsing
• Language: Python
• Homepage:
• Size: 17.6 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Network-of-Websites

docker image tag memgraph mem

docker run -p 7687:7687 -v /yourlocation/data:/var/lib/memgraph mem

docker run mem

Example Run

python main.py delete

python main.py network https://memgraph.com

python main.py path https://memgraph.com https://discourse.memgraph.com