https://github.com/rishavmehra/spider

this code for mapping the URL with full path
https://github.com/rishavmehra/spider

Last synced: 3 months ago
JSON representation

this code for mapping the URL with full path

• Host: GitHub
• URL: https://github.com/rishavmehra/spider
• Owner: rishavmehra
• Created: 2021-02-10T07:59:31.000Z (over 4 years ago)
• Default Branch: main
• Last Pushed: 2021-02-10T08:01:46.000Z (over 4 years ago)
• Last Synced: 2025-01-07T14:13:13.714Z (5 months ago)
• Language: Python
• Homepage:
• Size: 1000 Bytes
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# spider

import requests

import re

import urlparse

target_url = raw_input("Enter URL :")

target_links = []

def extract_link_from(url):

    response = requests.get(url)

    return re.findall('(?:href=")(.*?)"', response.content)

def crawl(url):

    href_link = extract_link_from(url)

    for link in href_link:

        link = urlparse.urljoin(url, link)

        if "#" in link:

            link = link.split("#")[0]

        if target_url in link and link not in target_links:

            target_links.append(link)

            print(link)

            crawl(link)

crawl(target_url)

s