https://github.com/rggh/beautifulsoup

Web Scraping with BS4
https://github.com/rggh/beautifulsoup

Last synced: about 1 year ago
JSON representation

Web Scraping with BS4

• Host: GitHub
• URL: https://github.com/rggh/beautifulsoup
• Owner: RGGH
• Created: 2020-11-17T18:21:23.000Z (over 5 years ago)
• Default Branch: main
• Last Pushed: 2020-12-17T18:41:04.000Z (over 5 years ago)
• Last Synced: 2025-02-02T03:32:51.778Z (over 1 year ago)
• Language: Python
• Size: 85 KB
• Stars: 2
• Watchers: 1
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# BeautifulSoup

Web Scraping with BS4 - see Branch for additions

#### The branch has MySQL version

#### Noteworthy Code: 

    'set' - need to convert it back to a list

    'urljoin`

#### Conditonal logic with soup.select

    for a in soup.select('a'):

                if 'expose' in a['href']:

                            ls_expose.append(a['href'])

# VPN Rotate

    server=$(cat '/etc/openvpn/privatvpn.conf' | grep remote -m1 | cut -d" " -f2)

    nextserver=$(grep -A1 $server /etc/openvpn/list.txt|grep -v $server)

    

## sed

    sed -i "s!$server!$nextserver2!"

  

 # Proxy Version = immo24de_prx

 update line 22 with trial API key - you will need to sign up with them first

 

     response = client.get(iurl) # this is in place of "requests.get(iurl)