https://github.com/manzil-infinity180/website-scrapper
A demo project for scrapping website using puppeteer(js/ts) and scrapy(python)
https://github.com/manzil-infinity180/website-scrapper
internship jobs remote scraping
Last synced: 4 months ago
JSON representation
A demo project for scrapping website using puppeteer(js/ts) and scrapy(python)
- Host: GitHub
- URL: https://github.com/manzil-infinity180/website-scrapper
- Owner: manzil-infinity180
- Created: 2024-11-11T10:59:27.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-10-28T05:08:27.000Z (8 months ago)
- Last Synced: 2025-10-28T07:07:03.439Z (8 months ago)
- Topics: internship, jobs, remote, scraping
- Language: TypeScript
- Homepage:
- Size: 32 MB
- Stars: 3
- Watchers: 1
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# website-scrapper
* This is just the demo project for scraping different websites and social media
- I am using puppeteer(javascript/typescript) with cheerio and other library or package
- Using scrpy(python) + with some proxy provider
- puppeteer-extra-plugin-stealth: To make it harder for bot detection technology to detect headless browser instances.
- puppeteer-extra-plugin-recaptcha: To solve reCAPTCHAs and hCaptchas automatically.
- puppeteer-extra-plugin-adblocker: To remove ads and trackers, reducing bandwidth and load times as a result.
- puppeteer-extra-plugin-devtools: To make browser debugging possible from anywhere by creating a secure tunnel to the DevTools.
- puppeteer-extra-plugin-repl: To make debugging fun with an interactive REPL (Read-Eval-Print-Loop) interface.
- puppeteer-extra-plugin-block-resources: To block dynamically page resources such as images, media, CSS, and JS files.
- puppeteer-extra-plugin-anonymize-ua: To anonymize the User-Agent header on page navigation. Learn about why this is important in our guide on User-Agent for web scraping.
- puppeteer-extra-plugin-user-preferences: To set custom Chrome/Chromium user preferences.
## Demo - (Under Development)
> Read More
> https://www.webshare.io/academy-article/puppeteer-extra#conclusion
> Scrape Any **Twitter** Post
> [Click Here](https://github.com/manzil-infinity180/website-scrapper/tree/main/src/twitter)
## See Latest YC job/internship
> [Click Here](https://github.com/manzil-infinity180/website-scrapper/tree/main/yc-daily-post)
## See Latest Javascript Remote job
> [Click Here](https://github.com/manzil-infinity180/website-scrapper/tree/main/javascript-jobs-post)
