https://github.com/phillipahereza/scraper-api
An API for scraping JS-Rendered websites
https://github.com/phillipahereza/scraper-api
chrome-headless docker fastapi python3 selenium starlette
Last synced: 2 months ago
JSON representation
An API for scraping JS-Rendered websites
- Host: GitHub
- URL: https://github.com/phillipahereza/scraper-api
- Owner: phillipahereza
- Created: 2019-05-01T14:06:52.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2019-05-04T08:16:00.000Z (about 7 years ago)
- Last Synced: 2025-05-20T19:19:08.913Z (about 1 year ago)
- Topics: chrome-headless, docker, fastapi, python3, selenium, starlette
- Language: Python
- Size: 4.88 KB
- Stars: 4
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Scraping a JS-Rendered webpage
### Background Information
When a browser wants to access a webpage, it sends a request to the server on which the files that make up the webpage are located.
The server then sends a response consisting of the page’s source code back to the browser. The browser then interprets the HTML, CSS, etc, in the source code,
allows any Javascript to run, and displays the page.
Of the parts that comprise the source code of a webpage, the Javascript code is of particular interest to us in this article.
When a page is loaded into a browser, it becomes a document object, whose HTML elements, or element nodes, Javascript can access.
As it runs, Javascript can create new HTML elements and append them to the document. A page on which this occurs is called a page
rendered by Javascript. The new elements will not be present in the original source code of the page, the code that you see when
you right-click and press “view page source”; but they will be present in the code that you see if you download the page as an HTML
document and open it in a text editor, or in the code in the “elements” tab in your browser’s developer tools. This HTML code, which
can be retrieved by JavaScript using the DOM's innerHTMLproperty, constitutes the code of the completed webpage that the browser displays
after the Javascript has finished running, and has all of the data that you need for scraping.
When scraping using python or any other language, a library like urllib or requests sends a request to the server, it receives the source code of the webpage,
just like a browser does. However, Python can not run the Javascript and allow it to create the elements that hold the content that you need to scrape.
And that's where this app comes in, to scrape a JS rendered website;
```
POST /scrape/
```
*Request Body*
```
{
"url": "https://google.com"
}
```