{"id":23663062,"url":"https://github.com/phillipahereza/scraper-api","last_synced_at":"2026-04-15T18:31:56.358Z","repository":{"id":102407312,"uuid":"184425496","full_name":"phillipahereza/scraper-api","owner":"phillipahereza","description":"An API for scraping JS-Rendered websites","archived":false,"fork":false,"pushed_at":"2019-05-04T08:16:00.000Z","size":5,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-20T19:19:08.913Z","etag":null,"topics":["chrome-headless","docker","fastapi","python3","selenium","starlette"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/phillipahereza.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-01T14:06:52.000Z","updated_at":"2024-08-08T01:25:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"c73a7907-02c4-4013-b657-aeb230bb8bed","html_url":"https://github.com/phillipahereza/scraper-api","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/phillipahereza/scraper-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phillipahereza%2Fscraper-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phillipahereza%2Fscraper-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phillipahereza%2Fscraper-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phillipahereza%2Fscraper-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/phillipahereza","download_url":"https://codeload.github.com/phillipahereza/scraper-api/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phillipahereza%2Fscraper-api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31854703,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"ssl_error","status_checked_at":"2026-04-15T15:24:39.138Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chrome-headless","docker","fastapi","python3","selenium","starlette"],"created_at":"2024-12-29T05:16:05.684Z","updated_at":"2026-04-15T18:31:56.347Z","avatar_url":"https://github.com/phillipahereza.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Scraping a JS-Rendered webpage\n\n### Background Information\nWhen a browser wants to access a webpage, it sends a request to the server on which the files that make up the webpage are located. \nThe server then sends a response consisting of the page’s source code back to the browser. The browser then interprets the HTML, CSS, etc, in the source code, \nallows any Javascript to run, and displays the page.\n\nOf the parts that comprise the source code of a webpage, the Javascript code is of particular interest to us in this article. \nWhen a page is loaded into a browser, it becomes a document object, whose HTML elements, or element nodes, Javascript can access. \nAs it runs, Javascript can create new HTML elements and append them to the document. A page on which this occurs is called a page \nrendered by Javascript. The new elements will not be present in the original source code of the page, the code that you see when \nyou right-click and press “view page source”; but they will be present in the code that you see if you download the page as an HTML \ndocument and open it in a text editor, or in the code in the “elements” tab in your browser’s developer tools. This HTML code, which \ncan be retrieved by JavaScript using the DOM's innerHTMLproperty, constitutes the code of the completed webpage that the browser displays \nafter the Javascript has finished running, and has all of the data that you need for scraping.\n\nWhen scraping using python or any other language, a library like urllib or requests sends a request to the server, it receives the source code of the webpage,\njust like a browser does. However, Python can not run the Javascript and allow it to create the elements that hold the content that you need to scrape. \nAnd that's where this app comes in, to scrape a JS rendered website;\n\n```\nPOST /scrape/\n```\n*Request Body*\n```\n{\n    \"url\": \"https://google.com\"\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphillipahereza%2Fscraper-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphillipahereza%2Fscraper-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphillipahereza%2Fscraper-api/lists"}