{"id":20455315,"url":"https://github.com/alexmhack/js_driven_scraping","last_synced_at":"2025-10-25T08:11:00.714Z","repository":{"id":114089557,"uuid":"148101619","full_name":"Alexmhack/js_driven_scraping","owner":"Alexmhack","description":"Scraping Javascript Driven Websites Using Python-Selenium","archived":false,"fork":false,"pushed_at":"2018-09-11T15:04:47.000Z","size":39,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-09T02:44:19.152Z","etag":null,"topics":["beginner-friendly","beginner-project","python-selenium","python-tutorial","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Alexmhack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-09-10T05:04:27.000Z","updated_at":"2023-08-27T01:37:18.000Z","dependencies_parsed_at":"2023-06-12T13:15:18.617Z","dependency_job_id":null,"html_url":"https://github.com/Alexmhack/js_driven_scraping","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Alexmhack/js_driven_scraping","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alexmhack%2Fjs_driven_scraping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alexmhack%2Fjs_driven_scraping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alexmhack%2Fjs_driven_scraping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alexmhack%2Fjs_driven_scraping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Alexmhack","download_url":"https://codeload.github.com/Alexmhack/js_driven_scraping/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Alexmhack%2Fjs_driven_scraping/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280923593,"owners_count":26414266,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-25T02:00:06.499Z","response_time":81,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beginner-friendly","beginner-project","python-selenium","python-tutorial","python3"],"created_at":"2024-11-15T11:18:32.631Z","updated_at":"2025-10-25T08:11:00.708Z","avatar_url":"https://github.com/Alexmhack.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# js_driven_scraping\nScraping Javascript Driven Websites Using Python\n\nRun ```js_scrape.py``` file and you should get lesser results of the images then \nactually exists on the website. This is due to the javascript getting loaded while\nthe website is also loading so python ```requests``` cannot scrape that for us\n\nFor scraping javascript-driven websites we need a more powerful python package\nwhich is [selenium-python](https://selenium-python.readthedocs.io/)\n\nUsing the instructions given in the docs for selenium-python install the selenium \nand firefox drivers for selenium. Be sure to donwload \n\n![geckodriver](https://github.com/Alexmhack/js_driven_scraping/blob/master/images/Capture.PNG)\n\nDownload ```32bit``` or ```64bit``` according to our specs for windows, unzip the \nfolder and add the path of that folder in **system variables**\n\nCreate a new file named ```using_selenium.py``` \n\n```\nimport requests\nfrom bs4 import BeautifulSoup\nfrom selenium import webdriver\n\ndriver = webdriver.Firefox()\n```\n\nWhen you run the file the firefox browser with a new window should open. \n\nFor opening a url in browser window, close the firefox browser and add \n\n```\ndriver.get('https://google.com')\n```\n\nat last. Then run the file again and the browser should open the google website\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexmhack%2Fjs_driven_scraping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexmhack%2Fjs_driven_scraping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexmhack%2Fjs_driven_scraping/lists"}