{"id":22497151,"url":"https://github.com/chaudharypraveen98/scraper","last_synced_at":"2026-04-09T22:45:49.661Z","repository":{"id":119074307,"uuid":"272059741","full_name":"chaudharypraveen98/Scraper","owner":"chaudharypraveen98","description":"It is a python based script which is running on python based framework Flask and Fast Api. We are using the Ngrok to deploy our localhost to web.","archived":false,"fork":false,"pushed_at":"2021-07-09T16:52:10.000Z","size":184,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-27T21:33:15.501Z","etag":null,"topics":["cli","fastapi","flask","pandas","requestshtml","scraping","script"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chaudharypraveen98.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-13T17:56:50.000Z","updated_at":"2024-08-01T19:10:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"9ebb5fbf-0746-4ece-b1af-3aa8caffb06e","html_url":"https://github.com/chaudharypraveen98/Scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/chaudharypraveen98/Scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaudharypraveen98%2FScraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaudharypraveen98%2FScraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaudharypraveen98%2FScraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaudharypraveen98%2FScraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chaudharypraveen98","download_url":"https://codeload.github.com/chaudharypraveen98/Scraper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaudharypraveen98%2FScraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270781393,"owners_count":24643820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","fastapi","flask","pandas","requestshtml","scraping","script"],"created_at":"2024-12-06T20:16:56.475Z","updated_at":"2026-04-09T22:45:49.226Z","avatar_url":"https://github.com/chaudharypraveen98.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scraper\nA scraper script which is running on python's framework Flask and Fast Api can scrape BoxOfficeMojo website. We are using the Ngrok to deploy our localhost to web.\n\n### Web preview\n\u003cimg src=\"output.JPG\"\u003e\n\n### What we are going to build?\n1. We are creating a script to scrape the BoxOfficeMojo and export it to excel using the [pandas library](https://pandas.pydata.org/) by making Dataframe using the nested dictionary.\n\n2. Then, setting up the [Flask](https://flask.palletsprojects.com/en/2.0.x/) Server to run periodic scraping using the get request on a particular url endpoint after a certain interval of time.\n\n3. Then, making a url endpoint to serve the data which we had scraped till now.\n\n4. We are making our localhost server accessible from anywhere using the [Ngrok](https://ngrok.com/).\n\n5. Lastly, We are making a cmdlet or powershell script to run from anywhere wether command prompt or window power shell.\n\n## Step 1 -\u003e Writing scraper and exporting data (Scraper.py)\n### Writing scraper\nWe will use requestsHTML to scrape the data with the help of css selectors.\n\n\u003cstrong\u003eBut, What are selectors/locators?\u003c/strong\u003e\n\nA CSS Selector is a combination of an element selector and a value which identifies the web element within a web page.\n\n\u003cb\u003eThe choice of locator depends largely on your Application Under Test\u003c/b\u003e\n\n\u003cb\u003eId\u003c/b\u003e\nAn element’s id in XPATH is defined using: “[@id='example']” and in CSS using: “#” - ID's must be unique within the DOM.\nExamples:\n```\nXPath: //div[@id='example']\nCSS: #example\n```\n\n\u003cb\u003eElement Type\u003c/b\u003e\nThe previous example showed //div in the xpath. That is the element type, which could be input for a text box or button, img for an image, or \"a\" for a link. \n\n```\nXpath: //input or\nCss: =input\n```\n\n\u003cb\u003eDirect Child\u003c/b\u003e\nHTML pages are structured like XML, with children nested inside of parents. If you can locate, for example, the first link within a div, you can construct a string to reach it. A direct child in XPATH is defined by the use of a “/“, while on CSS, it’s defined using “\u003e”. \nExamples:\n```                        \nXPath: //div/a\nCSS: div \u003e a\n```\n\n\u003cb\u003eChild or Sub-Child\u003c/b\u003e\nWriting nested divs can get tiring - and result in code that is brittle. Sometimes you expect the code to change, or want to skip layers. If an element could be inside another or one of its children, it’s defined in XPATH using “//” and in CSS just by a whitespace.\nExamples:\n```\nXPath: //div//a\nCSS: div a\n```\n\n\u003cb\u003eClass\u003c/b\u003e\nFor classes, things are pretty similar in XPATH: “[@class='example']” while in CSS it’s just “.” \nExamples:\n```\nXPath: //div[@class='example']\nCSS: .example\n```\n\n\u003cstrong\u003eLibraries Required :-\u003c/strong\u003e\n\u003col\u003e\n    \u003cli\u003eRequest-html\u003c/li\u003e\n    \u003cli\u003eRequests\u003c/li\u003e\n    \u003cli\u003ePandas\u003c/li\u003e\n\u003c/ol\u003e\n\n\u003cstrong\u003eSo, How to install them ?\u003c/strong\u003e\nRun the following commands in python shell\n```\npip install requests-html\npip install requests\npip install pandas\n```\n\n\u003cstrong\u003eGetting the html code using the Requests library\u003c/strong\u003e\n\nIt will take the filename as the keyword argument if save data is true. By default it will not save the html code in a file.\n\nWe are making a get request then we check for the response  status code. if status_code is in range(200,299) then its Ok.\nThen the html code is feed into `pharse_and_extract` function\n```\ndef url_to_text(url, filename=\"world.html\", save=False):\n    r = requests.get(url)\n    if r.status_code == 200:\n    html_text = r.text\n    if save:\n    with open(filename, 'w') as f:\n    f.write(html_text)\n    return html_text\n    return None\n```\n\n\u003cstrong\u003eHtml Parsing\u003c/strong\u003e\n\nThe response code is parsed into html code using the requestsHTML HTML parser.\n```\ndef pharse_and_extract(url, name=2020):\n    html_text = url_to_text(url)\n    if html_text is None:\n    return \"\"\n    r_html = HTML(html=html_text)\n```\n\n\u003cstrong\u003eScraping\u003c/strong\u003e\n\nWe will use the css selectors to locate the element and get the required data.\n```\ndef pharse_and_extract(url, name=2020):\n    html_text = url_to_text(url)\n    if html_text is None:\n    return \"\"\n    r_html = HTML(html=html_text)\n\n    # scraping starts from here\n\n    table_class = \".imdb-scroll-table\"\n    r_table = r_html.find(table_class)\n    table_data = []\n\n    if len(r_table) == 1:\n    table = r_table[0]\n    header_col = table.find(\"th\")\n    header_names = [x.text for x in header_col]\n    rows = table.find(\"tr\")\n    for row in rows[1:]:\n    cols = row.find(\"td\")\n    row_data = []\n    for i, col in enumerate(cols):\n    row_data.append(col.text)\n    table_data.append(row_data)\n```\n\n\u003cstrong\u003eExporting into CSV file\u003c/strong\u003e\n\nOnce the data is scraped, then it is loaded into Pandas Dataframe and then exported into CSV file.\n```\ndef pharse_and_extract(url, name=2020):\n    html_text = url_to_text(url)\n    if html_text is None:\n    return \"\"\n    r_html = HTML(html=html_text)\n    table_class = \".imdb-scroll-table\"\n    r_table = r_html.find(table_class)\n    table_data = []\n\n    if len(r_table) == 1:\n    table = r_table[0]\n    header_col = table.find(\"th\")\n    header_names = [x.text for x in header_col]\n    rows = table.find(\"tr\")\n    for row in rows[1:]:\n    cols = row.find(\"td\")\n    row_data = []\n    for i, col in enumerate(cols):\n    row_data.append(col.text)\n    table_data.append(row_data)\n\n    # exporting data starts from here\n\n    path = os.path.join(BASE_DIR, 'data')\n    os.makedirs(path, exist_ok=True)\n    filepath = os.path.join('data', f'{name}.csv')\n\n    # loading into dataframe\n    df = pd.DataFrame(table_data, columns=header_names)\n    df.to_csv(filepath, index=False)\n```\n\n## Step 2 -\u003e Setting up Flask Server\n\nWe are making a url endpoint, so that we can set up periodic scraping through a post request.\n```\nfrom flask import Flask\nfrom scraper import run as scraper_runner\nfrom logger import log_save\n\napp = Flask(__name__)\n\n@app.route(\"/box-office\", methods=['POST'])\ndef box_office_view():\n    log_save()\n    scraper_runner()\n    return \"done\"\n```\n## Step 3 -\u003e Serving the scraped data using FastAPI\nwe will load the data into dataframe from a csv file and then return a dictionary data to the forntend\n```\nfrom fastapi import FastAPI\nfrom scraper import run as scraper_runner\nfrom logger import log_save\nimport os\nimport pandas as pd\n\napp = FastAPI()\n\n# Getting the file location\n\ncurrent = (os.path.join(os.getcwd(), 'data'))\nos.makedirs(current, exist_ok=True)\nfile = os.path.join(current, \"box_office_cleaned.csv\")\n\n# Serving the file on a get request\n\n@app.get(\"/box-office-collect\")\ndef abc():\n    df = pd.read_csv(file)\n    return df.to_dict(\"Rank\")\n\n```\nWe can even setup periodic scraping in FastAPI too by simple command given below -:\n```\n@app.post(\"/box-office\")\ndef box_office():\n    log_save()\n    scraper_runner()\n    return {\"data\": \"success\"}\n\n```\n\n## Step 4 -\u003e Deployment  \nFor deployment,  We are using the \u003cstrong\u003eNgrok\u003c/strong\u003e to deploy our localhost to web.\u003cspan\u003e\u003ca href=\"https://ngrok.com/\"\u003eFor More Info\u003c/a\u003e\u003c/span\u003e\nMake a python file (trigger.py)\nOnce our flask server is exposed through ngrok then we can make a post request to make a periodic scraping. We can use tools for that purpose like :-\n\u003col\u003e\n    \u003cli\u003eUptimeRobot\u003c/li\u003e\n    \u003cli\u003eFreshworks\u003c/li\u003e\n    \u003cli\u003eUptrends\u003c/li\u003e\n\u003c/ol\u003e\n\n```\nimport requests\n\nurl = \"http://9d44e5091b7d.ngrok.io\"\nendpoint = f'{url}/box-office'\nr = requests.post(endpoint, json={})\nprint(r.text)\n```\n\n## Step 5 -\u003e Making a ps1 script\nIt will help us to run command from anywhere like cmd, powershell etc\n\n\u003cb\u003eMake a file with ps1 extension\u003c/b\u003e\n\u003cstrong\u003eFor Flask Server\u003c/strong\u003e\n\n```\nset FLASK_APP=flask_server.py\n$env:FLASK_APP = \"flask_server.py\"\nflask run --host=127.0.0.1 --port=8000\n```\n\n\u003cstrong\u003eFor FastAPI Server\u003c/strong\u003e\n\n`uvicorn fast_server:app --port 8888`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchaudharypraveen98%2Fscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchaudharypraveen98%2Fscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchaudharypraveen98%2Fscraper/lists"}