{"id":22497145,"url":"https://github.com/chaudharypraveen98/quotestutorial","last_synced_at":"2025-10-06T12:22:20.533Z","repository":{"id":119074261,"uuid":"304677934","full_name":"chaudharypraveen98/QuotesTutorial","owner":"chaudharypraveen98","description":"This is a scraper which scrapes the quotes on the website using the scrapy framework","archived":false,"fork":false,"pushed_at":"2021-07-10T17:13:25.000Z","size":10010,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-27T21:33:21.141Z","etag":null,"topics":["python","scraping","scrapy-framework"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chaudharypraveen98.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-16T16:12:40.000Z","updated_at":"2021-07-10T17:13:34.000Z","dependencies_parsed_at":null,"dependency_job_id":"e11de20b-8d2d-4d80-9f70-827e1b0e09b6","html_url":"https://github.com/chaudharypraveen98/QuotesTutorial","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/chaudharypraveen98/QuotesTutorial","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaudharypraveen98%2FQuotesTutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaudharypraveen98%2FQuotesTutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaudharypraveen98%2FQuotesTutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaudharypraveen98%2FQuotesTutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chaudharypraveen98","download_url":"https://codeload.github.com/chaudharypraveen98/QuotesTutorial/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chaudharypraveen98%2FQuotesTutorial/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278608591,"owners_count":26015003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-06T02:00:05.630Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","scraping","scrapy-framework"],"created_at":"2024-12-06T20:16:52.161Z","updated_at":"2025-10-06T12:22:20.501Z","avatar_url":"https://github.com/chaudharypraveen98.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## **QuotesScrapy**\nThis scraper is based on the scrapy framework with pagination feature. It uses fake user agents to bypass the security.\n\nSteps to run the projects:-\n1. Activate virtual env with `. env/bin/activate`\n2. Install requirements using `pip install -r requirements.txt`\n3. Run the following commands:-\n\u003cbr\u003e`scrapy crawl QuotesScraper`\n\n\u003ch3\u003eTopics -\u003e scrapy-framework, python, scraping\u003c/h3\u003e\n\n\u003ch5\u003eSource Code Link -\u003e \u003cu\u003e\u003ca href=\"https://github.com/chaudharypraveen98/QuotesTutorial\"\u003eGitHub\u003c/a\u003e\u003c/u\u003e\u003c/h5\u003e\n\n\u003cstrong\u003eWhat We are going to do?\u003c/strong\u003e\n\u003col\u003e\n    \u003cli\u003eSetting up the Scrapy Project\u003c/li\u003e\n    \u003cli\u003eWriting a Scraper to scrapes quotes\u003c/li\u003e\n    \u003cli\u003eCleaning and Pipelining to store data in sqlite3 database\u003c/li\u003e\n    \u003cli\u003eSetting up Fake user agents and proxies\u003c/li\u003e\n\u003c/ol\u003e\n\n## Step 1 -\u003e Setting up the Scrapy Project  \n\n\u003ch3\u003eCreating a QuotesTutorial project\u003c/h3\u003e\n\n\u003cstrong\u003eBefore creating , we must know about \u003cb\u003eScrapy\u003c/b\u003e\u003c/strong\u003e\n\nScrapy is a free and open-source web-crawling framework written in Python. Originally designed for web scraping, it can also be used to extract data using APIs or as a general-purpose web crawler.\n\nTo initialize the project\n\n```scrapy startproject quotes-tutorial```\n\nThis will create a tutorial directory with the following contents:\n\n```\nquotes-tutorial/\nscrapy.cfg            # deploy configuration file\n\nquotes-tutorial/             # project's Python module, you'll import your code from here\n__init__.py\n\nitems.py          # project items definition file\n\nmiddlewares.py    # project middlewares file\n\npipelines.py      # project pipelines file\n\nsettings.py       # project settings file\n\nspiders/          # a directory where you'll later put your spiders\n__init__.py\n```\n\n## Step 2 -\u003e Writing our scraper  \n\nIt will extract quotes from \u003ca href=\"https://www.goodreads.com\"\u003eGood Reads website\u003c/a\u003e\n\n\u003cb\u003eBefore moving ahead , we must be aware of the selectors\u003c/b\u003e\n\n\u003cstrong\u003eWhat are selectors/locators?\u003c/strong\u003e\n\nA CSS Selector is a combination of an element selector and a value which identifies the web element within a web page.\n\n\u003cb\u003eThe choice of locator depends largely on your Application Under Test\u003c/b\u003e\n\n\u003cb\u003eId\u003c/b\u003e\nAn element’s id in XPATH is defined using: “[@id='example']” and in CSS using: “#” - ID's must be unique within the DOM.\nExamples:\n\n```\nXPath: //div[@id='example']\nCSS: #example\n```\n\n\u003cb\u003eElement Type\u003c/b\u003e\nThe previous example showed //div in the xpath. That is the element type, which could be input for a text box or button, img for an image, or \"a\" for a link. \n\n```\nXpath: //input or\nCss: =input\n```\n\n\u003cb\u003eDirect Child\u003c/b\u003e\nHTML pages are structured like XML, with children nested inside of parents. If you can locate, for example, the first link within a div, you can construct a string to reach it. A direct child in XPATH is defined by the use of a “/“, while on CSS, it’s defined using “\u003e”. \nExamples:\n\n```                        \nXPath: //div/a\nCSS: div \u003e a\n```\n\n\u003cb\u003eChild or Sub-Child\u003c/b\u003e\nWriting nested divs can get tiring - and result in code that is brittle. Sometimes you expect the code to change, or want to skip layers. If an element could be inside another or one of its children, it’s defined in XPATH using “//” and in CSS just by a whitespace.\nExamples:\n\n```\nXPath: //div//a\nCSS: div a\n```\n\n\u003cb\u003eClass\u003c/b\u003e\nFor classes, things are pretty similar in XPATH: “[@class='example']” while in CSS it’s just “.” \nExamples:\n\n```\nXPath: //div[@class='example']\nCSS: .example\n```\nWe have inherited the spider class from scrapy-framework. The \u003ctt\u003epage_number\u003c/tt\u003e is the next page to be scraped\n\u003ctt\u003estart_urls\u003c/tt\u003e provide the entry point to the scraper if no url is explicitly given.\n\u003ctt\u003e_parse\u003c/tt\u003e function is main function from where the scraping starts.\nWe have used the css selectors explained above.\n\n```\nimport scrapy\n\nfrom ..items import QuotesItem\n\n\nclass QuotesScraper(scrapy.Spider):\npage_number = 2\nname = \"QuotesScraper\"\nstart_urls = [\"https://www.goodreads.com/quotes/tag/inspirational\"]\n\ndef _parse(self, response, **kwargs):\nitem = QuotesItem()\nfor quote in response.css(\".quote\"):\ntitle = quote.css(\".quoteText::text\").extract_first()\nauthor = quote.css(\".authorOrTitle::text\").extract_first()\nitem[\"title\"] = title\nitem[\"author\"] = author\nyield item\n\n# Uncomment the below lines if you want to scrape all the pages in that website and comment the rest uncomment line\n\n# next_btn = response.css(\"a.next_page::attr(href)\").get()\n# if next_btn is not None:\n#     yield response.follow(next_btn, callback=self._parse())\nnext_page=f\"https://www.goodreads.com/quotes/tag/inspirational?page={QuotesScraper.page_number}\"\nif QuotesScraper.page_number \u003c 3:\nQuotesScraper.page_number += 1\nyield response.follow(next_page, callback=self._parse)\n```\n\n\u003cstrong\u003eBut what are QuotesItem here?\u003c/strong\u003e\n\nItems are containers that will be loaded with the scraped data; they work like simple python dicts but provide additional protection against populating undeclared fields, to prevent typos.\n\n```\nimport scrapy\n\n\nclass QuotesItem(scrapy.Item):\n# define the fields for your item here like:\ntitle = scrapy.Field()\nauthor = scrapy.Field()\n```\n\n## Step 3 -\u003e Pipelining to store the data in sqlite3 database  \n\n```\nimport sqlite3\n\n\nclass QuotesPipeline:\ndef __init__(self):\nself.create_connection()\nself.create_table()\n\ndef process_item(self, item, spider):\nself.db_store(item)\nreturn item\n\ndef create_connection(self):\nself.conn = sqlite3.connect(\"quotes.db\")\nself.curr = self.conn.cursor()\n\ndef create_table(self):\nself.curr.execute(\"\"\"DROP TABLE IF EXISTS quote_table\"\"\")\nself.curr.execute(\"\"\"create table quote_table( title text, author text)\"\"\")\n\ndef db_store(self, item):\nself.curr.execute(\"\"\"insert into quote_table values(?,?)\"\"\", (\nitem[\"title\"],\nitem[\"author\"]\n))\nself.conn.commit()\n```\nInitiate a **QuotesPipeline** Class. \n\n\u003ctt\u003e__init__\u003c/tt\u003e functions creates a connection between the sqlite3 database and the program with the help of \u003ctt\u003ecreate_connection\u003c/tt\u003e function. It is also responsible for creating a Quotes table with the \u003ctt\u003ecreate_table\u003c/tt\u003e function.\n\u003ctt\u003eprocess_item\u003c/tt\u003e function will process all the quotes items and will store into the database with the help of \u003ctt\u003edb_store\u003c/tt\u003e function.                    \n\n## Step 4 -\u003e Setting up Fake user agents and proxies  \nAs we are scraping on a large scale we need to avoid banning our IP\nWe are using two libraries : -\n\u003col\u003e\n    \u003cli\u003e\u003ca href=\"https://github.com/rejoiceinhope/scrapy-proxy-pool\"\u003escrapy-proxy-pool\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"https://pypi.org/project/scrapy-user-agents/\"\u003escrapy-user-agents\u003c/a\u003e\u003c/li\u003e\n\u003c/ol\u003e\n\u003cstrong\u003e1. scrapy-proxy-pool\u003c/strong\u003e\nIt will provide a bunch of proxies to ensure security of our real IP.\n\nEnable this middleware by adding the following settings to your settings.py:\n\n```PROXY_POOL_ENABLED = True```\n\nThen add rotating_proxies middlewares to your DOWNLOADER_MIDDLEWARES:\n\n```\nDOWNLOADER_MIDDLEWARES = {\n# ...\n'scrapy_proxy_pool.middlewares.ProxyPoolMiddleware': 610,\n'scrapy_proxy_pool.middlewares.BanDetectionMiddleware': 620,\n# ...\n}\n```\n\nAfter this all requests will be proxied using proxies.\n\n\u003cstrong\u003e2. scrapy-user-agents\u003c/strong\u003e\n\nRandom User-Agent middleware picks up User-Agent strings based on Python User Agents and MDN.\n\nTurn off the built-in UserAgentMiddleware and add RandomUserAgentMiddleware.\n\n```\nDOWNLOADER_MIDDLEWARES = {\n'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,\n'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,\n}\n```\n\n## Deployment  \nRunning Scrapy spiders in your local machine is very convenient for the (early) development stage, but not so much when you need to execute long-running spiders or move spiders to run in production continuously. This is where the solutions for deploying Scrapy spiders come in.\n\nPopular choices for deploying Scrapy spiders are:\n\u003col\u003e\n    \u003cli\u003e\u003ca href=\"https://github.com/scrapy/scrapyd\"\u003eScrapyd (open source)\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"https://www.zyte.com/scrapy-cloud\"\u003eZyte Scrapy Cloud (cloud-based)\u003c/a\u003e\u003c/li\u003e\n\u003c/ol\u003e\n\n## Web Preview / Output  \n\u003ca href=\"quotes.JPG\"\u003e\u003cimg class=\"img-fluid\" src=\"quotes.JPG\" alt=\"web preview\" /\u003e\u003c/a\u003e\n\u003cspan class=\"caption text-muted\"\u003eWeb preview on deployment\u003c/span\u003e\n\nPlaceholder text by \u003ca href=\"https://chaudharypraveen98.github.io/\"\u003ePraveen Chaudhary\u003c/a\u003e\u0026middot; Images by\u003ca href=\"hhttps://chaudharypraveen98.github.io/binarybeast/\"\u003eBinary Beast\u003c/a\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchaudharypraveen98%2Fquotestutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchaudharypraveen98%2Fquotestutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchaudharypraveen98%2Fquotestutorial/lists"}