{"id":13464663,"url":"https://github.com/jmg/crawley","last_synced_at":"2026-04-02T02:41:11.486Z","repository":{"id":141663909,"uuid":"2339512","full_name":"jmg/crawley","owner":"jmg","description":"Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.","archived":false,"fork":false,"pushed_at":"2023-04-14T18:37:02.000Z","size":1710,"stargazers_count":186,"open_issues_count":10,"forks_count":33,"subscribers_count":22,"default_branch":"master","last_synced_at":"2024-10-29T17:51:37.910Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://project.crawley-cloud.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jmg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2011-09-07T04:59:35.000Z","updated_at":"2024-09-21T12:47:36.000Z","dependencies_parsed_at":"2024-01-17T01:34:51.618Z","dependency_job_id":null,"html_url":"https://github.com/jmg/crawley","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmg%2Fcrawley","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmg%2Fcrawley/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmg%2Fcrawley/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmg%2Fcrawley/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jmg","download_url":"https://codeload.github.com/jmg/crawley/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245454082,"owners_count":20617972,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T14:00:48.280Z","updated_at":"2025-12-27T10:13:58.242Z","avatar_url":"https://github.com/jmg.png","language":"Python","funding_links":[],"categories":["All","Python"],"sub_categories":[],"readme":"# Pythonic Crawling / Scraping Framework Built on Eventlet \n\n------------------------------------------------------------------\n\n[![Build Status](https://travis-ci.org/crawley-project/crawley.svg)](https://travis-ci.org/crawley-project/crawley)\n[![Code Climate](https://codeclimate.com/github/crawley-project/crawley/badges/gpa.svg)](https://codeclimate.com/github/crawley-project/crawley)\n[![Stories in Ready](https://badge.waffle.io/crawley-project/crawley.png?label=ready\u0026title=Ready)](https://waffle.io/crawley-project/crawley)\n\n### Features\n\n* High Speed WebCrawler built on Eventlet.\n* Supports relational databases engines like Postgre, Mysql, Oracle, Sqlite.\n* Supports NoSQL databased like Mongodb and Couchdb. **New!**\n* Export your data into Json, XML or CSV formats. **New!**\n* Command line tools.\n* Extract data using your favourite tool. XPath or Pyquery (A Jquery-like library for python).\n* Cookie Handlers.\n* Very easy to use (see the example).\n\n### Documentation\n\nhttp://packages.python.org/crawley/\n\n### Project WebSite\n\nhttp://project.crawley-cloud.com/\n\n------------------------------------------------------------------\n\n### To install crawley run\n\n```bash\n~$ python setup.py install\n```\n\n### or from pip\n\n```bash\n~$ pip install crawley\n```\n\n------------------------------------------------------------------\n\n### To start a new project run\n\n```bash\n~$ crawley startproject [project_name]\n~$ cd [project_name]\n```\n\n------------------------------------------------------------------\n\n### Write your Models\n\n```python\n\"\"\" models.py \"\"\"\n\nfrom crawley.persistance import Entity, UrlEntity, Field, Unicode\n\nclass Package(Entity):\n    \n    #add your table fields here\n    updated = Field(Unicode(255))    \n    package = Field(Unicode(255))\n    description = Field(Unicode(255))\n```\n\n------------------------------------------------------------------\n\n### Write your Scrapers\n\n```python\n\"\"\" crawlers.py \"\"\"\n\nfrom crawley.crawlers import BaseCrawler\nfrom crawley.scrapers import BaseScraper\nfrom crawley.extractors import XPathExtractor\nfrom models import *\n\nclass pypiScraper(BaseScraper):\n    \n    #specify the urls that can be scraped by this class\n    matching_urls = [\"%\"]\n    \n    def scrape(self, response):\n                        \n        #getting the current document's url.\n        current_url = response.url        \n        #getting the html table.\n        table = response.html.xpath(\"/html/body/div[5]/div/div/div[3]/table\")[0]\n        \n        #for rows 1 to n-1\n        for tr in table[1:-1]:\n                        \n            #obtaining the searched html inside the rows\n            td_updated = tr[0]\n            td_package = tr[1]\n            package_link = td_package[0]\n            td_description = tr[2]\n            \n            #storing data in Packages table\n            Package(updated=td_updated.text, package=package_link.text, description=td_description.text)\n\n\nclass pypiCrawler(BaseCrawler):\n    \n    #add your starting urls here\n    start_urls = [\"http://pypi.python.org/pypi\"]\n    \n    #add your scraper classes here    \n    scrapers = [pypiScraper]\n    \n    #specify you maximum crawling depth level    \n    max_depth = 0\n    \n    #select your favourite HTML parsing tool\n    extractor = XPathExtractor\n```\n\n### Configure your settings\n\n```python\n\"\"\" settings.py \"\"\"\n\nimport os \nPATH = os.path.dirname(os.path.abspath(__file__))\n\n#Don't change this if you don't have renamed the project\nPROJECT_NAME = \"pypi\"\nPROJECT_ROOT = os.path.join(PATH, PROJECT_NAME)\n\nDATABASE_ENGINE = 'sqlite'     \nDATABASE_NAME = 'pypi'  \nDATABASE_USER = ''             \nDATABASE_PASSWORD = ''         \nDATABASE_HOST = ''             \nDATABASE_PORT = ''     \n\nSHOW_DEBUG_INFO = True\n```\n\n------------------------------------------------------------------\n\n### Finally, just run the crawler\n\n```bash\n~$ crawley run\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmg%2Fcrawley","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjmg%2Fcrawley","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmg%2Fcrawley/lists"}