{"id":13464564,"url":"https://github.com/howie6879/ruia","last_synced_at":"2025-05-14T22:08:33.159Z","repository":{"id":38420600,"uuid":"140359487","full_name":"howie6879/ruia","owner":"howie6879","description":"Async Python 3.6+ web scraping micro-framework based on asyncio","archived":false,"fork":false,"pushed_at":"2023-07-01T08:23:28.000Z","size":4675,"stargazers_count":1754,"open_issues_count":8,"forks_count":181,"subscribers_count":41,"default_branch":"master","last_synced_at":"2025-04-13T19:38:47.418Z","etag":null,"topics":["aiohttp","asyncio","asyncio-spider","crawler","crawling-framework","middlewares","python","python-ruia","ruia","spider","uvloop"],"latest_commit_sha":null,"homepage":"https://www.howie6879.com/ruia/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/howie6879.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-07-10T01:12:54.000Z","updated_at":"2025-03-30T19:47:18.000Z","dependencies_parsed_at":"2024-01-05T21:20:16.367Z","dependency_job_id":"ba52e797-4083-48ab-97ba-2834ea5ba89c","html_url":"https://github.com/howie6879/ruia","commit_stats":{"total_commits":367,"total_committers":19,"mean_commits":19.31578947368421,"dds":0.444141689373297,"last_synced_commit":"68a45028830a6e6b88eda00abf021174a8ceaceb"},"previous_names":["howie6879/aspider"],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/howie6879%2Fruia","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/howie6879%2Fruia/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/howie6879%2Fruia/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/howie6879%2Fruia/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/howie6879","download_url":"https://codeload.github.com/howie6879/ruia/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254235700,"owners_count":22036964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiohttp","asyncio","asyncio-spider","crawler","crawling-framework","middlewares","python","python-ruia","ruia","spider","uvloop"],"created_at":"2024-07-31T14:00:46.370Z","updated_at":"2025-05-14T22:08:28.144Z","avatar_url":"https://github.com/howie6879.png","language":"Python","funding_links":[],"categories":["Python","All","Misc"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/howie6879/ruia/master/docs/images/logo.png\" width='120px' height='120px' alt=\"Ruia logo\" \u003e\n\u003c/p\u003e\n\u003ch1 align=\"center\"\u003eRuia\u003c/h1\u003e\n\u003cp align=\"center\"\u003e🕸️ Async Python 3.6+ web scraping micro-framework based on asyncio.\u003c/p\u003e\n\u003cp align=\"center\"\u003e\u003cstrong\u003e⚡ Write less, run faster.\u003c/strong\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://travis-ci.org/howie6879/ruia\"\u003e\u003cimg src=\"https://travis-ci.org/howie6879/ruia.svg?branch=master\" alt=\"travis\"\u003e\u003c/a\u003e\n\u003ca href=\"https://travis-ci.org/howie6879/ruia\"\u003e\u003cimg src=\"https://codecov.io/gh/howie6879/ruia/branch/master/graph/badge.svg\" alt=\"codecov\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/ruia\"\u003e\u003cimg src=\"https://img.shields.io/pypi/pyversions/ruia.svg\" alt=\"PyPI - Python Version\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/ruia/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/ruia.svg\" alt=\"PyPI\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pepy.tech/project/ruia\"\u003e\u003cimg src=\"https://pepy.tech/badge/ruia/month\" alt=\"Downloads\"\u003e\u003c/a\u003e\n\u003ca href=\"https://gitter.im/howie6879_ruia/community?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\"\u003e\u003cimg src=\"https://badges.gitter.im/howie6879_ruia/community.svg\" alt=\"gitter\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\n![](https://raw.githubusercontent.com/howie6879/ruia/master/docs/images/ruia_demo.png)\n\n## Overview\n\nRuia is an async web scraping micro-framework, written with `asyncio` and `aiohttp`, \naims to make crawling url as convenient as possible.\n\n**Write less, run faster**:\n\n-   Documentation: [中文文档][doc_cn] |[documentation][doc_en]\n-   Organization: [python-ruia][Organization]\n-   Plugin: [awesome-ruia](https://github.com/python-ruia/awesome-ruia)(Any contributions you make are **greatly appreciated**!)\n\n## Features\n\n-   **Easy**: Declarative programming\n-   **Fast**: Powered by asyncio\n-   **Extensible**: Middlewares and plugins\n-   **Powerful**: JavaScript support\n\n## Installation\n\n``` shell\n# For Linux \u0026 Mac\npip install -U ruia[uvloop]\n\n# For Windows\npip install -U ruia\n\n# New features\npip install git+https://github.com/howie6879/ruia\n```\n\n## Tutorials\n\n1.  [Overview](https://docs.python-ruia.org/en/tutorials/overview.html)\n2.  [Installation](https://docs.python-ruia.org/en/tutorials/installation.html)\n3.  [Define Data Items](https://docs.python-ruia.org/en/tutorials/item.html)\n4.  [Spider Control](https://docs.python-ruia.org/en/tutorials/spider.html)\n5.  [Request \u0026 Response](https://docs.python-ruia.org/en/tutorials/request.html)\n6.  [Customize Middleware](https://docs.python-ruia.org/en/tutorials/middleware.html)\n7.  [Write a Plugins](https://docs.python-ruia.org/en/tutorials/plugins.html)\n\n\n## TODO\n\n- [x] Cache for debug, to decreasing request limitation, [ruia-cache](https://github.com/python-ruia/ruia-cache)\n- [x] Provide an easy way to debug the script, [ruia-shell](https://github.com/python-ruia/ruia-shell)\n- [ ] Distributed crawling/scraping\n\n## Contribution\n\nRuia is still under developing, feel free to open issues and pull requests:\n\n-   Report or fix bugs\n-   Require or publish plugins\n-   Write or fix documentation\n-   Add test cases\n\n\u003c!-- To get src for img: https://api.github.com/users/username --\u003e\n\u003ca href=\"https://github.com/howie6879\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/17047388?s=60\u0026v=4\" title=\"howie6879\" width=\"50\" height=\"50\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/panhaoyu\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/23495987?s=60\u0026v=4\" title=\"panhaoyut\" width=\"50\" height=\"50\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/mirzazulfan\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/36124339?s=64\u0026v=4\" title=\"mirzazulfan\" width=\"50\" height=\"50\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/abmyii\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/52673001?s=60\u0026v=4\" title=\"abmyii\" width=\"50\" height=\"50\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/maxzheng\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/9684260?s=60\u0026v=4\" title=\"maxzheng\" width=\"50\" height=\"50\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/ruter\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/8568876?s=60\u0026v=4\" title=\"ruter\" width=\"50\" height=\"50\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/duolaAOA\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/26339233?s=60\u0026v=4\" title=\"duolaAOA\" width=\"50\" height=\"50\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/fengdongfa1995\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/20141092?s=60\u0026v=4\" title=\"fengdongfa1995\" width=\"50\" height=\"50\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/daijiangtian\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/18069191?s=60\u0026v=4\" title=\"daijiangtian\" width=\"50\" height=\"50\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/scott-stoltzman-consulting\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/66376167?s=60\u0026v=4\" title=\"consulting\" width=\"50\" height=\"50\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/Leezj9671\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/11917826?s=60\u0026v=4\" title=\"Leezj9671\" width=\"50\" height=\"50\"\u003e\u003c/a\u003e\n\n!!!Notice: We use [black](https://github.com/psf/black) to format the code.\n\n## Thanks\n\n-   [aiohttp](https://github.com/aio-libs/aiohttp/)\n-   [demiurge](https://github.com/matiasb/demiurge)\n\n[doc_cn]: https://www.howie6879.cn/ruia/\n[doc_en]: https://howie6879.github.io/ruia/\n[Awesome]: https://github.com/python-ruia/awesome-ruia\n[Organization]: https://github.com/python-ruia\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhowie6879%2Fruia","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhowie6879%2Fruia","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhowie6879%2Fruia/lists"}