https://github.com/wuyue92tree/crwy

一个简单的web爬虫框架，借鉴scrapy结构开发而来，并为scrapy使用者提供通用轮子^.^
https://github.com/wuyue92tree/crwy

gevent python scrapy utils

Last synced: 21 days ago
JSON representation

一个简单的web爬虫框架，借鉴scrapy结构开发而来，并为scrapy使用者提供通用轮子^.^

• Host: GitHub
• URL: https://github.com/wuyue92tree/crwy
• Owner: wuyue92tree
• License: mit
• Created: 2016-08-09T06:44:04.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2020-11-09T04:06:53.000Z (about 4 years ago)
• Last Synced: 2024-05-28T15:48:29.361Z (7 months ago)
• Topics: gevent, python, scrapy, utils
• Language: Python
• Homepage: http://wuyue92tree.antio.top/opensource/crwy.html
• Size: 1 MB
• Stars: 13
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Crwy

[![PyPI Version](https://img.shields.io/pypi/v/Crwy.svg)](https://pypi.python.org/pypi/Crwy)

[![Download Status](https://img.shields.io/pypi/dm/django-adminlte-ui.svg)](https://pypi.python.org/pypi/Crwy)

[![Build Status](https://travis-ci.org/wuyue92tree/crwy.svg)](https://travis-ci.org/wuyue92tree/crwy)

[![License Status](https://img.shields.io/github/license/wuyue92tree/crwy)](https://raw.githubusercontent.com/wuyue92tree/crwy/master/LICENS)

# 简介

Crwy是一个轻量级的爬虫抓取框架，参考Scrapy框架结构开发而来。该框架提供了实用的爬虫模板，旨在帮助大家快速实现爬虫任务，高效开发。并为scrapy使用者提供通用轮子^.^。新增了gevent，使爬虫异步执行，速度更快。

# 运行环境

 * Python3

 * Works on Linux, Mac OSX

# 安装

快速安装

```

pip install crwy

```

or

前往下载: https://pypi.python.org/pypi/Crwy/

# TODO

- 完善scrapy_plugs

- 完善selenium_api

- 兼容python3