https://github.com/crawlab-team/crawlab-lite

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
https://github.com/crawlab-team/crawlab-lite

crawlab crawler crawler-management crawling-tasks platform scrapy scrapy-ui scrapyd scrapyd-ui spider web-crawler

Last synced: about 2 hours ago
JSON representation

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

• Host: GitHub
• URL: https://github.com/crawlab-team/crawlab-lite
• Owner: crawlab-team
• License: bsd-3-clause
• Created: 2020-03-20T01:21:16.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2023-02-09T20:14:22.000Z (almost 2 years ago)
• Last Synced: 2023-11-07T14:21:10.772Z (about 1 year ago)
• Topics: crawlab, crawler, crawler-management, crawling-tasks, platform, scrapy, scrapy-ui, scrapyd, scrapyd-ui, spider, web-crawler
• Language: Vue
• Homepage:
• Size: 2.36 MB
• Stars: 204
• Watchers: 13
• Forks: 69
• Open Issues: 9
• Metadata Files:
  • Readme: README-zh.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Crawlab Lite



  

    

  

  

    

  

  

    

  

  

    

  

  

    

  



中文 | [English](https://github.com/crawlab-team/crawlab-lite#readme)

[Crawlab](https://github.com/crawlab-team/crawlab) 轻量版本, 基于 Golang 的爬虫管理平台，支持任意语言编写的爬虫。

相比较 [Crawlab](https://github.com/crawlab-team/crawlab)，该版本专注于单机上的爬虫管理，平台运行不依赖任何的外部数据库，去除了大量非必要功能。

:warning: 目前该版本仍在前期开发中，部分功能可能不稳定。

## 快速开始

#### 通过 Docker Compose 运行

1. 在任意目录下创建 `docker-compose.yml`，内容如下：

```yaml

version: '3'

services:

  master:

    image: zkqiang/crawlab-lite:latest

    container_name: master

    ports:

      - "8080:8080"

    volumes:

      - "./data:/app/data"  # 数据持久化的挂载

```

2. 在目录下运行命令：

```bash

docker-compose up -d

```

3. 访问 `http://localhost:8080`

#### 源代码运行

1. 克隆仓库

```bash

git clone https://github.com/crawlab-team/crawlab-lite

cd crawlab-lite

```

2. 运行后端

```bash

cd backend

go run main.go

```

3. 运行前端

```bash

cd ../frontend

npm i && npm run serve

```

4. 访问 `http://localhost:8080`

## 截图

#### 爬虫列表

![](https://github.com/crawlab-team/crawlab-docs/blob/master/assets/images/lite-spider-list.png)

#### 任务列表

![](https://github.com/crawlab-team/crawlab-docs/blob/master/assets/images/lite-task-list.png)

#### 定时列表

![](https://github.com/crawlab-team/crawlab-docs/blob/master/assets/images/lite-schedule-list.png)

## 与 Crawlab 比较

| | Crawlab Lite | Crawlab |

| :---: | :---: | :---: |

| 跨语言爬虫 | ✅ | ✅ |

| 多节点部署 | ❌ | ✅ |

| 定时任务 | ✅ | ✅ |

| 查看日志 | ✅ | ✅ |

| 爬虫版本管理 | ✅ | ❌ |

| 数据统计 | ❌ | ✅ |

| 消息通知 | ❌ | ✅ |

| 在线编辑 | ❌ | ✅ |

| 可配置爬虫 | ❌ | ✅ |

| SDK | ❌ | ✅ |