https://github.com/izumisy/scalable-crawler

Scalable crawler, fully-managed by Google Cloud Platrom
https://github.com/izumisy/scalable-crawler

crawler docker gcp golang ruby

Last synced: 19 days ago
JSON representation

Scalable crawler, fully-managed by Google Cloud Platrom

• Host: GitHub
• URL: https://github.com/izumisy/scalable-crawler
• Owner: IzumiSy
• Created: 2020-02-08T11:26:42.000Z (almost 5 years ago)
• Default Branch: master
• Last Pushed: 2024-02-06T03:33:22.000Z (11 months ago)
• Last Synced: 2024-10-30T23:13:51.791Z (2 months ago)
• Topics: crawler, docker, gcp, golang, ruby
• Language: Go
• Homepage:
• Size: 270 KB
• Stars: 2
• Watchers: 3
• Forks: 0
• Open Issues: 3
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Scalable Crawler

CloudNative scalable crawler built on Google Cloud Platform

![architecture](docs/architecture.jpeg)

## Why Cloudbuild?

If we just want to run crawler application, there are options such as GAE, Cloud Run, and more, but CloudBuild has the way longer timeout. It can runs application to the extent of 24 hours long at maximum. The longer the better because crawling is the process that sometimes takes long time.

## Setup

### Crawler

```bash

$ make build

$ make push

```

### Backend

```bash

$ make deploy

```