https://github.com/medcl/gopa-abandoned

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）
https://github.com/medcl/gopa-abandoned

crawler golang lightweight spider

Last synced: 3 months ago
JSON representation

GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）

• Host: GitHub
• URL: https://github.com/medcl/gopa-abandoned
• Owner: medcl
• License: other
• Archived: true
• Created: 2013-07-08T08:56:23.000Z (over 11 years ago)
• Default Branch: master
• Last Pushed: 2017-06-18T00:21:09.000Z (over 7 years ago)
• Last Synced: 2024-06-27T08:35:30.414Z (4 months ago)
• Topics: crawler, golang, lightweight, spider
• Language: Go
• Homepage:
• Size: 22.2 MB
• Stars: 94
• Watchers: 10
• Forks: 30
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGES.md
  • License: LICENSE
  • Security: docs/security.md

Awesome Lists containing this project

README

        
# Gopa #

![https://github.com/medcl/gopa](https://raw.githubusercontent.com/medcl/gopa/master/static/assets/img/logo.png)

[狗爬], A Spider Written in Go.

[![Travis](https://travis-ci.org/medcl/gopa.svg?branch=master)](https://travis-ci.org/medcl/gopa)

[![Go Report Card](https://goreportcard.com/badge/github.com/medcl/gopa)](https://goreportcard.com/report/github.com/medcl/gopa)

## Build Gopa ##

Mac/Linux: Run `make build` to build the Gopa

Windows:  go with this wiki - [How to build GOPA on windows](https://github.com/medcl/gopa/wiki/How-to-build-GOPA-on-windows)

## Download ##

[Release](https://github.com/medcl/gopa/releases)

## Start Gopa ##

After download/build the binary file, run `./gopa` to start the Gopa 

Run `./gopa -h` to get the full list of commandline options

* -log option : logging level,can be set to `trace`,`debug`,`info`,`warn`,`error` ,default is `info`

* -daemon option : run in background as daemon

* -pprof option : enable and setup pprof/expvar service, eg: localhost:6060 , the endpoint will be: http://localhost:6060/debug/pprof/ and http://localhost:6060/debug/vars

* -cpuprofile option : write cpu profile to this file

* -memprofile option : write memory profile to this file

## Stop Gopa ##

It's safety to press `ctrl+c` stop the current running Gopa, Gopa will handle the rest,saving the checkpoint,

you may restore the job later,the world is still in your hand.

If you are running `Gopa` as daemon, you can stop it like this:

```

 kill -QUIT `pgrep gopa`

```

## UI

Visit `http://127.0.0.1:9001/` for more details.

License

=======

Released under the [Apache License, Version 2.0](https://github.com/medcl/gopa/blob/master/LICENSE) .