Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/javahongxi/golab
Go 生态研究 🏡
https://github.com/javahongxi/golab
go golang
Last synced: 10 days ago
JSON representation
Go 生态研究 🏡
- Host: GitHub
- URL: https://github.com/javahongxi/golab
- Owner: javahongxi
- Created: 2019-09-28T03:52:49.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2023-05-05T02:28:58.000Z (over 1 year ago)
- Last Synced: 2024-08-03T23:30:43.317Z (4 months ago)
- Topics: go, golang
- Language: Go
- Homepage: http://go.hongxi.org
- Size: 259 KB
- Stars: 42
- Watchers: 6
- Forks: 7
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-golang-repositories - golab
README
# golab
```
git clone https://github.com/javahongxi/golab.git
go get ./...
```
### About GoLab
- lang: Go语法学习与经典示例
- crawler: 爬虫项目单任务版及并发版
- crawler_distributed: 爬虫项目分布式版
- pipeline: 搭建并行处理管道
- gin: The fastest full-featured web framework
### 爬虫项目并发版演示
- 启动相亲网站 [mockserver](https://github.com/javahongxi/mockserver)
```shell
go run mockserver/main.go
```
- 启动爬虫
```shell
go run crawler/main.go
```
### 爬虫项目分布式版演示
- 启动相亲网站 [mockserver](https://github.com/javahongxi/mockserver)
```shell
go run mockserver/main.go
```
- 下载ES并本地启动
```shell
bin/elasticsearch
```
- 启动存储服务(数据存储到ES)
```shell
cd crawler_distributed/persist/server
go run itemsaver.go --port=9090
```
- 启动多个worker服务(爬取数据)
```shell
cd crawler_distributed/worker/server
go run worker.go --port=9091
go run worker.go --port=9092
```
- 启动爬虫
```shell
cd crawler_distributed
go run main.go --itemsaver_host=:9090 --worker_hosts=:9091,:9092
```
### 并行处理管道演示
示例：对大量数据的外部排序进行并行处理
- 生成大量数据(写到文件)
```shell
go run pipeline/createfile/main.go
```
- 演示单机版并行处理
```shell
go run pipeline/demo/main.go
```
- 演示网络版并行处理
```shell
go run pipeline/netdemo/main.go
```
### Go Projects
- https://github.com/moby/moby
- https://github.com/docker/docker-ce
- https://github.com/kubernetes/kubernetes
- https://github.com/etcd-io/etcd
- https://github.com/gin-gonic/gin
- https://github.com/hashicorp/consul
- https://github.com/micro/go-micro
- https://github.com/nsqio/nsq
- https://github.com/elastic/beats
- https://github.com/pingcap/tidb
- https://github.com/CodisLabs/codis
- https://github.com/baidu/bfe
- https://github.com/caddyserver/caddy
- https://github.com/cockroachdb/cockroach
© [hongxi.org](http://hongxi.org) | [go.hongxi.org](http://go.hongxi.org)