Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/youngjuning/homebrew-juejin-spider
爬取掘金专栏并转为Markdown保存到本地
https://github.com/youngjuning/homebrew-juejin-spider
colly golang spider
Last synced: 2 months ago
JSON representation
爬取掘金专栏并转为Markdown保存到本地
- Host: GitHub
- URL: https://github.com/youngjuning/homebrew-juejin-spider
- Owner: youngjuning
- License: mit
- Created: 2020-08-13T15:09:39.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2020-08-18T15:07:10.000Z (over 4 years ago)
- Last Synced: 2024-06-19T03:10:09.594Z (7 months ago)
- Topics: colly, golang, spider
- Language: Go
- Homepage:
- Size: 36.7 MB
- Stars: 13
- Watchers: 3
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# 掘金专栏小爬虫
> 声明：掘金对于文章侵权是严厉打击的，不管你在哪搞事情，都能搞掉你。
这个脚本很简单，爬取掘金专栏详情页并把内容转成Markdown之后保存在本地。这是在写 [极速入门Go并爬取掘金专栏 | 🏆 技术专题第二期](https://juejin.im/post/6860522117423857678) 时写的小玩意。
## 安装
### 下载 Tab
```sh
$ brew tap youngjuning/juejin-spider https://github.com/youngjuning/homebrew-juejin-spider.git
```
### 下载脚本
```sh
$ brew install youngjuning/juejin-spider/juejin
```
## 升级
```sh
$ brew upgrade youngjuning/juejin-spider/juejin
```
## 使用
```sh
$ juejin -post 6860522117423857678 -root ~/Desktop
```
```sh
$ juejin -h
Usage of juejin:
-post string
文章编号 (default "6859538537830858759")
-root string
文件保存的根目录 (default "/Users/yangjunning/juejin")
```
## 打包并使用Homebrew发布脚本
> 感谢 [使用HomeBrew发布脚本](https://www.jianshu.com/p/e88831aac62a) 这篇文章。
### 1、打包成可执行文件
```sh
$ go build juejin.go
```
### 2、将可执行文件打包成 tar.gz 的格式
```shell
$ tar zcvf juejin_0.0.1.tar.gz juejin
```
上传到git，供配方软连接到这个脚本文件。
### 3、使用 `brew create --tab user/repo` 创建药方
```sh
$ brew create \
https://github.com/youngjuning/homebrew-juejin-spider/raw/master/juejin_0.0.1.tar.gz \
--tap youngjuning/homebrew-juejin-spider
```
我们需要对安装方式做一下调整:
```ruby
def install
bin.install "juejin"
end
```
做完这些操作后，保存，提交到git上。
### 4、安装脚本
```shell
$ brew install youngjuning/juejin-spider/juejin
```
执行 `juejin -h` 检查是否成功：
```shell
$ juejin -h
Usage of juejin:
-post string
文章编号 (default "6859538537830858759")
-root string
文件保存的根目录 (default "/Users/yangjunning/juejin")
```