https://github.com/pradumnasaraf/scrapy
Scrapy is a CLI tool used to scrape data from various websites
https://github.com/pradumnasaraf/scrapy
cli cobra-cli golang goquery scraper
Last synced: 3 months ago
JSON representation
Scrapy is a CLI tool used to scrape data from various websites
- Host: GitHub
- URL: https://github.com/pradumnasaraf/scrapy
- Owner: Pradumnasaraf
- License: apache-2.0
- Created: 2023-05-18T18:49:39.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-07-27T19:04:09.000Z (about 2 years ago)
- Last Synced: 2024-10-04T22:01:13.962Z (about 1 year ago)
- Topics: cli, cobra-cli, golang, goquery, scraper
- Language: Go
- Homepage: https://pkg.go.dev/github.com/Pradumnasaraf/scrapy
- Size: 31.3 KB
- Stars: 4
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
Awesome Lists containing this project
README
# Scrapy
**Scrapy** is a CLI tool used to scrape data from various websites. To install the Scrapy CLI, use the command `go install github.com/Pradumnasaraf/scrapy@latest`. Go will automatically install it in your `$GOPATH/bin` directory, which should be in your `$PATH`.
Once installed, you can use the `scrapy` CLI command. To confirm installation, type `scrapy` at the command line.
> **Note** If you are getting an error like `command not found: scrapy`, then you need to add `$GOPATH/bin` to your `$PATH` environment variable. For that you can refer to [this](https://gist.github.com/Pradumnasaraf/ca6f9a0507089a4c44881446cdda4aa3).
## ⭐️ Features
- Scape Ebay products and their prices
- Scrape TechCrunch articles in different categories
## 📝 Usage
```
Usage:
scrapy [command] [Flags]
Available Commands:
ebay [query] Scrape Ebay products and their prices.
ebay [query] --page [Number of pages]
tc Scrape TechCrunch articles in different categories.
tc --category [Category name]
```
eg: `scrapy ebay "iphone 12 mini" --page 5`
eg: `scrapy tc --category "startups"`
## 📜 License
This project is licensed under the Apache-2.0 license - see the [LICENSE](LICENSE) file for details.
## 🛡 Security
If you discover a security vulnerability within this project, please check the [SECURITY](SECURITY.md) for more information.