Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/william1nguyen/carlist-crawler-python
A Pipeline for extracting data from Carlist.my and load to ElasticSearch
https://github.com/william1nguyen/carlist-crawler-python
crawling-python elasticsearch etl-pipeline scrapy
Last synced: about 1 month ago
JSON representation
A Pipeline for extracting data from Carlist.my and load to ElasticSearch
- Host: GitHub
- URL: https://github.com/william1nguyen/carlist-crawler-python
- Owner: william1nguyen
- Created: 2024-04-23T08:39:09.000Z (8 months ago)
- Default Branch: master
- Last Pushed: 2024-04-28T13:36:48.000Z (8 months ago)
- Last Synced: 2024-08-01T20:45:51.034Z (5 months ago)
- Topics: crawling-python, elasticsearch, etl-pipeline, scrapy
- Language: Python
- Homepage:
- Size: 52.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## CRAWLY
Scrapy Scripts for scraping data
### Run Commands
* Clone Project and go to `crawly/`
```
$ git clone [email protected]:natalieconan/crawly.git
$ cd crawly
```* (Optional) To install pipenv with `Homebrew`:
```
$ brew install pipenv
```* Activate Python Virtual Env using `Pipenv` and install packages
```
$ pipenv shell
$ pipenv install
```* Finally, run spider for crawling
```
$ scrapy crawl ${spider_name}
```In this case `spider_name = carlist`, so run this command to start crawling:
```
$ scrapy crawl carlist
```