https://github.com/harryandriyan/warta-scrap
Indonesia Index News Crawler, including 10 online media
https://github.com/harryandriyan/warta-scrap
python scraper scrapy
Last synced: 11 months ago
JSON representation
Indonesia Index News Crawler, including 10 online media
- Host: GitHub
- URL: https://github.com/harryandriyan/warta-scrap
- Owner: harryandriyan
- Created: 2017-08-23T09:30:16.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2018-10-12T02:45:36.000Z (over 7 years ago)
- Last Synced: 2024-07-29T09:33:24.733Z (almost 2 years ago)
- Topics: python, scraper, scrapy
- Language: Python
- Homepage:
- Size: 394 KB
- Stars: 76
- Watchers: 6
- Forks: 28
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- Awesome-Indonesia-NLP - Crawler
README
# warta-scrap
Indonesia Index News Crawler, including 10 online
### Online Media List:
- Detik.com
http://news.detik.com/indeks
- Republika.co.id
http://www.republika.co.id/indeks
- Viva.co.id
http://www.viva.co.id/indeks
- Kompas.com
http://indeks.kompas.com/
- Antaranews.com
http://www.antaranews.com/terkini
- Tempo.co
https://www.tempo.co/indeks
- Okezone.com
http://index.okezone.com/
- Liputan6.com
http://www.liputan6.com/indeks
- Merdeka.com
https://www.merdeka.com/berita-hari-ini/
- Tirto.id
https://tirto.id/indeks
### Installation :
Open Terminal, and clone this repo:
> git clone https://github.com/harryandriyan/warta-scrap
Go to project folder
> cd warta-scrap
Setup virtualenv
> virtualenv venv
Activate virtualenv
> . venv/bin/activate
Install requirements
> pip install -r requirements.txt
### How to use
Open the specific project, example
> cd republika
Run crawl command, example
> scrapy crawl republika -o sampleResult.json -t json