Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/yasoob/torrents-crawler
This is a scrapy project in which I have implemented several crawlers for different torrent and direct link websites.
https://github.com/yasoob/torrents-crawler
Last synced: 2 months ago
JSON representation
This is a scrapy project in which I have implemented several crawlers for different torrent and direct link websites.
- Host: GitHub
- URL: https://github.com/yasoob/torrents-crawler
- Owner: yasoob
- License: mpl-2.0
- Created: 2013-09-22T13:58:22.000Z (about 11 years ago)
- Default Branch: master
- Last Pushed: 2020-10-01T10:24:07.000Z (about 4 years ago)
- Last Synced: 2024-07-05T09:35:51.621Z (3 months ago)
- Language: Python
- Size: 20.5 KB
- Stars: 59
- Watchers: 9
- Forks: 33
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Torrents-Crawler
================
This is a scrapy project in which I have implemented several crawlers for different torrent and direct link websites. This project crawls for the title and the url only right now because this was my only requirement.
### Dependency:
- Scrapy ```pip install scrapy```
### Websites currently supported:
- Piratebay
- Kickass
- 1337x
- SkidrowCrack
- TorrentDownloads
- SumoTorrent
- BitSnoop
### How to run a crawler:
If you want to run a crawler then goto the top level directory which is ```piratebay``` and issue this command:
```$ scrapy crawl piratebay``` (see TODO)
This command will run the piratebay crawler.
Names for all of the crawlers are:
- piratebay
- leetx
- bitsnoop
- sumotorrent
- torrentdownloads
- kickass
- skidrowcrack
### TODO:
Currently I have not added any pipeline so right now if we want to save the crawled data we will have to issue one of the following commands:
- ```scrapy crawl piratebay -o items.json -t json```
- ```scrapy crawl piratebay -o items.csv -t csv```
- ```scrapy crawl piratebay -o items.xml -t xml```
- ```scrapy crawl piratebay -o items.json -t jsonlines```
If anyone is willing to write an ItemPipeline then feel free to submit a PR
### How to add support:
If you want to add further support for some more websites then feel free to edit this crawler. Just go to ```piratebay/piratebay/spiders/piratebay_spider.py``` and edit that file. Copy one of the class and edit it to adapt for another website. Basically you will have to edit the xpaths, link to .txt file and the name of the crawler. After that copy all of the links which you want to crawl and paste them in a new text file in ```piratebay/links/```. Thats it. After that just run your crawler.
### Author:
- M.Yasoob Ullah Khalid ([email protected])
### Support:
[Official scrapy docs](http://doc.scrapy.org/en/latest/intro/tutorial.html). If there is an issue with the scripts present here do open an issue or if you want me to add support for a website do tell me.
### Contribute:
You can contribute literally anything. Whether a new crawler or some small optimisations. Just do a PR and make this project grow.
**Note:** This project is not maintained anymore.