https://github.com/feluelle/pastebin-crawler-lib
A library for web crawling http://pastebin.com
https://github.com/feluelle/pastebin-crawler-lib
pastebin web-crawling
Last synced: about 1 year ago
JSON representation
A library for web crawling http://pastebin.com
- Host: GitHub
- URL: https://github.com/feluelle/pastebin-crawler-lib
- Owner: feluelle
- License: mit
- Created: 2016-12-27T21:32:44.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2017-11-17T20:35:33.000Z (over 8 years ago)
- Last Synced: 2025-03-16T16:48:32.559Z (about 1 year ago)
- Topics: pastebin, web-crawling
- Language: C#
- Homepage:
- Size: 11.7 KB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# PastebinCrawlerLib
A library for web crawling http://pastebin.com
## Is this legal?
According to pastebin's offical website it is.
See http://pastebin.com/faq#17
```
I got blocked! Can I scrape your website?
Yes, but we do limit the amount of requests that people can make,
so it is very possible that you get blocked from time to time.
If you want to scrape our platform more intensely,
we have a custom scraper API available where we can whitelist your IP, so you don't get blocked anymore.
This feature is only available for LIFETIME PRO users. To learn more, visit our scraping page.
```
See its FAQ section http://pastebin.com/faq for more information.
## ToDo
- pastebin-crawler-lib [contains PastebinCrawler, PastebinDocument]
- text-analystics-api [returns {name:string, result:ICollection}]
if contains password then split by ' '
if contains email then
- pastebin-data-sniffer (+ Filter)