https://github.com/kant2002/ncrawler

Web Crawler written in C#
https://github.com/kant2002/ncrawler

crawler scrapper

Last synced: 2 months ago
JSON representation

Web Crawler written in C#

• Host: GitHub
• URL: https://github.com/kant2002/ncrawler
• Owner: kant2002
• License: lgpl-2.1
• Created: 2015-01-01T21:12:26.000Z (almost 10 years ago)
• Default Branch: master
• Last Pushed: 2019-10-13T21:16:40.000Z (about 5 years ago)
• Last Synced: 2024-10-03T12:31:17.461Z (3 months ago)
• Topics: crawler, scrapper
• Language: C#
• Homepage:
• Size: 5.12 MB
• Stars: 41
• Watchers: 14
• Forks: 14
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

README

        
ncrawler

========

Copy of NCrawler from http://ncrawler.codeplex.com/

Simple and very efficient multithreaded web crawler with pipeline based processing written in C#. 

Contains HTML, Text, PDF, and IFilter document processors and language detection(Google). 

Easy to add pipeline steps to extract, use and alter information.

## Build Nuget packages

Create debug packages

    .\Build.ps1 -VersionSuffix build002

Create release packages

    .\Build.ps1