https://github.com/pourmand1376/crawler

Simple Crawler, Indexer and Search Engine Web Application
https://github.com/pourmand1376/crawler

crawler csharp csharp-code dotnet mvc

Last synced: about 1 month ago
JSON representation

Simple Crawler, Indexer and Search Engine Web Application

• Host: GitHub
• URL: https://github.com/pourmand1376/crawler
• Owner: pourmand1376
• License: mit
• Created: 2019-03-08T14:33:09.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2023-05-31T20:28:38.000Z (about 3 years ago)
• Last Synced: 2025-01-14T16:05:29.442Z (over 1 year ago)
• Topics: crawler, csharp, csharp-code, dotnet, mvc
• Language: JavaScript
• Homepage:
• Size: 20.4 MB
• Stars: 0
• Watchers: 2
• Forks: 2
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Crawler

Simple Crawler and Indexer and Search Engine Web Application

![](https://github.com/pourmand1376/Crawler/blob/master/Demo/Demo1.gif)

![](https://github.com/pourmand1376/Crawler/blob/master/Demo/Demo2.gif)

![release](https://img.shields.io/github/license/pourmand1376/Crawler.svg)

[![Build status](https://ci.appveyor.com/api/projects/status/0sgkbd7r0lf9cf2r?svg=true)](https://ci.appveyor.com/project/pourmand1376/crawler)

## Nuget Restore

Just open the project and right click the solution and choose **nuget** package restore. Wait till package restore completes. 

## Configuration

1. Build and run the first project called Crawler. It uses its seed and downlaods the sites recursively (_Breath First Search_) and stores it in ***Data.Db*** and ***Crawler.Db*** file. Whenever you feel the gathered data is enough, simply close the program.  

2. Build and run the second project called Indexer. You should copy ***Crawler.Db*** file from previous section here. After opening the program, It starts indexing the downloaded data and generates three files ***Sites.Db***, ***TitleIndex.Db***, and ***BodyIndex.Db***.

3. Copy files generated from previous section to **App_Data** folder. 

Enjoy.