Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/pourmand1376/crawler

Simple Crawler, Indexer and Search Engine Web Application
https://github.com/pourmand1376/crawler

crawler csharp csharp-code dotnet mvc

Last synced: 27 days ago
JSON representation

Simple Crawler, Indexer and Search Engine Web Application

Host: GitHub
URL: https://github.com/pourmand1376/crawler
Owner: pourmand1376
License: mit
Created: 2019-03-08T14:33:09.000Z (over 5 years ago)
Default Branch: master
Last Pushed: 2023-03-03T06:38:27.000Z (over 1 year ago)
Last Synced: 2023-03-04T05:03:37.035Z (over 1 year ago)
Topics: crawler, csharp, csharp-code, dotnet, mvc
Language: JavaScript
Homepage:
Size: 20.4 MB
Stars: 0
Watchers: 1
Forks: 0
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

        # Crawler

Simple Crawler and Indexer and Search Engine Web Application

![](https://github.com/pourmand1376/Crawler/blob/master/Demo/Demo1.gif)

![](https://github.com/pourmand1376/Crawler/blob/master/Demo/Demo2.gif)

![release](https://img.shields.io/github/license/pourmand1376/Crawler.svg)

[![Build status](https://ci.appveyor.com/api/projects/status/0sgkbd7r0lf9cf2r?svg=true)](https://ci.appveyor.com/project/pourmand1376/crawler)

## Nuget Restore

Just open the project and right click the solution and choose **nuget** package restore. Wait till package restore completes. 

## Configuration

1. Build and run the first project called Crawler. It uses its seed and downlaods the sites recursively (_Breath First Search_) and stores it in ***Data.Db*** and ***Crawler.Db*** file. Whenever you feel the gathered data is enough, simply close the program.  

2. Build and run the second project called Indexer. You should copy ***Crawler.Db*** file from previous section here. After opening the program, It starts indexing the downloaded data and generates three files ***Sites.Db***, ***TitleIndex.Db***, and ***BodyIndex.Db***.

3. Copy files generated from previous section to **App_Data** folder. 

Enjoy.