{"id":16548699,"url":"https://github.com/pourmand1376/crawler","last_synced_at":"2026-05-07T00:38:42.819Z","repository":{"id":38005403,"uuid":"174552585","full_name":"pourmand1376/Crawler","owner":"pourmand1376","description":"Simple Crawler, Indexer and Search Engine Web Application","archived":false,"fork":false,"pushed_at":"2023-05-31T20:28:38.000Z","size":21431,"stargazers_count":0,"open_issues_count":2,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-14T16:05:29.442Z","etag":null,"topics":["crawler","csharp","csharp-code","dotnet","mvc"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pourmand1376.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-08T14:33:09.000Z","updated_at":"2022-04-16T11:56:59.000Z","dependencies_parsed_at":"2024-11-14T20:36:42.542Z","dependency_job_id":"d4701c86-0e61-4174-b66c-ed5e299cc5dc","html_url":"https://github.com/pourmand1376/Crawler","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pourmand1376%2FCrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pourmand1376%2FCrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pourmand1376%2FCrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pourmand1376%2FCrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pourmand1376","download_url":"https://codeload.github.com/pourmand1376/Crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241852158,"owners_count":20030969,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","csharp","csharp-code","dotnet","mvc"],"created_at":"2024-10-11T19:26:45.484Z","updated_at":"2026-05-07T00:38:42.765Z","avatar_url":"https://github.com/pourmand1376.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Crawler\nSimple Crawler and Indexer and Search Engine Web Application\n![](https://github.com/pourmand1376/Crawler/blob/master/Demo/Demo1.gif)\n![](https://github.com/pourmand1376/Crawler/blob/master/Demo/Demo2.gif)\n\n\n![release](https://img.shields.io/github/license/pourmand1376/Crawler.svg)\n[![Build status](https://ci.appveyor.com/api/projects/status/0sgkbd7r0lf9cf2r?svg=true)](https://ci.appveyor.com/project/pourmand1376/crawler)\n\n\n## Nuget Restore\nJust open the project and right click the solution and choose **nuget** package restore. Wait till package restore completes. \n\n## Configuration\n1. Build and run the first project called Crawler. It uses its seed and downlaods the sites recursively (_Breath First Search_) and stores it in ***Data.Db*** and ***Crawler.Db*** file. Whenever you feel the gathered data is enough, simply close the program.  \n\n2. Build and run the second project called Indexer. You should copy ***Crawler.Db*** file from previous section here. After opening the program, It starts indexing the downloaded data and generates three files ***Sites.Db***, ***TitleIndex.Db***, and ***BodyIndex.Db***.\n\n3. Copy files generated from previous section to **App_Data** folder. \n\nEnjoy. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpourmand1376%2Fcrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpourmand1376%2Fcrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpourmand1376%2Fcrawler/lists"}