https://github.com/tntmeijs/searchengine
Simple search engine using .NET, NodeJS, and PostgreSQL.
https://github.com/tntmeijs/searchengine
csharp dotnet javascript nodejs postgresql search-engine
Last synced: 3 months ago
JSON representation
Simple search engine using .NET, NodeJS, and PostgreSQL.
- Host: GitHub
- URL: https://github.com/tntmeijs/searchengine
- Owner: tntmeijs
- Created: 2020-08-02T11:39:06.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2022-12-10T10:10:20.000Z (over 3 years ago)
- Last Synced: 2025-01-15T06:49:55.185Z (over 1 year ago)
- Topics: csharp, dotnet, javascript, nodejs, postgresql, search-engine
- Language: C#
- Homepage:
- Size: 5.08 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Welcome
Hi there, this repository is a small personal project of mine. One day I wondered how search engines such as Google, DuckDuckGo, and Bing! worked. This is why I set out on a quest to build my own simple search engine.
The initial goal of this project was to build a working search engine. I did not feel the need to polish it and really turn it into a usable search engine. This was just a project to help me learn NodeJS, .NET, and PostgreSQL.
Demonstration: [https://www.youtube.com/watch?v=b5r_njqtAkc](https://www.youtube.com/watch?v=b5r_njqtAkc)
# Known issues / future improvements
- Web crawler sometimes fails to respect more complex robots.txt files.
- Use multithreading to improve overall crawl speed. Indexing pages can be easily done in parallel.
- Search engine web pages could use style improvements. It's pretty dull right now.
- The PostgreSQL database is a bit of a mess, can be split up into multiple tables to make it easier to manage.
- Back-up system to avoid losing indexed data.
- Implement [http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf](Page Rank) to improve search results.
- Split the crawler and search engine backend database into two to avoid writing to a production database.
# Structure
The search engine is powered by a PostgreSQL database. A custom web crawler is used to index websites and save them to the database.
This database is then used by the search engine server to display results.
- `/WebScraper` contains the web crawler, folder needs to be renamed as it is not a scraper.
- `/WebServer` contains the NodeJS / ExpressJS webserver used to power the search engine.
# Contributing
I always welcome pull-requests and issues. So if you have any suggestions or improvements, feel free to let me know by submitting a ticket on GitHub.
Cheers!