Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/amirsorouri00/my-search-engine
https://github.com/amirsorouri00/my-search-engine
Last synced: about 5 hours ago
JSON representation
- Host: GitHub
- URL: https://github.com/amirsorouri00/my-search-engine
- Owner: Amirsorouri00
- License: gpl-3.0
- Created: 2019-06-10T15:41:56.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2022-12-08T05:14:27.000Z (almost 2 years ago)
- Last Synced: 2023-03-04T03:18:34.115Z (over 1 year ago)
- Language: HTML
- Size: 103 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# my-search-engine
# web-search-engine-ui
# web-search-engnin
## DEMO
## INSTALL AND RUN
### REQUIREMENTS
This tool requires *Python3+* and the web search engine API (see link above).
### WITH PIP
```
& git clone https://github.com/Amirsorouri00/my-search-engine.git
& cd my-search-engine
& python3.7 -m venv search-engine-venvironment
& source search-engine-venvironment/bin/activate
& pip install -r requirements.txt
$ python manage.py makemigrations
$ python manage.py migrate
```
And, in another CLI :
```
sudo docker-compose up
```
### WITH DOCKER
## USAGE AND EXAMPLES
To use the search engine, just type this endpoint in your web browser : http://localhost/
## USEFULL LINKS
* https://www.freecodecamp.org/news/elasticsearch-with-django-the-easy-way-909375bc16cb/
* https://github.com/adamwattis/elasticsearch-example
* https://docs.scrapy.org/en/latest/topics/benchmarking.html?highlight=redirect%20enabled
* https://en.wikipedia.org/wiki/Sitemaps
* https://docs.scrapy.org/en/latest/topics/extensions.html?highlight=pagecount#extension-settings
* https://docs.scrapy.org/en/latest/topics/practices.html#running-multiple-spiders-in-the-same-process
* https://www.digitalocean.com/community/tutorials/how-to-crawl-a-web-page-with-scrapy-and-python-3
* https://opensourcehacker.com/2011/03/08/installing-and-using-scrapy-web-crawler-to-search-text-on-multiple-sites/
* https://github.com/naqushab/SearchEngineScrapy
* https://github.com/scrapinghub/frontera
* http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-billion-webpages-in-40-hours/
* https://github.com/BruceDone/awesome-crawler
## LICENCE
MIT