Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/amirsorouri00/dsl-se
This is a MVP provided based on the "Search Engine And Data Mining" Course. The idea behind this project is the forked project which its link provided is
https://github.com/amirsorouri00/dsl-se
container crawler distributed-systems docker docker-compose elasticsearch pagerank search-engine
Last synced: about 1 month ago
JSON representation
This is a MVP provided based on the "Search Engine And Data Mining" Course. The idea behind this project is the forked project which its link provided is
- Host: GitHub
- URL: https://github.com/amirsorouri00/dsl-se
- Owner: Amirsorouri00
- License: apache-2.0
- Created: 2019-07-04T10:03:19.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2022-12-08T05:18:18.000Z (about 2 years ago)
- Last Synced: 2023-03-04T03:18:32.680Z (almost 2 years ago)
- Topics: container, crawler, distributed-systems, docker, docker-compose, elasticsearch, pagerank, search-engine
- Language: HTML
- Homepage: https://github.com/Amirsorouri00/web-search-engine
- Size: 3.59 MB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 19
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Distributed System - Search Engine
This Part and all other foo bar part of this document will be completed ASAP. Foobar is a Python library for dealing with word pluralization. feel free to contibute and ask.
## Installation
### Django Crawler & DSTea
Use the package manager [pip](https://pip.pypa.io/en/stable/) to install foobar.
```bash
$ cd [crawler_django, dstea]
$ python -m venv py_environment
$ source py_environment/bin/activate
$ pip install -r requirements.txt
$ python manage.py runserver
$ Done!!
```
### Docker-Compose
```bash
$ cd composer-docker
$ sudo docker-compose up
```
## Usage
```python
import foobar
foobar.pluralize('word') # returns 'words'
foobar.pluralize('goose') # returns 'geese'
foobar.singularize('phenomena') # returns 'phenomenon'
```
## Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.
## License
[Apache License 2.0](https://choosealicense.com/licenses/apache-2.0/)
## Commands
```bash
curl -d "url=http://www.laravel.com" -X POST http://localhost:8080/crawler/crawl/
```
## Usefull Links
* https://blog.sneawo.com/blog/2017/09/07/how-to-install-pillow-psycopg-pylibmc-packages-in-pythonalpine-image/
* https://www.peterbe.com/plog/synonyms-with-elasticsearch-dsl
## Images
## Usefull Links
* https://www.elastic.co/guide/en/elasticsearch/guide/master/dynamic-indices.html
* https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-analyzers.html
* http://www.devinline.com/2018/09/elasticsearch-inverted-index-and-its-storage.html
* https://gist.github.com/HonzaKral/d90d344bca18ffa71139ac11b9f83124
* https://faculty.math.illinois.edu/~riveraq2/teaching/simcamp16/PageRankwithPython.html
* https://github.com/mtusman/stock-eagle/blob/master/stockeagle/eagle.py