{"id":16391657,"url":"https://github.com/timzatko/fiit-vinf-1","last_synced_at":"2026-01-16T00:40:10.508Z","repository":{"id":38472044,"uuid":"212027730","full_name":"timzatko/fiit-vinf-1","owner":"timzatko","description":"School project - data crawling, storing using ElasticSearch and visualisation.","archived":false,"fork":false,"pushed_at":"2023-01-06T06:23:13.000Z","size":4207,"stargazers_count":0,"open_issues_count":20,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-08T18:14:18.831Z","etag":null,"topics":["angular","crawler","elasticsearch"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/timzatko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-01T06:32:49.000Z","updated_at":"2021-05-12T09:23:31.000Z","dependencies_parsed_at":"2023-02-05T12:46:31.517Z","dependency_job_id":null,"html_url":"https://github.com/timzatko/fiit-vinf-1","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timzatko%2Ffiit-vinf-1","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timzatko%2Ffiit-vinf-1/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timzatko%2Ffiit-vinf-1/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timzatko%2Ffiit-vinf-1/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/timzatko","download_url":"https://codeload.github.com/timzatko/fiit-vinf-1/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246933384,"owners_count":20857055,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["angular","crawler","elasticsearch"],"created_at":"2024-10-11T04:46:57.323Z","updated_at":"2026-01-16T00:40:10.465Z","avatar_url":"https://github.com/timzatko.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FIIT VINF 1\n\n[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white)](https://github.com/pre-commit/pre-commit)\n\nThis project is an implementation of a school course assignment (Information Retrieval at Faculty of Informatics and Information Technologies Slovak Technical University in Bratislava).\nThe simplified version of the assignment:\n\n1. Crawl a large amount of structure data\n1. Use Elastic Search as a data storage\n1. Make some non-trivial Elasticsearch queries\n1. Create a user interface to browse the data\n\n**Note:** The overall solution is not production-ready - eg. there should be some kind of API between the frontend and the Elasticsearch (so the Elasticsearch endpoint is not publicly exposed). However, this was not the aim of this project.\n\nI decided to crawl books from [https://www.barnesandnoble.com/](https://www.barnesandnoble.com/).\n\n## Preview\n\n![Preview](./assets/preview.gif)\n\n## Prerequisites\n\n- [Docker](https://www.docker.com/)\n- [Node](https://nodejs.org/en/) (use version `v11.12.0`)\n- [pre-commit](https://pre-commit.com/), don't forget to run pre-commit install in the repository root\n\n## Running the App\n\nFirstly install the dependencies (for the crawler and frontend).\n\n```bash\nnpm install\n```\n\nThen the app can be launched via docker-compose.\n\n```bash\ndocker-compose up\n```\n\n- App (frontend) will be running on [http://localhost:8000](http://localhost:8000)\n- Elasticsearch will be running on [http://localhost:9200](http://localhost:9200)\n- Kibana will be running on [http://localhost:5601](http://localhost:5601)\n\nAt this time, the website is empty, as there are not any data in the database, use crawler to populate it with the data.\n\n## Crawler\n\nCrawler crawls various items (mainly books, but also CDs, e-books etc.) from [https://www.barnesandnoble.com/](https://www.barnesandnoble.com/).\nIt fetches the sitemap and then it collects all item data from the links. It parses the DOM of the website to extract the data by using CSS selectors.\nIt also does the preprocessing of the data like removing the whitespaces or formatting of the numerical values.\n\n### Usage\n\n1. Navigate to the _crawler_ directory `cd crawler`\n2. Run the crawler `npm start`. This will fetch the sitemap and will crawl all sites in parallel (websites and crawled data are in `crawler/out` and `crawler/out-html`).\n3. Setup the _items_ index in Elasticsearch by running `sh setup/elasticsearch-index.sh`\n4. Import data to the Elastic `npm run export` (the Elasticsearch need to be running), the data will be sent in multiple bulks.\n\n## Development\n\nPrettier is run before every commit ensuring that the code looks best, it can also be run manually via `npm run prettier` in the root of the repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimzatko%2Ffiit-vinf-1","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimzatko%2Ffiit-vinf-1","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimzatko%2Ffiit-vinf-1/lists"}