Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/fanqingsong/web_full_stack_application
show full stack technology applications : Scrapy + webservice[restful] + websocket + VueJS + MongoDB
https://github.com/fanqingsong/web_full_stack_application
mongodb mongoose requests restifyjs scrapy socketio vue websocket
Last synced: 17 days ago
JSON representation
show full stack technology applications : Scrapy + webservice[restful] + websocket + VueJS + MongoDB
- Host: GitHub
- URL: https://github.com/fanqingsong/web_full_stack_application
- Owner: fanqingsong
- Created: 2018-06-22T15:27:22.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2022-12-08T12:00:48.000Z (almost 2 years ago)
- Last Synced: 2024-10-12T19:10:59.433Z (about 1 month ago)
- Topics: mongodb, mongoose, requests, restifyjs, scrapy, socketio, vue, websocket
- Language: JavaScript
- Homepage:
- Size: 5.07 MB
- Stars: 16
- Watchers: 3
- Forks: 3
- Open Issues: 26
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Full Stack Web Application
### Purpose
This is a demo project for showing how one full stack web application should be like.
### Introduction
features:
- [X] scrawler get data into database
- [X] webserver push data to browser periodically
---
## Demo
## Architecture
```
+-----------+ +------------+ +------------+ +--------------+
| | frontend | | | | data | |
| Browser +<----------------+ Vue Server| | RestAPI +<--------------+ Scrapy |
| | | | | | | |
+------+----+ +------------+ +------+-----+ +--------------+
^ |
| |
| |data
| |
| |
| v
| +-------------+ +------+------+
| data | | data | |
+---------------------+ Websocket +<-------------+ MongoDB |
| | | |
+-------------+ +-------------+
```
## Workflow
1. scratch quotes from scrapy offical demo site(quotes), save to database(MongoDB), and show them through web.
2. Front end is vue.js project, including WebSocket client(socketio) component.
3. Back end is websocket server, which query data form MongoDB through ORM mongoose every period(1s), pushing quotes data to front end.
4. data picker use scrapy to crawl quotes to save to MongoDB.
---
## Technology Stack
Scrapy + webservice[restful] + websocket + VueJS + MongoDB
| Component | Description |
| ----------- | ----------- |
| Scrapy | scratch data from web |
| webservice | serve restapi for data storation and fetch |
| websocket | push data to browser |
| VueJS | frontend |
| MongoDB | DB |
---
## Install
``` bash
./bin/install.sh
```
## Run
``` bash
# run all services
./bin/start.sh
# stop all services
./bin/stop.sh
# start scrapy
./bin/start_scrapy.sh
```
## Reference
- [x] [mongodb install](https://www.howtoforge.com/tutorial/install-mongodb-on-ubuntu-18.04/)