{"id":13694988,"url":"https://github.com/fanqingsong/web_full_stack_application","last_synced_at":"2025-03-21T10:31:44.498Z","repository":{"id":39825002,"uuid":"138318700","full_name":"fanqingsong/web_full_stack_application","owner":"fanqingsong","description":"show full stack technology applications : Scrapy + webservice[restful] + websocket + VueJS + MongoDB","archived":false,"fork":false,"pushed_at":"2022-12-08T12:00:48.000Z","size":5312,"stargazers_count":16,"open_issues_count":26,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-18T00:05:26.938Z","etag":null,"topics":["mongodb","mongoose","requests","restifyjs","scrapy","socketio","vue","websocket"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fanqingsong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-22T15:27:22.000Z","updated_at":"2022-03-06T05:20:16.000Z","dependencies_parsed_at":"2022-08-28T00:23:54.695Z","dependency_job_id":null,"html_url":"https://github.com/fanqingsong/web_full_stack_application","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fanqingsong%2Fweb_full_stack_application","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fanqingsong%2Fweb_full_stack_application/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fanqingsong%2Fweb_full_stack_application/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fanqingsong%2Fweb_full_stack_application/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fanqingsong","download_url":"https://codeload.github.com/fanqingsong/web_full_stack_application/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244777904,"owners_count":20508809,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mongodb","mongoose","requests","restifyjs","scrapy","socketio","vue","websocket"],"created_at":"2024-08-02T17:01:54.999Z","updated_at":"2025-03-21T10:31:43.434Z","avatar_url":"https://github.com/fanqingsong.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"## Full Stack Web Application\n\n### Purpose\n\nThis is a demo project for showing how one full stack web application should be like.\n\n### Introduction\n\nfeatures:\n- [X] scrawler get data into database\n- [X] webserver push data to browser periodically \n\n---\n\n## Demo\n\n![page](snapshot.png)\n\n## Architecture\n\n```\n+-----------+                 +------------+              +------------+               +--------------+\n|           |      frontend   |            |              |            |      data     |              |\n|   Browser +\u003c----------------+  Vue Server|              |   RestAPI  +\u003c--------------+    Scrapy    |\n|           |                 |            |              |            |               |              |\n+------+----+                 +------------+              +------+-----+               +--------------+\n       ^                                                         |\n       |                                                         |\n       |                                                         |data\n       |                                                         |\n       |                                                         |\n       |                                                         v\n       |                     +-------------+              +------+------+\n       |     data            |             |    data      |             |\n       +---------------------+  Websocket  +\u003c-------------+   MongoDB   |\n                             |             |              |             |\n                             +-------------+              +-------------+\n\n```\n\n## Workflow\n\n1. scratch quotes from scrapy offical demo site(quotes), save to database(MongoDB), and show them through web.\n2. Front end is vue.js project, including WebSocket client(socketio) component.\n3. Back end is websocket server, which query data form MongoDB through ORM mongoose every period(1s), pushing quotes data to front end.\n4. data picker use scrapy to crawl quotes to save to MongoDB.\n\n\n---\n\n## Technology Stack\n\nScrapy + webservice[restful] + websocket + VueJS + MongoDB\n\n| Component | Description |\n| ----------- | ----------- |\n| Scrapy | scratch data from web |\n| webservice | serve restapi for data storation and fetch |\n| websocket | push data to browser |\n|  VueJS | frontend |\n| MongoDB | DB |\n\n---\n\n## Install\n\n``` bash\n\n./bin/install.sh\n\n```\n\n## Run\n\n``` bash\n# run all services\n./bin/start.sh\n\n# stop all services\n./bin/stop.sh\n\n# start scrapy\n./bin/start_scrapy.sh\n```\n\n\n## Reference\n- [x] [mongodb install](https://www.howtoforge.com/tutorial/install-mongodb-on-ubuntu-18.04/)\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffanqingsong%2Fweb_full_stack_application","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffanqingsong%2Fweb_full_stack_application","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffanqingsong%2Fweb_full_stack_application/lists"}