Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/uranusx86/dcard-crawler-analyzer
get Dcard & Meteor forum content and analyze !
https://github.com/uranusx86/dcard-crawler-analyzer
crawl crawler dcard nlp python word-cloud word-count word-frequency
Last synced: 2 days ago
JSON representation
get Dcard & Meteor forum content and analyze !
- Host: GitHub
- URL: https://github.com/uranusx86/dcard-crawler-analyzer
- Owner: uranusx86
- License: mit
- Created: 2019-03-23T02:52:11.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2019-07-18T14:21:38.000Z (over 5 years ago)
- Last Synced: 2024-11-20T11:08:21.985Z (2 months ago)
- Topics: crawl, crawler, dcard, nlp, python, word-cloud, word-count, word-frequency
- Language: Python
- Homepage:
- Size: 40 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Dcard-Crawler-Analyzer
get Dcard & Meteor forum content and analyze !
And also provide a docker image version
# Dependency
python3
## pypi package
* flask
* flask_script
* flask_migrate
* Flask-SQLAlchemy
* gunicorn
* requests
* cfscrape
* beautifulsoup4
* jieba
## apt package
* nodejs 8+
# Install
```bash=
# install dependency
apt-get install nodejs
pip install requirements.txt # it highly recommend install packages in the virual env
# environment variable
export APP_SETTINGS="config.DevelopmentConfig"
export DATABASE_URL="sqlite:///WHERE_YOU_WANT_PUT_DB"
# database migration
cd app
python3 manage.py db init
python3 manage.py db migrate
python3 manage.py db upgrade
```
# Run
```bash=
cd app
python3 dcard.py # for Dcard
python3 meteor.py # for Meteor
```
# Docker image
You can also build crawler from dockerfile
```bash=
docker build . --tag uranusx86/forum_crawler --no-cache
docker run -dt --name forumcrawler -p 80:8000 uranusx86/forum_crawler
```