https://github.com/letieu/fb-crawler
https://github.com/letieu/fb-crawler
Last synced: 10 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/letieu/fb-crawler
- Owner: letieu
- Created: 2023-08-27T15:53:50.000Z (almost 3 years ago)
- Default Branch: master
- Last Pushed: 2024-06-12T02:23:14.000Z (about 2 years ago)
- Last Synced: 2025-03-26T04:13:58.182Z (over 1 year ago)
- Language: TypeScript
- Size: 245 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# FB Crawl
## How to run
```sh
cp .env.example .env
docker-compose build
docker-compose up -d
```
## Check logs
```sh
# logs chrome
docker-compose logs chrome1 -f
docker-compose logs chrome2 -f
# logs worker
docker compose logs id-worker -f
docker compose logs detail-worker -f
# produce job
docker-compose exec exec board node dist/scripts/producers/post-id.js
docker-compose exec exec board node dist/scripts/producers/post-detail.js
```
## Crontab
```
0 0 * * * /usr/bin/docker compose -f /root/fb-crawler/docker-compose.yml exec board node dist/scripts/mark-expire-post.js >> /var/log/fb-crawl-expire.log
0 1,13 * * * /usr/bin/docker compose -f /root/fb-crawler/docker-compose.yml exec board node dist/scripts/producers/post-id.js >> /var/log/fb-crawl-id.log
0 2,14 * * * /usr/bin/docker compose -f /root/fb-crawler/docker-compose.yml exec board node dist/scripts/producers/like-page.js >> /var/log/fb-crawl-like-page.log
0 3,15 * * * /usr/bin/docker compose -f /root/fb-crawler/docker-compose.yml exec board node dist/scripts/producers/ads-id.js >> /var/log/fb-crawl-ads-id.log
0 4,16 * * * /usr/bin/docker compose -f /root/fb-crawler/docker-compose.yml exec board node dist/scripts/producers/post-detail.js >> /var/log/fb-crawl-detail.log
0 5,17 * * * /usr/bin/docker compose -f /root/fb-crawler/docker-compose.yml exec board node dist/scripts/producers/ads-detail.js >> /var/log/fb-crawl-ads-detail.log
```