https://github.com/letieu/ads-transparency-craw
https://github.com/letieu/ads-transparency-craw
Last synced: over 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/letieu/ads-transparency-craw
- Owner: letieu
- Created: 2023-12-31T16:14:28.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2024-01-23T04:42:09.000Z (over 2 years ago)
- Last Synced: 2025-01-31T05:53:35.947Z (over 1 year ago)
- Language: TypeScript
- Size: 202 KB
- Stars: 1
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Crawl adstransparency.google.com
## Run local development server
- Setup
```bash
npm install
cp .env.example .env # and edit
# Start cronicle
docker compose -f compose-dev.yaml up cronicle
# Login to cronicle to create API key (http://localhost:3012), default user/pass: admin/admin
# Init cronicle events
API_KEY=xxx CRAWLER_HOST=http://host.docker.internal:3000 ./init-cronicle_events.sh
# Update .env file: CRONICLE_CRAWL_EVENT_ID, CRONICLE_API_KEY
```
- Start development server
```bash
docker compose -f compose-dev.yaml up -d
npm run start
```
## Run production server
- Setup and run
```bash
cp .env.example .env
# Start cronicle
docker compose -f compose-prod.yaml up cronicle
# Login to cronicle to create API key (http://localhost:3012), default user/pass: admin/admin
# Init cronicle events
API_KEY=xxx ./init-cronicle_events.sh
# Update .env file: CRONICLE_CRAWL_EVENT_ID, CRONICLE_CRAWL_DOMAIN_EVENT_ID, CRONICLE_API_KEY and database details
# Start production server
docker compose -f compose-prod.yaml up -d
# Check logs
docker compose -f compose-prod.yaml logs crawler -f
```