https://github.com/jakubgania/github-api-scraper-producer-consumer
https://github.com/jakubgania/github-api-scraper-producer-consumer
api aws container docker graphql http postgresql producer-consumer pulumi python3 redis sql uv
Last synced: 6 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/jakubgania/github-api-scraper-producer-consumer
- Owner: jakubgania
- Created: 2025-08-13T21:33:32.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2025-09-29T23:50:43.000Z (6 months ago)
- Last Synced: 2025-09-30T01:25:33.997Z (6 months ago)
- Topics: api, aws, container, docker, graphql, http, postgresql, producer-consumer, pulumi, python3, redis, sql, uv
- Language: Python
- Homepage:
- Size: 82 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# github-api-scraper-producer-consumer
This is a scraper project for the GitHub GraphQL API, built with a producer–consumer architecture.
Important Information ⚠️
If you want to run this app on your MacBook for several hours or days, be sure to keep the following in mind. Connect you MacBook to a power source and then go to:
System settings -> Options
And activate the following option:
"Prevent automatic sleeping on power adapter when the display is off"
### If you want to view or test the dashboard locally
Go to this folder:
```bash
dashboard-analytics/aws
```
Then run this command:
```bash
python3 -m http.server 8000
```
Then open in your browser: `http://localhost:8000/index.html`
### Docker commands
```bash
docker build -t api-github-scraper .
```
```bash
docker-compose up -d
```
```bash
docker-compose down -v
```
```bash
docker run \
--name github-app-1 \
--label type=worker \
--network api-github-scraper \
-e GITHUB_API_TOKEN=your-github-api-token \
-e POSTGRES_DSN="postgresql://postgres:postgres@github_postgres:5432/postgres" \
-e REDIS_HOST="redis-container" \
-e CONTAINER_ID=github-app-1
--memory 128m \
api-github-scraper
```
```bash
docker ps --filter "label=type=worker"
```
```bash
docker run -d --name redisinsight -p 5540:5540 redis/redisinsight:latest
```
```bash
docker ps -a
```
```bash
docker rm container-name
```
### Resources
[https://github.com/astral-sh/uv](https://github.com/astral-sh/uv)
[https://formulae.brew.sh/formula/pulumi](https://formulae.brew.sh/formula/pulumi)
[https://formulae.brew.sh/formula/awscli](https://formulae.brew.sh/formula/awscli)
[https://hub.docker.com/r/redis/redisinsight](https://hub.docker.com/r/redis/redisinsight)
[https://hub.docker.com/_/postgres](https://hub.docker.com/_/postgres)
[https://docs.github.com/en/graphql](https://docs.github.com/en/graphql)
[https://github.com/agronholm/apscheduler](https://github.com/agronholm/apscheduler)