https://github.com/intellltech/steam-scraping-application
Info of Items Sellers Scraping app
https://github.com/intellltech/steam-scraping-application
django flask postgresql-database python scraping web-application
Last synced: 3 months ago
JSON representation
Info of Items Sellers Scraping app
- Host: GitHub
- URL: https://github.com/intellltech/steam-scraping-application
- Owner: intellltech
- License: mit
- Created: 2024-12-11T08:46:29.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-12-11T10:07:39.000Z (7 months ago)
- Last Synced: 2025-02-14T06:35:50.059Z (5 months ago)
- Topics: django, flask, postgresql-database, python, scraping, web-application
- Language: Python
- Homepage: https://steamcommunity.com/market/
- Size: 2.31 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## Item Seller Info from [Steam Market](https://steamcommunity.com/market/) Scraping app
### requirements
- Python 3
- PostgreSQL + 150GB SSD Storage
#### create tables and indexes in psql
```sql
CREATE TABLE listings(
id SERIAL PRIMARY KEY,
item_name VARCHAR(200),
time timestamp,
price int,
owner_name VARCHAR(200),
owner_avatar VARCHAR(200),
profile_link VARCHAR(200),
CONSTRAINT owner_item_constraint UNIQUE(owner_name, item_name, price, profile_link)
);
CREATE INDEX ON listings(owner_name);
CREATE INDEX ON listings(item_name);
CREATE INDEX ON listings(price);
CREATE INDEX ON listings(profile_link);
```
#### populate database with users
set API_KEY environment variable, edit steamid range if necessary and run `databaseupdater.py`
#### collect listings from steam
run `listingcollector.py`. it takes items from `listings.json`. you can re-generate dictionary with
`helpers.collect_item_urls` and `helpers.parse_id`
#### web-server
flask code is contained in `app.py`
## Help 📮
- Discord: @intellltech
- Telegram: @intellltech