https://github.com/intellltech/steam-scraping-application

Info of Items Sellers Scraping app
https://github.com/intellltech/steam-scraping-application

django flask postgresql-database python scraping web-application

Last synced: 30 days ago
JSON representation

Info of Items Sellers Scraping app

• Host: GitHub
• URL: https://github.com/intellltech/steam-scraping-application
• Owner: intellltech
• License: mit
• Created: 2024-12-11T08:46:29.000Z (about 1 month ago)
• Default Branch: main
• Last Pushed: 2024-12-11T10:07:39.000Z (about 1 month ago)
• Last Synced: 2024-12-11T10:36:57.821Z (about 1 month ago)
• Topics: django, flask, postgresql-database, python, scraping, web-application
• Language: Python
• Homepage: https://steamcommunity.com/market/
• Size: 0 Bytes
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
## Item Seller Info from [Steam Market](https://steamcommunity.com/market/) Scraping app

![Banner](https://github.com/intellltech/steam-scraping-application/blob/main/figures/a.png)

### requirements

 - Python 3

 - PostgreSQL + 150GB SSD Storage

#### create tables and indexes in psql

```sql

CREATE TABLE listings(

    id SERIAL PRIMARY KEY,

    item_name VARCHAR(200),

    time timestamp,

    price int,

    owner_name VARCHAR(200),

    owner_avatar VARCHAR(200),

    profile_link VARCHAR(200),

    CONSTRAINT owner_item_constraint UNIQUE(owner_name, item_name, price, profile_link)

);

CREATE INDEX ON listings(owner_name);

CREATE INDEX ON listings(item_name);

CREATE INDEX ON listings(price);

CREATE INDEX ON listings(profile_link);

```

#### populate database with users

set API_KEY environment variable, edit steamid range if necessary and run `databaseupdater.py`

#### collect listings from steam

run `listingcollector.py`. it takes items from `listings.json`. you can re-generate dictionary with 

`helpers.collect_item_urls` and `helpers.parse_id`

#### web-server

flask code is contained in `app.py` 

## Help 📮

- Discord: @intellltech

- Telegram: @intellltech