https://github.com/steffo99/unimore-bda-3-steffo
Terza attività di Big Data Analytics
https://github.com/steffo99/unimore-bda-3-steffo
data-science jupyter matplotlib pandas unimore-informatica
Last synced: 3 months ago
JSON representation
Terza attività di Big Data Analytics
- Host: GitHub
- URL: https://github.com/steffo99/unimore-bda-3-steffo
- Owner: Steffo99
- License: cc-by-sa-4.0
- Created: 2022-11-25T08:17:01.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-05-15T02:06:13.000Z (about 2 years ago)
- Last Synced: 2025-10-14T08:06:19.441Z (9 months ago)
- Topics: data-science, jupyter, matplotlib, pandas, unimore-informatica
- Language: Python
- Homepage:
- Size: 3.54 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 6
-
Metadata Files:
- Readme: README.ipynb
- License: LICENSE.txt
Awesome Lists containing this project
README
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\[ Stefano Pigozzi | Tema Data Analytics | Big Data Analytics | A.A. 2022/2023 | Unimore \\]"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2023-07-02T06:47:40.635561319Z",
"start_time": "2023-07-02T06:47:39.216048338Z"
}
},
"outputs": [],
"source": [
"%load_ext autoreload\n",
"%autoreload 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Ricerca di correlazioni nell'attività online relativa a videogiochi pubblicati sulla piattaforma Steam"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> ### Data processing e exploratory data analytics su dataset provenienti da più sorgenti\n",
">\n",
"> L’attività da svolgere consiste nel:\n",
">\n",
"> 1. Scegliere due o più dataset provenienti da due o più sorgenti.\n",
"> * Il dataset finale deve essere costituito almeno da due file.\n",
"> 2. Usando [pandas](https://pandas.pydata.org/) implementare le operazioni di data processing necessarie (principalmente join e selezioni) per mettere in collegamento i dataset e per preparare i dati al passo successivo\n",
"> 3. Usando pacchetti Python quali [pandas](https://pandas.pydata.org/), [scipy](https://scipy.org/), [matplotlib](https://matplotlib.org/) e [seaborn](https://seaborn.pydata.org/) implementare attività di data cleaning, exploratory data analysis estraendo dati statistici e di visualizzazione dei risultati attraverso il quale sia possibile \"raccontare qualcosa sui dati\" (storytelling), eventualmente partendo da dei quesiti di ricerca.\n",
"> L'uso dei pacchetti non deve necessariamente essere limitato alle istruzioni viste a lezione. Le documentazioni dei pacchetti stessi e i volumi messi a disposizione su Dolly fornisco spunti d’uso interessanti!\n",
"> 4. Produrre un notebook [Jupyter](https://jupyter.org/) che contenga:\n",
"> * una introduzione all’argomento scelto, alle sorgenti dati e agli obiettivi del progetto specificando eventualmente i quesiti di ricerca\n",
"> * una sezione per ogni fase del progetto di data analytics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sinossi"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In questa relazione si ricercano correlazioni tra vari parametri relativi all'attività online riguardante vari videogiochi per PC pubblicati sulla piattaforma di distribuzione [Steam](https://store.steampowered.com/), recuperando dati da varie fonti, quali la web API di [Steam](https://store.steampowered.com/) stesso, il progetto [SteamDB](https://steamdb.info/), il catalogo prezzi [IsThereAnyDeal](https://isthereanydeal.com/), e [Google Trends](https://trends.google.com/trends/), costruendo infine un funzione con discreta correlazione con il numero di giocatori attivi in un determinato giorno."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Introduzione"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[Steam](https://store.steampowered.com/) è una piattaforma di vendita e distribuzione videogiochi per PC creata da [Valve Corporation](https://www.valvesoftware.com/en/)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[Valve Corporation](https://www.valvesoftware.com/en/) pubblica quotidianamente [dati sul numero massimo di giocatori concorrenti](https://store.steampowered.com/charts/) di ciascun videogioco; questi dati vengono raccolti dal progetto indipendente [SteamDB](https://steamdb.info/), che li utilizza per mostrare la popolarità storica di ciascun prodotto."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Periodicamente, i videogiochi disponibili su [Steam](https://store.steampowered.com/) vengono scontati temporaneamente per incentivare gli utenti della piattaforma ad acquistarli; ciò in genere avviene tra le due e le cinque volte all'anno, principalmente nel periodo natalizio e attorno al solstizio di estate.\n",
"\n",
"Il progetto [SteamDB](https://steamdb.info/) monitora quotidianamente i prezzi di ciascun prodotto, aggregandoli per creare uno storico complessivo dei prezzi."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I videogiochi presenti su [Steam](https://store.steampowered.com/) possono essere anche acquistati attraverso siti di terze parti, che possono offrire prezzi o offerte diversi da quelli della piattaforma, pur sempre condividendone i servizi, come ad esempio [Humble Bundle](https://www.humblebundle.com/); il catalogo prezzi [IsThereAnyDeal](https://isthereanydeal.com/) si occupa di controllare periodicamente i prezzi su ciascun rivenditore e di aggregarli in un sito unico, creando poi uno storico globale dei prezzi migliori."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Gli sviluppatori dei videogiochi su [Steam](https://store.steampowered.com/) possono pubblicare periodicamente sulla piattaforma degli annunci relativi al proprio videogioco, tipicamente per informare i clienti della disponibilità di una nuova release."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Gli annunci pubblicati da ciascun prodotto sono recuperabili attraverso chiamate ad una [web API](https://developer.valvesoftware.com/wiki/Steam_Web_API#GetNewsForApp_.28v0002.29) pubblica."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Obiettivo dell'indagine"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"La popolarità di un videogioco è una metrica sociale, alimentata da tanti diversi fattori, che possono variare anche significativamente in base al genere in questione."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si ipotizza che uno di questi fattori sia la presenza di sconti, in quanto si è empiricamente notato che il numero di giocatori sembra aumentare in presenza di essi."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si ipotizza inoltre che il rilascio di nuovi contenuti porti a un picco improvviso seguito da una lenta diminuzione nel numero di giocatori concorrenti, circa corrispondente al tempo necessario per vedere le novità."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In questa indagine, si vuole provare a modellare una funzione che, utilizzando solo parametri esterni, ottenga una discreta correlazione con il numero di giocatori concorrenti di ciascun videogioco (senza ricorrere a tecniche di machine learning)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Oggetto dell'indagine"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Recuperare grandi quantità di dati relative a un vasto numero di prodotti diversi è difficoltoso, in quanto tutte le fonti che si desidera utilizzare implementano misure per prevenire lo scraping automatizzato."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Inoltre, la moltitudine di generi di videogiochi esistenti e di stili di comunicazione adottati dagli sviluppatori può introdurre notevole rumore e rendere questo studio impossibile. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si decide pertanto di concentrarsi esclusivamente sui prodotti con le seguenti caratteristiche:\n",
"\n",
"- ne è stata rilasciata la versione 1.0\n",
"- non hanno fattori fortemente sociali come origine della loro popolarità\n",
" - la loro componente principale non è il multigiocatore competitivo\n",
"- vengono scontati regolarmente\n",
"- non abusano della funzionalità di annunci per pubblicare informazioni non riguardanti il prodotto\n",
"- hanno venduto un numero significativo di copie\n",
" - hanno almeno 10000 recensioni pubblicate dai clienti"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si recupera quindi manualmente un piccolo sottoinsieme di dataset relativi a diversi prodotti con le caratteristiche indicate nella sezione precedente, descrivendo però accuratamente il metodo utilizzato per recuperare i dati in modo da poter estendere riproducibilmente la ricerca a qualsiasi altro prodotto."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I prodotti selezionati sono:\n",
"\n",
"- [Deep Rock Galactic](https://steamdb.info/app/548430/)\n",
"- [OMORI](https://steamdb.info/app/1150690/)\n",
"- [Potion Craft: Alchemist Simulator](https://steamdb.info/app/1210320/)\n",
"- [Untitled Goose Game](https://steamdb.info/app/837470/)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In più, al fine di mostrare quanta differenza può esserci tra un prodotto e l'altro, si recuperano anche dati relativi a questo videogioco, che non presenta le caratteristiche sopraelencate:\n",
"\n",
"- [Factorio](https://steamdb.info/app/427520/)\n",
" - non è mai stato scontato\n",
" - ha pubblicato per anni nella sezione annunci un diario settimanale dello sviluppo\n",
" - viene aggiornato regolarmente, ma solo per correzione di bug, e non per l'aggiunta di nuovi contenuti"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Struttura del progetto"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Il progetto è diviso in tre parti:\n",
"\n",
"- la directory `data` con i dati grezzi recuperati dalla rete, con a sua volta tante sottodirectory con i dati specifici di ciascun prodotto studiato;\n",
"- il package Python `unimore_bda_3`, contenente primitive specifiche all'elaborazione dei dati;\n",
"- questo notebook Jupyter, contenente celle per la rappresentazione dei dati elaborati."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Il package contiene un modulo particolare, `unimore_bda_3.prelude`, che importa tutte le dipendenze necessarie allo svolgimento dell'analisi e le ri-esporta con degli alias semplici e intuitivi."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Il package è compatibile con PEP518; per installarlo, è necessario eseguire i seguenti comandi dalla directory del progetto:\n",
"\n",
"```bash\n",
"python3 -v venv .venv\n",
"source venv/bin/activate # Assumendo l'utilizzo di Bash\n",
"pip install .\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si importa il modulo all'interno del notebook, in modo da avere tutti gli alias disponibili:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2023-07-02T06:47:40.693759958Z",
"start_time": "2023-07-02T06:47:39.216569683Z"
}
},
"outputs": [],
"source": [
"from unimore_bda_3.prelude import *"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Recupero manuale dei dati"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In questa sezione si recuperano manuale dati machine-readable da diverse fonti indipendenti fra loro, evitando di incappare in protezioni automatizzate insuperabili come CAPTCHA."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Google Trends](https://trends.google.com/trends/)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si cercano su Google Trends i prodotti oggetto di questa ricerca, facendo attenzione a selezionare l'*argomento \"Videogioco\"* e non il *termine di ricerca* con lo stesso nome per minimizzare i falsi positivi."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si configura poi la ricerca perchè sia relativa a *Tutto il mondo* e nell'arco di tempo *2004 - Presente*, in modo da avere dati generali relative alle tendenze globali, e poi si utilizza il pulsante *Download* per scaricare il file CSV relativo all'*Interesse nel tempo*."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I file CSV così recuperati sono inseriti al percorso `data/{nome}/gtrends-worldwide.csv`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [SteamDB](https://steamdb.info/)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si cercano su SteamDB i prodotti oggetto della ricerca."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Dalla pagina di ciascun prodotto si recupera l'App ID, un codice univoco utilizzato da Steam per identificare il software, e lo si salva all'interno del file `data/{nome}/steam_appid.txt`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si accede poi alla scheda *Price History*."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In fondo ad essa, nella omonima sezione, è presente un grafico dello storico prezzi, con a destra un bottone per il download del dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Il file recuperato viene salvato con il nome `data/{nome}/steamdb-price.csv`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si accede poi alla scheda *Charts*."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Poco sotto la selezione, si trova la sezione *Lifetime player count history*, contenente un grafico dello storico del numero di giocatori concorrenti dalla comparsa del prodotto su Steam, con a destra un bottone per il download del dataset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Il file recuperato viene salvato con il nome `data/{nome}/steamdb-players.csv`.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [IsThereAnyDeal](https://isthereanydeal.com/)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si cercano sul sito `https://new.isthereanydeal.com` i cinque prodotti relativi a questa ricerca."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si accede alla scheda *History*, e si utilizza il tool *Ispeziona Elemento* del browser Firefox per accedere al sorgente HTML della pagina dopo che essa ha eseguito il codice JavaScript necessario per la visualizzazione dei grafici."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Dal grafico, si risale la gerarchia degli elementi fino ad arrivare a un `
` con la classe `.js-chart-container`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Immediatamente dopo, è presente uno ``.\n",
"Ne si selezionano i contenuti, e li si salvano all'interno del file `data/{nome}/itad-price.js`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Importazione dei dati"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In questa sezione si descrive come caricare nel notebook i dati ottenuti manualmente, recuperandone automaticamente altri, convertendoli in `pd.DataFrame`, uniformandoli e rendendoli human-friendly."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Per ciascun dataset grezzo si è creato un modulo `unimore_bda_3.loaders.*` in grado di convertirlo in un `pd.DataFrame`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si importano tutti i loader all'interno di questo notebook Jupyter, raccolti all'interno della variabile `loaders`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"ExecuteTime": {
"end_time": "2023-07-02T06:47:40.694207576Z",
"start_time": "2023-07-02T06:47:40.298982419Z"
}
},
"outputs": [],
"source": [
"from unimore_bda_3 import loaders"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Importazione dati di Google Trends - `unimore_bda_3.loaders.gtrends`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Il *loader* dei dati di Google Trends utilizza la funzione `pd.read_csv` per trasformare il file comma-separated values direttamente in un `pd.DataFrame`.\n",
"\n",
"```python\n",
"dataframe = pd.read_csv(fd, sep=\",\", header=1)\n",
"```\n",
"\n",
"Utilizzando il kwarg `header`, Pandas è in grado di ignorare la riga di prefazione che Google Trends inserisce nei file scaricati:\n",
"\n",
"```csv\n",
"Categoria: Tutte le categorie\n",
"\n",
"Mese,Deep Rock Galactic: (Tutto il mondo)\n",
"2004-01,0\n",
"2004-02,0\n",
"2004-03,0\n",
"```\n",
"\n",
"Inoltre, il loader pulisce il dataset, effettuando i seguenti passi:\n",
"\n",
"1. rinomina le colonne, assegnando loro un titolo più human-friendly:\n",
"\n",
" ```python\n",
" dataframe.rename(inplace=True, columns={\n",
" \"Mese\": \"Date\",\n",
" f\"{query_name}: (Tutto il mondo)\": \"Google Trends · Score\",\n",
" })\n",
" ```\n",
"\n",
"2. converte in date le stringhe presenti nella colonna dell'indice:\n",
"\n",
" ```python\n",
" dataframe[\"Date\"] = pd.to_datetime(dataframe[\"Date\"])\n",
" ```\n",
"\n",
"3. converte in interi i numeri presenti nella colonna dei valori, trasformando in `0` i valori `\"< 1\"`, e scalando i valori da 0-100 a 0-1:\n",
"\n",
" ```python\n",
" dataframe[\"Google Trends · Score\"] = dataframe[\"Google Trends · Score\"].map(lambda x: int(x) if x != \"< 1\" else 0) / 100\n",
" ```\n",
"\n",
"4. imposta la colonna delle date come indice del dataframe:\n",
"\n",
" ```python\n",
" dataframe.set_index(\"Date\", inplace=True)\n",
" ```\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si verifica il corretto funzionamento del loader con questa chiamata ad esso:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2023-07-02T06:47:40.701305721Z",
"start_time": "2023-07-02T06:47:40.377967865Z"
}
},
"outputs": [
{
"data": {
"text/plain": " Google Trends · Score\nDate \n2004-01-01 0.00\n2004-02-01 0.00\n2004-03-01 0.00\n2004-04-01 0.00\n2004-05-01 0.01\n... ...\n2023-02-01 0.40\n2023-03-01 0.56\n2023-04-01 0.41\n2023-05-01 0.37\n2023-06-01 0.43\n\n[234 rows x 1 columns]",
"text/html": "<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>Google Trends · Score</th>\n </tr>\n <tr>\n <th>Date</th>\n <th></th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>2004-01-01</th>\n <td>0.00</td>\n </tr>\n <tr>\n <th>2004-02-01</th>\n <td>0.00</td>\n </tr>\n <tr>\n <th>2004-03-01</th>\n <td>0.00</td>\n </tr>\n <tr>\n <th>2004-04-01</th>\n <td>0.00</td>\n </tr>\n <tr>\n <th>2004-05-01</th>\n <td>0.01</td>\n </tr>\n <tr>\n <th>...</th>\n <td>...</td>\n </tr>\n <tr>\n <th>2023-02-01</th>\n <td>0.40</td>\n </tr>\n <tr>\n <th>2023-03-01</th>\n <td>0.56</td>\n </tr>\n <tr>\n <th>2023-04-01</th>\n <td>0.41</td>\n </tr>\n <tr>\n <th>2023-05-01</th>\n <td>0.37</td>\n </tr>\n <tr>\n <th>2023-06-01</th>\n <td>0.43</td>\n </tr>\n </tbody>\n</table>\n<p>234 rows × 1 columns</p>\n</div>"
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with open(\"./data/deeprockgalactic/gtrends-worldwide.csv\") as fd:\n",
" df = loaders.gtrends.load(fd, \"Deep Rock Galactic\")\n",
"df\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Importazione dati da SteamDB - `unimore_bda_3.loaders.steamdb`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I due *loader* dei dati da SteamDB funzionano in modo molto simile a quello di Google Trends, in quanto anch'esso usa `pd.read_csv`.\n",
"\n",
"```python\n",
"dataframe = pd.read_csv(fd, sep=\",\")\n",
"```\n",
"\n",
"Allo stesso modo, puliscono il dataset in un modo molto simile:\n",
"\n",
"```python\n",
"dataframe.rename(inplace=True, columns={\n",
" \"DateTime\": \"Date\",\n",
" \"Players\": \"SteamDB · Peak concurrent players\",\n",
" \"Average Players\": \"SteamDB · Day average of concurrent players\",\n",
" \"Flags\": \"SteamDB · Player count flags\",\n",
"})\n",
"\n",
"dataframe[\"Date\"] = pd.to_datetime(dataframe[\"Date\"])\n",
"\n",
"dataframe.set_index(\"Date\", inplace=True)\n",
"```\n",
"\n",
"```python\n",
"dataframe.rename(inplace=True, columns={\n",
" \"DateTime\": \"Date\",\n",
" \"Final price\": \"SteamDB · Steam\",\n",
" \"Flags\": \"SteamDB · Price flags\",\n",
"})\n",
"\n",
"dataframe[\"Date\"] = pd.to_datetime(dataframe[\"Date\"])\n",
"\n",
"dataframe.set_index(\"Date\", inplace=True)\n",
"```\n",
"\n",
"Si ha però una differenza significativa, che consiste nel raccoglimento dei valori per data, in quanto i dataset contengono anche gli orari di registrazione dei dati, che ai fini di questa ricerca possono essere ignorati.\n",
"\n",
"```python\n",
"dataframe = dataframe.groupby(dataframe.index.date).max()\n",
"dataframe.index = pd.to_datetime(dataframe.index)\n",
"```\n",
"\n",
"```python\n",
"dataframe = dataframe.groupby(dataframe.index.date).min()\n",
"dataframe.index = pd.to_datetime(dataframe.index)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si verifica il corretto funzionamento dei loader con queste chiamate ad essi:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"ExecuteTime": {
"end_time": "2023-07-02T06:47:40.704705835Z",
"start_time": "2023-07-02T06:47:40.406769392Z"
}
},
"outputs": [
{
"data": {
"text/plain": " SteamDB · Peak concurrent players \\\n2016-12-01 11.0 \n2016-12-02 NaN \n2016-12-03 NaN \n2016-12-04 NaN \n2016-12-05 NaN \n... ... \n2023-06-25 35016.0 \n2023-06-26 30017.0 \n2023-06-27 29568.0 \n2023-06-28 28539.0 \n2023-06-29 28042.0 \n\n SteamDB · Day average of concurrent players \\\n2016-12-01 NaN \n2016-12-02 NaN \n2016-12-03 NaN \n2016-12-04 NaN \n2016-12-05 NaN \n... ... \n2023-06-25 26276.0 \n2023-06-26 22421.0 \n2023-06-27 21949.0 \n2023-06-28 21347.0 \n2023-06-29 21347.0 \n\n SteamDB · Player count flags \n2016-12-01 NaN \n2016-12-02 NaN \n2016-12-03 NaN \n2016-12-04 NaN \n2016-12-05 NaN \n... ... \n2023-06-25 NaN \n2023-06-26 NaN \n2023-06-27 NaN \n2023-06-28 NaN \n2023-06-29 NaN \n\n[2402 rows x 3 columns]",
"text/html": "<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>SteamDB · Peak concurrent players</th>\n <th>SteamDB · Day average of concurrent players</th>\n <th>SteamDB · Player count flags</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>2016-12-01</th>\n <td>11.0</td>\n <td>NaN</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2016-12-02</th>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2016-12-03</th>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2016-12-04</th>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2016-12-05</th>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>...</th>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n </tr>\n <tr>\n <th>2023-06-25</th>\n <td>35016.0</td>\n <td>26276.0</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2023-06-26</th>\n <td>30017.0</td>\n <td>22421.0</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2023-06-27</th>\n <td>29568.0</td>\n <td>21949.0</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2023-06-28</th>\n <td>28539.0</td>\n <td>21347.0</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2023-06-29</th>\n <td>28042.0</td>\n <td>21347.0</td>\n <td>NaN</td>\n </tr>\n </tbody>\n</table>\n<p>2402 rows × 3 columns</p>\n</div>"
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with open(\"./data/deeprockgalactic/steamdb-players.csv\") as fd:\n",
" df = loaders.steamdb.load_players(fd)\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2023-07-02T06:47:40.709769142Z",
"start_time": "2023-07-02T06:47:40.448248117Z"
}
},
"outputs": [
{
"data": {
"text/plain": " SteamDB · Steam SteamDB · Price flags\n2018-02-28 22.99 NaN\n2018-04-24 20.69 NaN\n2018-04-27 22.99 NaN\n2018-05-10 17.24 NaN\n2018-05-14 22.99 NaN\n... ... ...\n2023-03-23 29.99 NaN\n2023-04-20 9.89 NaN\n2023-05-04 29.99 NaN\n2023-06-15 9.89 NaN\n2023-06-29 9.89 NaN\n\n[96 rows x 2 columns]",
"text/html": "<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>SteamDB · Steam</th>\n <th>SteamDB · Price flags</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>2018-02-28</th>\n <td>22.99</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2018-04-24</th>\n <td>20.69</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2018-04-27</th>\n <td>22.99</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2018-05-10</th>\n <td>17.24</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2018-05-14</th>\n <td>22.99</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>...</th>\n <td>...</td>\n <td>...</td>\n </tr>\n <tr>\n <th>2023-03-23</th>\n <td>29.99</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2023-04-20</th>\n <td>9.89</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2023-05-04</th>\n <td>29.99</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2023-06-15</th>\n <td>9.89</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2023-06-29</th>\n <td>9.89</td>\n <td>NaN</td>\n </tr>\n </tbody>\n</table>\n<p>96 rows × 2 columns</p>\n</div>"
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with open(\"./data/deeprockgalactic/steamdb-price.csv\") as fd:\n",
" df = loaders.steamdb.load_price(fd)\n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Importazione dati da IsThereAnyDeal - `unimore_bda_3.loaders.itad`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Il *loader* dei dati da IsThereAnyDeal ricerca i dataset JSON contenuti all'interno del codice JavaScript del sito web, utilizzati per generare dinamicamente i grafici che esso mostra.\n",
"\n",
"La regular expression utilizzata è la seguente:\n",
"\n",
"```python\n",
"PRICE_REGEX = re.compile(r\"\"\"Charts[.]Builder[(]setup, (.+?)[)]\"\"\")\n",
"```\n",
"\n",
"Per ciascun dataset trovato, il loader chiama un'altra funzione, adibita a convertirlo in un `pd.DataFrame`:\n",
"\n",
"```python\n",
"matches: list[str] = PRICE_REGEX.findall(data)\n",
"\n",
"return [_load_price_dataframe(match) for match in matches]\n",
"```\n",
"\n",
"La funzione di conversione a `pd.DataFrame` carica il dataset JSON con la funzione `json.loads`, poi per ciascuna serie presente all'interno di esso chiama un'ulteriore funzione che si occupa di convertirla in una `pd.Series`.\n",
"\n",
"```python\n",
"srcs: list = json.loads(match)\n",
"\n",
"dataframe = pd.DataFrame(\n",
" data=[_load_price_series(src) for src in srcs]\n",
").T\n",
"\n",
"dataframe.index = pd.to_datetime(dataframe.index)\n",
"\n",
"return dataframe\n",
"```\n",
"\n",
"La funzione di conversione a `pd.Series` converte le tuple `(indice, valore)` in due liste contenente tutti gli indici e tutti i valori, il formato richiesto da Pandas.\n",
"\n",
"```python\n",
"index = [pd.Timestamp(item[0], unit=\"ms\", tz=\"utc\") for item in src[\"data\"]]\n",
"data = [item[1] for item in src[\"data\"]]\n",
"\n",
"series = pd.Series(\n",
" data=data,\n",
" index=index,\n",
" name=f\"\"\"ITAD · {src[\"name\"]}\"\"\"\n",
")\n",
"\n",
"return series\n",
"```\n",
"\n",
"Inoltre, effettua lo stesso raggruppamento già visto nel loader di SteamDB per selezionare solo la data e non l'ora di ciascun valore:\n",
"\n",
"```python\n",
"if \"Worst\" in series.name or \"High\" in series.name:\n",
" series = series.groupby(series.index.date).max()\n",
"else:\n",
" series = series.groupby(series.index.date).min()\n",
"```\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si verifica il corretto funzionamento del loader con questa chiamata ad esso:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"ExecuteTime": {
"end_time": "2023-07-02T06:47:40.711484212Z",
"start_time": "2023-07-02T06:47:40.485670340Z"
}
},
"outputs": [
{
"data": {
"text/plain": " ITAD · Best Price ITAD · Best Regular Price \\\n2018-02-28 22.99 22.99 \n2018-04-24 20.69 22.99 \n2018-04-27 20.69 22.99 \n2018-05-10 17.24 22.99 \n2018-05-14 17.24 22.99 \n... ... ... \n2023-05-04 9.89 29.99 \n2023-05-05 9.89 29.99 \n2023-05-23 9.89 29.99 \n2023-06-15 9.89 29.99 \n2023-06-29 9.89 29.99 \n\n ITAD · Worst Regular Price ITAD · Historical Low \n2018-02-28 22.99 22.99 \n2018-04-24 22.99 20.69 \n2018-04-27 22.99 NaN \n2018-05-10 22.99 17.24 \n2018-05-14 22.99 NaN \n... ... ... \n2023-05-04 29.99 NaN \n2023-05-05 29.99 NaN \n2023-05-23 29.99 NaN \n2023-06-15 29.99 NaN \n2023-06-29 29.99 0.00 \n\n[151 rows x 4 columns]",
"text/html": "<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>ITAD · Best Price</th>\n <th>ITAD · Best Regular Price</th>\n <th>ITAD · Worst Regular Price</th>\n <th>ITAD · Historical Low</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>2018-02-28</th>\n <td>22.99</td>\n <td>22.99</td>\n <td>22.99</td>\n <td>22.99</td>\n </tr>\n <tr>\n <th>2018-04-24</th>\n <td>20.69</td>\n <td>22.99</td>\n <td>22.99</td>\n <td>20.69</td>\n </tr>\n <tr>\n <th>2018-04-27</th>\n <td>20.69</td>\n <td>22.99</td>\n <td>22.99</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2018-05-10</th>\n <td>17.24</td>\n <td>22.99</td>\n <td>22.99</td>\n <td>17.24</td>\n </tr>\n <tr>\n <th>2018-05-14</th>\n <td>17.24</td>\n <td>22.99</td>\n <td>22.99</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>...</th>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n </tr>\n <tr>\n <th>2023-05-04</th>\n <td>9.89</td>\n <td>29.99</td>\n <td>29.99</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2023-05-05</th>\n <td>9.89</td>\n <td>29.99</td>\n <td>29.99</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2023-05-23</th>\n <td>9.89</td>\n <td>29.99</td>\n <td>29.99</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2023-06-15</th>\n <td>9.89</td>\n <td>29.99</td>\n <td>29.99</td>\n <td>NaN</td>\n </tr>\n <tr>\n <th>2023-06-29</th>\n <td>9.89</td>\n <td>29.99</td>\n <td>29.99</td>\n <td>0.00</td>\n </tr>\n </tbody>\n</table>\n<p>151 rows × 4 columns</p>\n</div>"
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with open(\"./data/deeprockgalactic/itad-price.js\") as fd:\n",
" df = loaders.itad.load(fd)\n",
"df[0]\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Recupero e importazione dei dati degli annunci dalla web API di Steam - `unimore_bda_3.loaders.steam`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Il *loader* di dati da Steam non si basa su un dataset locale archiviato su file, bensì effettua dinamicamente richieste alla web API di Steam al fine di recuperare i dati più aggiornati disponibili relativi agli annunci di un prodotto con un dato app id.\n",
"\n",
"A tale scopo, fa uso del package `httpx` per richieste HTTP:\n",
"\n",
"```python\n",
"import httpx\n",
"\n",
"steam_api = httpx.Client(base_url=\"https://api.steampowered.com\")\n",
"```\n",
"\n",
"Il loader legge dal relativo file l'app id del prodotto di cui caricare i dati, e procede a effettuare richieste HTTP per esso:\n",
"\n",
"```python\n",
"appid = int(fd.read().strip())\n",
"data = fetch(appid=appid)\n",
"\n",
"return data\n",
"```\n",
"\n",
"Il recupero dei dati avviene in quattro fasi:\n",
"\n",
"1. Vengono effettuate ripetute richieste all'API fino a quando non sono stati recuperati tutti gli annunci esistenti:\n",
"\n",
" ```python\n",
" count = 100\n",
" enddate = {}\n",
" newsitems = []\n",
"\n",
" while count == 100:\n",
" request = steam_api.get(\n",
" \"/ISteamNews/GetNewsForApp/v0002/\",\n",
" params={\n",
" \"appid\": appid,\n",
" \"count\": 100,\n",
" \"format\": \"json\",\n",
" **enddate,\n",
" }\n",
" )\n",
" request.raise_for_status()\n",
" data = request.json()[\"appnews\"]\n",
" count = len(data[\"newsitems\"])\n",
" newsitems.extend(data[\"newsitems\"])\n",
" enddate = {\"enddate\": newsitems[-1][\"date\"]}\n",
"\n",
" return newsitems\n",
" ```\n",
"\n",
"2. Gli annunci recuperati vengono categorizzati in base al tag ad essi associato dalla web API:\n",
"\n",
" ```python\n",
" result = collections.defaultdict(list)\n",
"\n",
" for item in news:\n",
" tags = item.get(\"tags\", [])\n",
" if tags:\n",
" for tag in set(item.get(\"tags\", [])):\n",
" result[tag].append(item)\n",
" else:\n",
" result[\"no_tags\"].append(item)\n",
"\n",
" return result\n",
" ```\n",
"\n",
"3. Il numero di annunci per ciascun tag in ciascun giorno viene raccolto in una `pd.Series`:\n",
"\n",
" ```python\n",
" index = pd.to_datetime([datetime.fromtimestamp(item[\"date\"]) for item in news])\n",
"\n",
" return pd.Series(\n",
" data=[1 for _ in index],\n",
" index=index,\n",
" name=f\"\"\"Steam · Count of News tagged {name}\"\"\",\n",
" dtype=np.uint8,\n",
" )\n",
" ```\n",
"\n",
"4. Le `pd.Series` vengono unite in un dataframe unico, con i valori nulli riempiti a zero e i conteggi di annunci in una sola giornata ancora una volta raggruppati e sommati:\n",
"\n",
" ```python\n",
" raw_news = _load_news(appid=appid)\n",
" categorized_news = _categorize_news(news=raw_news)\n",
" serialized_news = [_serialize_news(name=name, news=news).to_frame() for name, news in categorized_news.items()]\n",
"\n",
" dataframe = utils.join_frames(*serialized_news).fillna(0)\n",
" dataframe = dataframe.groupby(dataframe.index.date).sum()\n",
" dataframe.index = pd.to_datetime(dataframe.index)\n",
"\n",
" return dataframe\n",
" ```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si verifica il corretto funzionamento del loader con questa chiamata ad esso:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"ExecuteTime": {
"end_time": "2023-07-02T06:47:43.612325961Z",
"start_time": "2023-07-02T06:47:40.550738777Z"
}
},
"outputs": [
{
"data": {
"text/plain": " Steam · Count of News tagged no_tags \\\n2016-12-15 1 \n2017-02-17 1 \n2017-02-23 1 \n2017-03-23 5 \n2017-04-01 1 \n... ... \n2023-06-01 2 \n2023-06-09 1 \n2023-06-14 1 \n2023-06-15 1 \n2023-06-29 1 \n\n Steam · Count of News tagged patchnotes \\\n2016-12-15 0.0 \n2017-02-17 1.0 \n2017-02-23 1.0 \n2017-03-23 0.0 \n2017-04-01 0.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged hide_store \\\n2016-12-15 0.0 \n2017-02-17 0.0 \n2017-02-23 0.0 \n2017-03-23 0.0 \n2017-04-01 0.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged ModAct_1401078964_1677758714_0 \\\n2016-12-15 0.0 \n2017-02-17 0.0 \n2017-02-23 0.0 \n2017-03-23 0.0 \n2017-04-01 0.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged mod_require_rereview \\\n2016-12-15 0.0 \n2017-02-17 0.0 \n2017-02-23 0.0 \n2017-03-23 0.0 \n2017-04-01 0.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged mod_reviewed \\\n2016-12-15 0.0 \n2017-02-17 0.0 \n2017-02-23 0.0 \n2017-03-23 0.0 \n2017-04-01 0.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged steam_award_vote_request \\\n2016-12-15 0.0 \n2017-02-17 0.0 \n2017-02-23 0.0 \n2017-03-23 0.0 \n2017-04-01 0.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged mod_hide_library_overview \\\n2016-12-15 0.0 \n2017-02-17 0.0 \n2017-02-23 0.0 \n2017-03-23 0.0 \n2017-04-01 0.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged steam_award_nomination_request \\\n2016-12-15 0.0 \n2017-02-17 0.0 \n2017-02-23 0.0 \n2017-03-23 0.0 \n2017-04-01 0.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged halloween \\\n2016-12-15 0.0 \n2017-02-17 0.0 \n2017-02-23 0.0 \n2017-03-23 0.0 \n2017-04-01 0.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged hide_library_overview \\\n2016-12-15 1.0 \n2017-02-17 1.0 \n2017-02-23 1.0 \n2017-03-23 5.0 \n2017-04-01 1.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged hide_library_detail \\\n2016-12-15 1.0 \n2017-02-17 1.0 \n2017-02-23 1.0 \n2017-03-23 5.0 \n2017-04-01 1.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged halloween2019candidate \\\n2016-12-15 0.0 \n2017-02-17 0.0 \n2017-02-23 0.0 \n2017-03-23 0.0 \n2017-04-01 0.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged auto_migrated \n2016-12-15 1.0 \n2017-02-17 1.0 \n2017-02-23 1.0 \n2017-03-23 5.0 \n2017-04-01 1.0 \n... ... \n2023-06-01 0.0 \n2023-06-09 0.0 \n2023-06-14 0.0 \n2023-06-15 0.0 \n2023-06-29 0.0 \n\n[204 rows x 14 columns]",
"text/html": "<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>Steam · Count of News tagged no_tags</th>\n <th>Steam · Count of News tagged patchnotes</th>\n <th>Steam · Count of News tagged hide_store</th>\n <th>Steam · Count of News tagged ModAct_1401078964_1677758714_0</th>\n <th>Steam · Count of News tagged mod_require_rereview</th>\n <th>Steam · Count of News tagged mod_reviewed</th>\n <th>Steam · Count of News tagged steam_award_vote_request</th>\n <th>Steam · Count of News tagged mod_hide_library_overview</th>\n <th>Steam · Count of News tagged steam_award_nomination_request</th>\n <th>Steam · Count of News tagged halloween</th>\n <th>Steam · Count of News tagged hide_library_overview</th>\n <th>Steam · Count of News tagged hide_library_detail</th>\n <th>Steam · Count of News tagged halloween2019candidate</th>\n <th>Steam · Count of News tagged auto_migrated</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>2016-12-15</th>\n <td>1</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>1.0</td>\n <td>1.0</td>\n <td>0.0</td>\n <td>1.0</td>\n </tr>\n <tr>\n <th>2017-02-17</th>\n <td>1</td>\n <td>1.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>1.0</td>\n <td>1.0</td>\n <td>0.0</td>\n <td>1.0</td>\n </tr>\n <tr>\n <th>2017-02-23</th>\n <td>1</td>\n <td>1.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>1.0</td>\n <td>1.0</td>\n <td>0.0</td>\n <td>1.0</td>\n </tr>\n <tr>\n <th>2017-03-23</th>\n <td>5</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>5.0</td>\n <td>5.0</td>\n <td>0.0</td>\n <td>5.0</td>\n </tr>\n <tr>\n <th>2017-04-01</th>\n <td>1</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>1.0</td>\n <td>1.0</td>\n <td>0.0</td>\n <td>1.0</td>\n </tr>\n <tr>\n <th>...</th>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n </tr>\n <tr>\n <th>2023-06-01</th>\n <td>2</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n </tr>\n <tr>\n <th>2023-06-09</th>\n <td>1</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n </tr>\n <tr>\n <th>2023-06-14</th>\n <td>1</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n </tr>\n <tr>\n <th>2023-06-15</th>\n <td>1</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n </tr>\n <tr>\n <th>2023-06-29</th>\n <td>1</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n </tr>\n </tbody>\n</table>\n<p>204 rows × 14 columns</p>\n</div>"
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with open(\"./data/deeprockgalactic/steam_appid.txt\") as fd:\n",
" df = loaders.steam.load(fd)\n",
"df\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"-----"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"ExecuteTime": {
"end_time": "2023-07-02T06:47:43.626658327Z",
"start_time": "2023-07-02T06:47:43.611434833Z"
}
},
"outputs": [],
"source": [
"# Pulizia ambiente\n",
"\n",
"try:\n",
" del fd\n",
"except NameError:\n",
" pass\n",
"\n",
"try:\n",
" del df\n",
"except NameError:\n",
" pass"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Elaborazione dei dati - `unimore_bda_3.processing`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In questa sezione si descrive come elaborare i `pd.DataFrame` caricati per unificarli in uno unico molto più grande, più facile da visualizzare e manipolare."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si crea un modulo Python chiamato `unimore_bda_3.processing` con tale scopo."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Lo si importa all'interno di questo notebook:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"ExecuteTime": {
"end_time": "2023-07-02T06:47:43.717295300Z",
"start_time": "2023-07-02T06:47:43.629417962Z"
}
},
"outputs": [],
"source": [
"from unimore_bda_3 import processing"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Elaborazione dei dati di un singolo prodotto - `process_game`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si definisce la funzione `process_game`, per raccogliere tutti i dati relativi a uno specifico videogioco in un unico `pd.DataFrame`, dato il suo titolo e il percorso alla cartella dei file contenenti i dataset.\n",
"\n",
"```python\n",
"def process_game(name: str, path: Path) -> pd.DataFrame:\n",
" ...\n",
"```\n",
"\n",
"Come prima cosa, essa determina il percorso di ciascun singolo file da processare:\n",
"\n",
"```python\n",
"steam_appid_path = path.joinpath(\"steam_appid.txt\")\n",
"gtrends_worldwide_path = path.joinpath(\"gtrends-worldwide.csv\")\n",
"steamdb_players_path = path.joinpath(\"steamdb-players.csv\")\n",
"steamdb_price_path = path.joinpath(\"steamdb-price.csv\")\n",
"itad_price_path = path.joinpath(\"itad-price.js\")\n",
"```\n",
"\n",
"In seguito, per ciascun percorso, utilizza il relativo *loader* per ottenere un `pd.DataFrame`:\n",
"\n",
"```python\n",
"with open(steam_appid_path) as fd:\n",
" steam_news: pd.DataFrame = loaders.steam.load(fd=fd)\n",
"\n",
"with open(gtrends_worldwide_path) as fd:\n",
" google_trends: pd.DataFrame = loaders.gtrends.load(fd=fd, query_name=name)\n",
"\n",
"with open(steamdb_players_path) as fd:\n",
" steamdb_players: pd.DataFrame = loaders.steamdb.load_players(fd=fd)\n",
"\n",
"with open(steamdb_price_path) as fd:\n",
" steamdb_price: pd.DataFrame = loaders.steamdb.load_price(fd=fd)\n",
"\n",
"with open(itad_price_path) as fd:\n",
" itad_prices: list[pd.DataFrame] = loaders.itad.load(fd=fd)\n",
"```\n",
"\n",
"Poi, utilizzando la seguente funzione di utility, effettua un outer join di tutti i `pd.DataFrame` recuperati:\n",
"\n",
"```python\n",
"def join_frames(*dfs: pd.DataFrame, **kwargs) -> pd.DataFrame:\n",
" \"\"\"\n",
" Join multiple :class:`pandas.DataFrame`s in a single expression.\n",
"\n",
" :param dfs: The :class:`pandas.DataFrame`s to join.\n",
" :param kwargs: Keyword arguments to pass to :meth:`pandas.DataFrame.join`.\n",
" :return: The resulting :class:`pandas.DataFrame`.\n",
" \"\"\"\n",
" if len(dfs) == 0:\n",
" return pd.DataFrame()\n",
" elif len(dfs) == 1:\n",
" return dfs[0]\n",
" else:\n",
" return dfs[0].join(dfs[1:], **kwargs)\n",
"```\n",
"\n",
"```python\n",
"dataframe: pd.DataFrame = utils.join_frames(steamdb_players, steamdb_price, google_trends, steam_news, *itad_prices)\n",
"```\n",
"\n",
"Successivamente, viene effettuato *forward fill* su diverse colonne, in quanto molti dei precedenti dataframe hanno valori solo in corrispondenza delle date in cui i valori sono cambiati:\n",
"\n",
"```python\n",
"dataframe[\"SteamDB · Steam\"].fillna(method=\"ffill\", inplace=True)\n",
"dataframe[\"Google Trends · Score\"].fillna(method=\"ffill\", inplace=True)\n",
"dataframe[\"ITAD · Best Price\"].fillna(method=\"ffill\", inplace=True)\n",
"dataframe[\"ITAD · Best Regular Price\"].fillna(method=\"ffill\", inplace=True)\n",
"dataframe[\"ITAD · Worst Regular Price\"].fillna(method=\"ffill\", inplace=True)\n",
"dataframe[\"ITAD · Historical Low\"].fillna(method=\"ffill\", inplace=True)\n",
"news_columns = list(filter(lambda c: c.startswith(\"Steam · Count of News tagged\"), dataframe.columns))\n",
"for news_col_name in news_columns:\n",
" dataframe[news_col_name].fillna(0, inplace=True)\n",
"```\n",
"\n",
"Si aggiungono poi alcuni dati derivati.\n",
"\n",
"Il primo di essi è una colonna booleana chiamata `Steam · Is there News?`, contenente `True` se in quel giorno è stato pubblicato un annuncio di qualsiasi tipo, e `False` altrimenti.\n",
"\n",
"```python\n",
"dataframe[\"Steam · Is there News?\"] = pd.Series(data=False, dtype=bool)\n",
"dataframe[\"Steam · Is there News?\"] = (dataframe[news_columns] > 0).any()\n",
"```\n",
"\n",
"Secondo, terzo, quarto, e quinto sono un gruppo di colonne contenente alcune metriche relative al loro massimo mai raggiunto:\n",
"\n",
"```python\n",
"dataframe[\"SteamDB · Relative concurrent players\"] = dataframe[\"SteamDB · Peak concurrent players\"] / dataframe[\"SteamDB · Peak concurrent players\"].max()\n",
"dataframe[\"SteamDB · Relative Steam price\"] = dataframe[\"SteamDB · Steam\"] / dataframe[\"SteamDB · Steam\"].max()\n",
"dataframe[\"ITAD · Relative Best Price\"] = dataframe[\"ITAD · Best Price\"] / dataframe[\"ITAD · Best Price\"].max()\n",
"```\n",
"\n",
"Come sesta e settima colonna, si calcola la differenza giornaliera di prezzo, rispettivamente con i dati di SteamDB (solo Steam) e IsThereAnyDeal (tutti i rivenditori):\n",
"\n",
"```python\n",
"dataframe[\"ITAD · Best price change from previous day\"] = dataframe[\"ITAD · Best Price\"].diff()\n",
"dataframe[\"SteamDB · Steam price change from previous day\"] = - dataframe[\"SteamDB · Steam\"].diff()\n",
"```\n",
"\n",
"Come fatto per gli annunci, come settima e ottava colonna si crea una colonna booleana che denota la presenza o l'assenza di uno sconto in un determinato giorno:\n",
"\n",
"```python\n",
"dataframe[\"SteamDB · Is there a discount?\"] = dataframe[\"SteamDB · Steam price change from previous day\"] < 0\n",
"dataframe[\"ITAD · Is there a discount?\"] = dataframe[\"ITAD · Best price change from previous day\"] < 0\n",
"```\n",
"\n",
"Infine, si combinano le colonne booleane per crearne altre due che combinano le precedenti per indicare la presenza di sconti *o* di un annuncio in un determinato giorno:\n",
"\n",
"```python\n",
"dataframe[\"Cumulative · Is something happening on Steam?\"] = dataframe[\"Steam · Is there News?\"] + dataframe[\"SteamDB · Is there a discount?\"]\n",
"dataframe[\"Cumulative · Is something happening?\"] = dataframe[\"Steam · Is there News?\"] + dataframe[\"SteamDB · Is there a discount?\"] + dataframe[\"ITAD · Is there a discount?\"]\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Si esegue la funzione `process_game` sui dataset dei cinque videogiochi considerati:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"ExecuteTime": {
"end_time": "2023-07-02T06:47:44.016906766Z",
"start_time": "2023-07-02T06:47:43.671112671Z"
}
},
"outputs": [
{
"data": {
"text/plain": " SteamDB · Peak concurrent players \\\n2016-12-01 11.0 \n2016-12-02 NaN \n2016-12-03 NaN \n2016-12-04 NaN \n2016-12-05 NaN \n... ... \n2023-06-25 35016.0 \n2023-06-26 30017.0 \n2023-06-27 29568.0 \n2023-06-28 28539.0 \n2023-06-29 28042.0 \n\n SteamDB · Day average of concurrent players \\\n2016-12-01 NaN \n2016-12-02 NaN \n2016-12-03 NaN \n2016-12-04 NaN \n2016-12-05 NaN \n... ... \n2023-06-25 26276.0 \n2023-06-26 22421.0 \n2023-06-27 21949.0 \n2023-06-28 21347.0 \n2023-06-29 21347.0 \n\n SteamDB · Player count flags SteamDB · Steam \\\n2016-12-01 NaN NaN \n2016-12-02 NaN NaN \n2016-12-03 NaN NaN \n2016-12-04 NaN NaN \n2016-12-05 NaN NaN \n... ... ... \n2023-06-25 NaN 9.89 \n2023-06-26 NaN 9.89 \n2023-06-27 NaN 9.89 \n2023-06-28 NaN 9.89 \n2023-06-29 NaN 9.89 \n\n SteamDB · Price flags Google Trends · Score \\\n2016-12-01 NaN 0.00 \n2016-12-02 NaN 0.00 \n2016-12-03 NaN 0.00 \n2016-12-04 NaN 0.00 \n2016-12-05 NaN 0.00 \n... ... ... \n2023-06-25 NaN 0.43 \n2023-06-26 NaN 0.43 \n2023-06-27 NaN 0.43 \n2023-06-28 NaN 0.43 \n2023-06-29 NaN 0.43 \n\n Steam · Count of News tagged no_tags \\\n2016-12-01 0.0 \n2016-12-02 0.0 \n2016-12-03 0.0 \n2016-12-04 0.0 \n2016-12-05 0.0 \n... ... \n2023-06-25 0.0 \n2023-06-26 0.0 \n2023-06-27 0.0 \n2023-06-28 0.0 \n2023-06-29 1.0 \n\n Steam · Count of News tagged patchnotes \\\n2016-12-01 0.0 \n2016-12-02 0.0 \n2016-12-03 0.0 \n2016-12-04 0.0 \n2016-12-05 0.0 \n... ... \n2023-06-25 0.0 \n2023-06-26 0.0 \n2023-06-27 0.0 \n2023-06-28 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged hide_store \\\n2016-12-01 0.0 \n2016-12-02 0.0 \n2016-12-03 0.0 \n2016-12-04 0.0 \n2016-12-05 0.0 \n... ... \n2023-06-25 0.0 \n2023-06-26 0.0 \n2023-06-27 0.0 \n2023-06-28 0.0 \n2023-06-29 0.0 \n\n Steam · Count of News tagged ModAct_1401078964_1677758714_0 ... \\\n2016-12-01 0.0 ... \n2016-12-02 0.0 ... \n2016-12-03 0.0 ... \n2016-12-04 0.0 ... \n2016-12-05 0.0 ... \n... ... ... \n2023-06-25 0.0 ... \n2023-06-26 0.0 ... \n2023-06-27 0.0 ... \n2023-06-28 0.0 ... \n2023-06-29 0.0 ... \n\n Steam · Is there News? SteamDB · Relative concurrent players \\\n2016-12-01 False 0.000236 \n2016-12-02 False NaN \n2016-12-03 False NaN \n2016-12-04 False NaN \n2016-12-05 False NaN \n... ... ... \n2023-06-25 False 0.750016 \n2023-06-26 False 0.642941 \n2023-06-27 False 0.633324 \n2023-06-28 False 0.611284 \n2023-06-29 True 0.600638 \n\n SteamDB · Relative Steam price ITAD · Relative Best Price \\\n2016-12-01 NaN NaN \n2016-12-02 NaN NaN \n2016-12-03 NaN NaN \n2016-12-04 NaN NaN \n2016-12-05 NaN NaN \n... ... ... \n2023-06-25 0.329777 0.329777 \n2023-06-26 0.329777 0.329777 \n2023-06-27 0.329777 0.329777 \n2023-06-28 0.329777 0.329777 \n2023-06-29 0.329777 0.329777 \n\n ITAD · Best price change from previous day \\\n2016-12-01 NaN \n2016-12-02 NaN \n2016-12-03 NaN \n2016-12-04 NaN \n2016-12-05 NaN \n... ... \n2023-06-25 0.0 \n2023-06-26 0.0 \n2023-06-27 0.0 \n2023-06-28 0.0 \n2023-06-29 0.0 \n\n SteamDB · Steam price change from previous day \\\n2016-12-01 NaN \n2016-12-02 NaN \n2016-12-03 NaN \n2016-12-04 NaN \n2016-12-05 NaN \n... ... \n2023-06-25 -0.0 \n2023-06-26 -0.0 \n2023-06-27 -0.0 \n2023-06-28 -0.0 \n2023-06-29 -0.0 \n\n SteamDB · Is there a discount? ITAD · Is there a discount? \\\n2016-12-01 False False \n2016-12-02 False False \n2016-12-03 False False \n2016-12-04 False False \n2016-12-05 False False \n... ... ... \n2023-06-25 False False \n2023-06-26 False False \n2023-06-27 False False \n2023-06-28 False False \n2023-06-29 False False \n\n Cumulative · Is something happening on Steam? \\\n2016-12-01 False \n2016-12-02 False \n2016-12-03 False \n2016-12-04 False \n2016-12-05 False \n... ... \n2023-06-25 False \n2023-06-26 False \n2023-06-27 False \n2023-06-28 False \n2023-06-29 True \n\n Cumulative · Is something happening? \n2016-12-01 False \n2016-12-02 False \n2016-12-03 False \n2016-12-04 False \n2016-12-05 False \n... ... \n2023-06-25 False \n2023-06-26 False \n2023-06-27 False \n2023-06-28 False \n2023-06-29 True \n\n[2402 rows x 36 columns]",
"text/html": "<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>SteamDB · Peak concurrent players</th>\n <th>SteamDB · Day average of concurrent players</th>\n <th>SteamDB · Player count flags</th>\n <th>SteamDB · Steam</th>\n <th>SteamDB · Price flags</th>\n <th>Google Trends · Score</th>\n <th>Steam · Count of News tagged no_tags</th>\n <th>Steam · Count of News tagged patchnotes</th>\n <th>Steam · Count of News tagged hide_store</th>\n <th>Steam · Count of News tagged ModAct_1401078964_1677758714_0</th>\n <th>...</th>\n <th>Steam · Is there News?</th>\n <th>SteamDB · Relative concurrent players</th>\n <th>SteamDB · Relative Steam price</th>\n <th>ITAD · Relative Best Price</th>\n <th>ITAD · Best price change from previous day</th>\n <th>SteamDB · Steam price change from previous day</th>\n <th>SteamDB · Is there a discount?</th>\n <th>ITAD · Is there a discount?</th>\n <th>Cumulative · Is something happening on Steam?</th>\n <th>Cumulative · Is something happening?</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>2016-12-01</th>\n <td>11.0</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>0.00</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>...</td>\n <td>False</td>\n <td>0.000236</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n </tr>\n <tr>\n <th>2016-12-02</th>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>0.00</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>...</td>\n <td>False</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n </tr>\n <tr>\n <th>2016-12-03</th>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>0.00</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>...</td>\n <td>False</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n </tr>\n <tr>\n <th>2016-12-04</th>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>0.00</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>...</td>\n <td>False</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n </tr>\n <tr>\n <th>2016-12-05</th>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>0.00</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>...</td>\n <td>False</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n </tr>\n <tr>\n <th>...</th>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n <td>...</td>\n </tr>\n <tr>\n <th>2023-06-25</th>\n <td>35016.0</td>\n <td>26276.0</td>\n <td>NaN</td>\n <td>9.89</td>\n <td>NaN</td>\n <td>0.43</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>...</td>\n <td>False</td>\n <td>0.750016</td>\n <td>0.329777</td>\n <td>0.329777</td>\n <td>0.0</td>\n <td>-0.0</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n </tr>\n <tr>\n <th>2023-06-26</th>\n <td>30017.0</td>\n <td>22421.0</td>\n <td>NaN</td>\n <td>9.89</td>\n <td>NaN</td>\n <td>0.43</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>...</td>\n <td>False</td>\n <td>0.642941</td>\n <td>0.329777</td>\n <td>0.329777</td>\n <td>0.0</td>\n <td>-0.0</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n </tr>\n <tr>\n <th>2023-06-27</th>\n <td>29568.0</td>\n <td>21949.0</td>\n <td>NaN</td>\n <td>9.89</td>\n <td>NaN</td>\n <td>0.43</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>...</td>\n <td>False</td>\n <td>0.633324</td>\n <td>0.329777</td>\n <td>0.329777</td>\n <td>0.0</td>\n <td>-0.0</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n </tr>\n <tr>\n <th>2023-06-28</th>\n <td>28539.0</td>\n <td>21347.0</td>\n <td>NaN</td>\n <td>9.89</td>\n <td>NaN</td>\n <td>0.43</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>...</td>\n <td>False</td>\n <td>0.611284</td>\n <td>0.329777</td>\n <td>0.329777</td>\n <td>0.0</td>\n <td>-0.0</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n </tr>\n <tr>\n <th>2023-06-29</th>\n <td>28042.0</td>\n <td>21347.0</td>\n <td>NaN</td>\n <td>9.89</td>\n <td>NaN</td>\n <td>0.43</td>\n <td>1.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>0.0</td>\n <td>...</td>\n <td>True</td>\n <td>0.600638</td>\n <td>0.329777</td>\n <td>0.329777</td>\n <td>0.0</td>\n <td>-0.0</td>\n <td>False</td>\n <td>False</td>\n <td>True</td>\n <td>True</td>\n </tr>\n </tbody>\n</table>\n<p>2402 rows × 36 columns</p>\n</div>"
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_deeprockgalactic = processing.process_game(\"Deep Rock Galactic\", Path(\"data/deeprockgalactic\"))\n",
"df_deeprockgalactic"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"ExecuteTime": {
"end_time": "2023-07-02T06:47:44.333257903Z",
"start_time": "2023-07-02T06:47:44.017726541Z"
}
},
"outputs": [
{
"data": {
"text/plain": " SteamDB · Peak concurrent players \\\n2020-12-25 2354.0 \n2020-12-26 2867.0 \n2020-12-27 2931.0 \n2020-12-28 2921.0 \n2020-12-29 2757.0 \n... ... \n2023-06-25 786.0 \n2023-06-26 780.0 \n2023-06-27 718.0 \n2023-06-28 696.0 \n2023-06-29 651.0 \n\n SteamDB · Day average of concurrent players \\\n2020-12-25 NaN \n2020-12-26 NaN \n2020-12-27 NaN \n2020-12-28 NaN \n2020-12-29 NaN \n... ... \n2023-06-25 613.0 \n2023-06-26 584.0 \n2023-06-27 580.0 \n2023-06-28 566.0 \n2023-06-29 566.0 \n\n SteamDB · Player count flags SteamDB · Steam \\\n2020-12-25 NaN 16.79 \n2020-12-26 NaN 16.79 \n2020-12-27 NaN 16.79 \n2020-12-28 NaN 16.79 \n2020-12-29 NaN 16.79 \n... ... ... \n2023-06-25 NaN 16.79 \n2023-06-26 NaN 16.79 \n2023-06-27 NaN 16.79 \n2023-06-28 NaN 16.79 \n2023-06-29 NaN 16.79 \n\n SteamDB · Price flags Google Trends · Score \\\n2020-12-25 NaN NaN \n2020-12-26 NaN NaN \n2020-12-27 NaN NaN \n2020-12-28 NaN NaN \n2020-12-29 NaN NaN \n... ... ... \n2023-06-25 NaN 0.57 \n2023-06-26 NaN 0.57 \n2023-06-27 NaN 0.57 \n2023-06-28 NaN 0.57 \n2023-06-29 NaN 0.57 \n\n Steam · Count of News tagged no_tags ITAD · Best Price \\\n2020-12-25 0.0 16.79 \n2020-12-26 0.0 16.79 \n2020-12-27 0.0 16.79 \n2020-12-28 0.0 16.79 \n2020-12-29 0.0 16.79 \n... ... ... \n2023-06-25 0.0 12.59 \n2023-06-26 0.0 12.59 \n2023-06-27 0.0 12.59 \n2023-06-28 0.0 12.59 \n2023-06-29 0.0 16.79 \n\n ITAD · Best Regular Price ITAD · Worst Regular Price ... \\\n2020-12-25 16.79 16.79 ... \n2020-12-26 16.79 16.79 ... \n2020-12-27 16.79 16.79 ... \n2020-12-28 16.79 16.79 ... \n2020-12-29 16.79 16.79 ... \n... ... ... ... \n2023-06-25 16.79 29.99 ... \n2023-06-26 16.79 29.99 ... \n2023-06-27 16.79 29.99 ... \n2023-06-28 16.79 29.99 ... \n2023-06-29 16.79 29.99 ... \n\n Steam · Is there News? SteamDB · Relative concurrent players \\\n2020-12-25 False 0.743525 \n2020-12-26 False 0.905559 \n2020-12-27 False 0.925774 \n2020-12-28 False 0.922615 \n2020-12-29 False 0.870815 \n... ... ... \n2023-06-25 False 0.248263 \n2023-06-26 False 0.246368 \n2023-06-27 False 0.226785 \n2023-06-28 False 0.219836 \n2023-06-29 False 0.205622 \n\n SteamDB · Relative Steam price ITAD · Relative Best Price \\\n2020-12-25 1.0 1.000000 \n2020-12-26 1.0 1.000000 \n2020-12-27 1.0 1.000000 \n2020-12-28 1.0 1.000000 \n2020-12-29 1.0 1.000000 \n... ... ... \n2023-06-25 1.0 0.749851 \n2023-06-26 1.0 0.749851 \n2023-06-27 1.0 0.749851 \n2023-06-28 1.0 0.749851 \n2023-06-29 1.0 1.000000 \n\n ITAD · Best price change from previous day \\\n2020-12-25 NaN \n2020-12-26 0.0 \n2020-12-27 0.0 \n2020-12-28 0.0 \n2020-12-29 0.0 \n... ... \n2023-06-25 0.0 \n2023-06-26 0.0 \n2023-06-27 0.0 \n2023-06-28 0.0 \n2023-06-29 4.2 \n\n SteamDB · Steam price change from previous day \\\n2020-12-25 NaN \n2020-12-26 -0.0 \n2020-12-27 -0.0 \n2020-12-28 -0.0 \n2020-12-29 -0.0 \n... ... \n2023-06-25 -0.0 \n2023-06-26 -0.0 \n2023-06-27 -0.0 \n2023-06-28 -0.0 \n2023-06-29 -0.0 \n\n SteamDB · Is there a discount? ITAD · Is there a discount? \\\n2020-12-25 False False \n2020-12-26 False False \n2020-12-27 False False \n2020-12-28 False False \n2020-12-29 False False \n... ... ... \n2023-06-25 False False \n2023-06-26 False False \n2023-06-27 False False \n2023-06-28 False False \n2023-06-29 False False \n\n Cumulative · Is something happening on Steam? \\\n2020-12-25 False \n2020-12-26 False \n2020-12-27 False \n2020-12-28 False \n2020-12-29 False \n... ... \n2023-06-25 False \n2023-06-26 False \n2023-06-27 False \n2023-06-28 False \n2023-06-29 False \n\n Cumulative · Is something happening? \n2020-12-25 False \n2020-12-26 False \n2020-12-27 False \n2020-12-28 False \n2020-12-29 False \n... ... \n2023-06-25 False \n2023-06-26 False \n2023-06-27 False \n2023-06-28 False \n2023-06-29 False \n\n[917 rows x 23 columns]",
"text/html": "<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>SteamDB · Peak concurrent players</th>\n <th>SteamDB · Day average of concurrent players</th>\n <th>SteamDB · Player count flags</th>\n <th>SteamDB · Steam</th>\n <th>SteamDB · Price flags</th>\n <th>Google Trends · Score</th>\n <th>Steam · Count of News tagged no_tags</th>\n <th>ITAD · Best Price</th>\n <th>ITAD · Best Regular Price</th>\n <th>ITAD · Worst Regular Price</th>\n <th>...</th>\n <th>Steam · Is there News?</th>\n <th>SteamDB · Relative concurrent players</th>\n <th>SteamDB · Relative Steam price</th>\n <th>ITAD · Relative Best Price</th>\n <th>ITAD · Best price change from previous day</th>\n <th>SteamDB · Steam price change from previous day</th>\n <th>SteamDB · Is there a discount?</th>\n <th>ITAD · Is there a discount?</th>\n <th>Cumulative · Is something happening on Steam?</th>\n <th>Cumulative · Is something happening?</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>2020-12-25</th>\n <td>2354.0</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>16.79</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>0.0</td>\n <td>16.79</td>\n <td>16.79</td>\n <td>16.79</td>\n <td>...</td>\n <td>False</td>\n <td>0.743525</td>\n <td>1.0</td>\n <td>1.000000</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n </tr>\n <tr>\n <th>2020-12-26</th>\n <td>2867.0</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>16.79</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>0.0</td>\n <td>16.79</td>\n <td>16.79</td>\n <td>16.79</td>\n <td>...</td>\n <td>False</td>\n <td>0.905559</td>\n <td>1.0</td>\n <td>1.000000</td>\n <td>0.0</td>\n <td>-0.0</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n </tr>\n <tr>\n <th>2020-12-27</th>\n <td>2931.0</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>16.79</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>0.0</td>\n <td>16.79</td>\n <td>16.79</td>\n <td>16.79</td>\n <td>...</td>\n <td>False</td>\n <td>0.925774</td>\n <td>1.0</td>\n <td>1.000000</td>\n <td>0.0</td>\n <td>-0.0</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n <td>False</td>\n </tr>\n <tr>\n <th>2020-12-28</th>\n <td>2921.0</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>16.79</td>\n <td>NaN</td>\n <td>NaN</td>\n <td>0.0</td>\n <td>16.79</td>\n