https://github.com/planetarium/NineChronicles.DataProvider
https://github.com/planetarium/NineChronicles.DataProvider
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/planetarium/NineChronicles.DataProvider
- Owner: planetarium
- Created: 2021-04-30T04:55:45.000Z (about 4 years ago)
- Default Branch: development
- Last Pushed: 2024-04-13T06:10:41.000Z (about 1 year ago)
- Last Synced: 2024-04-13T22:56:25.753Z (about 1 year ago)
- Language: C#
- Size: 6.17 MB
- Stars: 4
- Watchers: 11
- Forks: 16
- Open Issues: 18
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# NineChronicles DataProvider
- NineChronicles.DataProvider is an off-chain service that stores NineChronicles game action data to a database mainly for game analysis.
- Currently, this service only supports `MySQL` database.
## Table of Contents
- [Pre-requisite](#pre-requisite)
- [Run](#run)
- [Development Guide](#development-guide)
- [Current Table Descriptions](#current-table-descriptions)
- [Migrating Past Chain Data to MySQL Database](#migrating-past-chain-data-to-mysql-database)
## Pre-requisite
- [MySQL](https://www.mysql.com/) and [Entity Framework Core tools](https://learn.microsoft.com/en-us/ef/core/cli/dotnet) should be installed in the local machine.
## Run
- Before running the program, please refer to the option values in the latest official [9c-launcher-config.json](https://release.nine-chronicles.com/9c-launcher-config.json) and fill out the variables in [appsettings.json](https://github.com/planetarium/NineChronicles.DataProvider/blob/development/NineChronicles.DataProvider.Executable/appsettings.json).
- In [appsettings.json](https://github.com/planetarium/NineChronicles.DataProvider/blob/development/NineChronicles.DataProvider.Executable/appsettings.json), `AppProtocolVersionToken`, `StorePath`, `PeerStrings`, `MySqlConnectionString` properties **MUST** be filled to run the program.
- To setup the NineChronicles mainnet blockchain store to use in the `StorePath`, download and extract the latest [mainnet snapshot](http://snapshots.nine-chronicles.com/main/partition/full/9c-main-snapshot.zip) to a desired location.
```
$ dotnet run --project ./NineChronicles.DataProvider.Executable/ --
```
## Development Guide
- This section lays out the steps in how to log a new action in the database.
- The [TransferAsset](https://github.com/planetarium/lib9c/blob/development/Lib9c/Action/TransferAsset.cs) action is used as an example in this guide.
### 1. Setup Database
- To setup the database, navigate to [NineChronicles.DataProvider/NineChronicles.DataProvider.Executable](https://github.com/planetarium/NineChronicles.DataProvider/tree/development/NineChronicles.DataProvider.Executable) directory on your terminal and run the following migration command.
```
dotnet ef database update -- [Connection String]
- Connection String example: "server=localhost;database=data_provider;port=3306;uid=root;pwd=root;"
```
### 2. Create Model
- In [NineChronicles.DataProvider/NineChronicles.DataProvider/Store/Models](https://github.com/planetarium/NineChronicles.DataProvider/tree/development/NineChronicles.DataProvider/Store/Models) directory, create a model file called `TransferAssetModel.cs`.
- In general, `TxId`, `BlockIndex`, `Date`, and `Timestamp` are useful to add as default properties in a model because these values will help with query speed when table size increases.
```
namespace NineChronicles.DataProvider.Store.Models
{
using System;
using System.ComponentModel.DataAnnotations;
public class TransferAssetModel
{
[Key]
public string? TxId { get; set; }
public long BlockIndex { get; set; }
public string? Sender { get; set; }
public string? Recipient { get; set; }
public decimal Amount { get; set; }
public DateOnly Date { get; set; }
public DateTimeOffset TimeStamp { get; set; }
}
}
```
- In [NineChronicles.DataProvider/NineChronicles.DataProvider/Store/NineChroniclesContext.cs](https://github.com/planetarium/NineChronicles.DataProvider/blob/development/NineChronicles.DataProvider/Store/NineChroniclesContext.cs), add a DbSet called `TransferAssets` and its description for reference.
```
// Table for storing TransferAsset actions
public DbSet TransferAssets => Set();
```
### 2. Create Store Method
- In [NineChronicles.DataProvider/NineChronicles.DataProvider/Store/MySqlStore.cs](https://github.com/planetarium/NineChronicles.DataProvider/blob/development/NineChronicles.DataProvider/Store/MySqlStore.cs), add a following method that stores the `TransferAsset` data into MySQL.
```
public void StoreTransferAsset(TransferAssetModel model)
{
using NineChroniclesContext ctx = _dbContextFactory.CreateDbContext();
TransferAssetModel? prevModel =
ctx.TransferAssets.FirstOrDefault(r => r.TxId == model.TxId);
if (prevModel is null)
{
ctx.TransferAssets.Add(model);
}
else
{
prevModel.BlockIndex = model.BlockIndex;
prevModel.Sender = model.Sender;
prevModel.Recipient = model.Recipient;
prevModel.Amount = model..Amount;
prevModel.Date = model.Date;
prevModel.TimeStamp = model.TimeStamp;
ctx.TransferAssets.Update(prevModel);
}
ctx.SaveChanges();
}
```
### 3. (Optional) Add Data Getter
In some cases, you need to handle state to get data to make model.
To do this easily, you can make your own data getter inside [NineChronicles.DataProvider/DataRendering/](https://github.com/planetarium/NineChronicles.DataProvider/tree/development/NineChronicles.DataProvider/DataRendering).
### 4. Render & Store Action Data
- In [NineChronicles.DataProvider/NineChronicles.DataProvider/RenderSubscriber.cs](https://github.com/planetarium/NineChronicles.DataProvider/blob/development/NineChronicles.DataProvider/RenderSubscriber.cs), add a following render code
```
_actionRenderer.EveryRender()
.Subscribe(ev =>
{
try
{
if (ev.Exception is null && ev.Action is { } transferAsset)
{
var model = new TransferAssetModel()
{
TxId = transferAsset.TxId,
BlockIndex = transferAsset.BlockIndex,
Sender = transferAsset.Sender,
Recipient = transferAsset.Recipient,
Amount = Convert.ToDecimal(transferAsset.Amount.GetQuantityString()),
Date = DateOnly.FromDateTime(_blockTimeOffset.DateTime),
TimeStamp = _blockTimeOffset,
};
MySqlStore.StoreTransferAsset(model);
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
```
### 5. Add Database Migration
- Navigate to [NineChronicles.DataProvider/NineChronicles.DataProvider.Executable](https://github.com/planetarium/NineChronicles.DataProvider/tree/development/NineChronicles.DataProvider.Executable) directory on your terminal and run the following migration command
```
dotnet ef migrations add AddTransferAsset -- [Connection String]
- Connection String example: "server=localhost;database=data_provider;port=3306;uid=root;pwd=root;"
```
## Current Table Descriptions
- Tables that `NineChronicles.DataProvider` stores data into are listed in [NineChroniclesContext.cs](https://github.com/planetarium/NineChronicles.DataProvider/blob/development/NineChronicles.DataProvider/Store/NineChroniclesContext.cs).
- Please refer to each `DbSet`'s comment in [NineChroniclesContext.cs](https://github.com/planetarium/NineChronicles.DataProvider/blob/development/NineChronicles.DataProvider/Store/NineChroniclesContext.cs) for table descriptions.
## Migrating Past Chain Data to MySQL Database
### Migrate All Action Data
- This command migrates all action data based on [DataRendering](https://github.com/planetarium/NineChronicles.DataProvider/blob/development/NineChronicles.DataProvider/DataRendering) to the designated MySQL database.
- Options such as `offset` and `limit` are provided to specify which block data to migrate.
- **IMPORTANT)** This migration tool requires you to have the necessary `state` data in the blocks you want to migrate (If your chain store lacks the `state` data, this tool will not work).
```
Usage: NineChronicles.DataProvider.Executable mysql-migration [--store-path ] [--mysql-server ] [--mysql-port ] [--mysql-username ] [--mysql-password ] [--mysql-database ] [--offset ] [--limit ] [--help]
Migrate action data in rocksdb store to mysql db.
Options:
-o, --store-path Rocksdb path to migrate. (Required)
--mysql-server A hostname of MySQL server. (Required)
--mysql-port A port of MySQL server. (Required)
--mysql-username The name of MySQL user. (Required)
--mysql-password The password of MySQL user. (Required)
--mysql-database The name of MySQL database to use. (Required)
--offset offset of block index (no entry will migrate from the genesis block).
--limit limit of block count (no entry will migrate to the chain tip).
-h, --help Show help message
```
### Migrate Battle Arena Ranking Data
- This command calculates the `battle arena` ranking of all participants at a specific block index (`migration-block-index`) and inserts the data to a designated mysql database.
```
Usage: NineChronicles.DataProvider.Executable battle-arena-ranking-migration [--store-path ] [--mysql-server ] [--mysql-port ] [--mysql-username ] [--mysql-password ] [--mysql-database ] [--migration-block-index ] [--help]
Migrate battle arena ranking data at a specific block index to a mysql database.
Options:
-o, --store-path RocksDB store path to migrate. (Required)
--mysql-server Hostname of MySQL server. (Required)
--mysql-port Port of MySQL server. (Required)
--mysql-username Name of MySQL user. (Required)
--mysql-password Password of MySQL user. (Required)
--mysql-database Name of MySQL database. (Required)
--migration-block-index Block index to migrate.
-h, --help Show help message
```
### Migrate User Staking Data
- This command calculates the `NCG staking amount` of all participants at a specific block index (`migration-block-index`) and inserts the data to a designated mysql database.
- Currently, the `slack-token` and `slack-channel` options for sending the data in `csv` format are required, however, these will be changed to optional in the near future.
```
Usage: NineChronicles.DataProvider.Executable user-staking-migration [--store-path ] [--mysql-server ] [--mysql-port ] [--mysql-username ] [--mysql-password ] [--mysql-database ] [--
slack-token ] [--slack-channel ] [--migration-block-index ] [--help]
Migrate staking amounts of users at a specific block index to a mysql database.
Options:
-o, --store-path Rocksdb store path to migrate. (Required)
--mysql-server Hostname of MySQL server. (Required)
--mysql-port Port of MySQL server. (Required)
--mysql-username Name of MySQL user. (Required)
--mysql-password Password of MySQL user. (Required)
--mysql-database Name of MySQL database to use. (Required)
--slack-token slack token to send the migration data. (Required)
--slack-channel slack channel that receives the migration data. (Required)
--migration-block-index Block index to migrate.
-h, --help Show help message
```