Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/skecskes/nftdatacollector

Get some NFT Collections stats and some social media stats
https://github.com/skecskes/nftdatacollector

Last synced: 3 days ago
JSON representation

Get some NFT Collections stats and some social media stats

Awesome Lists containing this project

README 

        
# Task



- Pick 5 NFT collections of your choice

- Collect social media data and build metric of your choice that could be used in a pricing model

- The test should not take more than 2 hours, anything you cannot do, please write bullet points to explain your next steps

Extra:

- questions to clarify assumptions

- design recommendations

- ideas of metrics/features for pricing model



#Schemas



## OpenSea



### Asset



    id, integer

    token_id , string

    name, string

    last_sale_total_price, string/null

    last_sale_event_type, string/null

    last_sale_auction_type, string/null

    last_sale_event_timestamp, string/null

    last_sale_quantity, int/null

    asset_contract_address, string

    asset_contract_created_date, string

    asset_contract_name, string

    asset_contract_schema_name, string

    collection_creation_date, string/null

    collection_discord_url, string

    collection_name, string

    collection_slug, string

    owner_address, string



This resource is created by `Ingestion.java` and result can be found in `spark-warehouse/ingestion` folder. Unfortunately I still didn't receive my apikey for opensea and requests are failing due to rate limiter. I was able to pull data from one collection only.



### Collection



    created_date, string

    description, string

    discord_url, string

    external_url, string

    name, string

    slug, string

    stats_count, int

    stats_num_owners, int

    stats_average_price, float

    stats_market_cap, float

    stats_total_sales, int

    stats_total_volume, float

    primary_asset_contracts_address, string

    primary_asset_contracts_symbol, string

    primary_asset_contracts_opensea_version, string



Parts of this schema are ingested through ingestion process into `Asset` schema. If needed, we could pull more details from json responses.



### LastSale



    total_price, string

    quantity, integer

    event_timestamp, string

    event_type, string

    auction_type, string

    successful, boolean



Some parts of LastSale were added to `Asset` schema through `ingestion` process. We could pull more details if needed. 



## Twitter



    author_id, string

    created_at, string

    id, string

    lang, string

    text, string



Twitter search is done with SocialEnrichment.java and that was last thing I managed to deliver in 2 hours timeframe. I can also see that tweets search returned a lot of irrelevant bot activity tweets. That would need to be filtered down and query improved.

Example of tweets csv should be in `spark-warehouse/tweets` folder.

    

TODO:

- list of retweet users

- list of like users

Once we get the list of tweets above we should make another call to Twitter api to get more details about each tweet (likes, retweets). That should be fairly easy, but I run out of time.



### Transaction



- token_id

- minted on OS or not

- collection name

- collection id

- list of transactions

- last price data

- list of auctions



TODO: Run out of time to implement transactions, the transactions could be pulled down from opensea (https://docs.opensea.io/reference/retrieving-orders), or alternatively we could use third party API that reads transactions from etherum network directly, because we know contract and id of token.



#Use cases



- auction price vs last transaction price

- average of last 3

- timelapse between last and last 3(closeness metric)



# Insight into current state 



- monax is building pipeline for the analytics that will power part of their product



# Design



I created two spark applications, first will ingest data about nft tokens and second will pull data from twitter about 

corresponding nft collections. We could pull more data about transactions from opensea or other 3rp party api to be able

to run som models on data.

Outcome csv should be in `spark-warehouse` folders.

Going to future, we would deploy this app as an AWS stepfunction that could be triggered by events. The AWS step 

function would consist of java spark jobs (either the 2 that I created in this demo: `Ingestion.java`& 

`SocialEnrichment.java` ) or we could create more jobs as we need. The spark would run in EMR cluster. EMR cluster is

hadoop fleet of EC2 instances that work in distributed manner on our jobs and after the jobs are done, then 

the EMR cluster would be terminated. The output would be stored in s3 buckets as parquet files, so that we can do 

further processing on our internal data, or create own data warehouse, or we could create athena tables using AWS glue

for analysis run by Scientists.



PS: There are just so many improvements that we can do with this, so the best would be to have a chat and answer your

questions directly



Best regards,

Stefan Kecskes