{"id":13652519,"url":"https://github.com/metaplex-foundation/digital-asset-validator-plugin","last_synced_at":"2025-04-12T20:46:11.965Z","repository":{"id":43362831,"uuid":"511161350","full_name":"metaplex-foundation/digital-asset-validator-plugin","owner":"metaplex-foundation","description":"The NFT Geyser plugin that powers metaplex APIs","archived":false,"fork":false,"pushed_at":"2025-03-26T21:16:06.000Z","size":1938,"stargazers_count":43,"open_issues_count":1,"forks_count":26,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-04T00:04:27.389Z","etag":null,"topics":["blockchain","metaplex","nft","rust","solana"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/metaplex-foundation.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-06T14:03:30.000Z","updated_at":"2025-03-26T20:57:21.000Z","dependencies_parsed_at":"2024-01-12T19:47:27.539Z","dependency_job_id":"474a62fa-7005-41a2-ae55-1c7fc2d25a58","html_url":"https://github.com/metaplex-foundation/digital-asset-validator-plugin","commit_stats":{"total_commits":604,"total_committers":25,"mean_commits":24.16,"dds":0.7433774834437086,"last_synced_commit":"4ce246ff2d5b2684ee351d66408651f6c7bb3a84"},"previous_names":[],"tags_count":64,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metaplex-foundation%2Fdigital-asset-validator-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metaplex-foundation%2Fdigital-asset-validator-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metaplex-foundation%2Fdigital-asset-validator-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metaplex-foundation%2Fdigital-asset-validator-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/metaplex-foundation","download_url":"https://codeload.github.com/metaplex-foundation/digital-asset-validator-plugin/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248631702,"owners_count":21136559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blockchain","metaplex","nft","rust","solana"],"created_at":"2024-08-02T02:01:00.051Z","updated_at":"2025-04-12T20:46:11.936Z","avatar_url":"https://github.com/metaplex-foundation.png","language":"Rust","funding_links":[],"categories":["Code"],"sub_categories":["Indexers"],"readme":"## Digital Asset Validator Plugin\n\nThis repo houses a validator plugin that is a key part of the Metaplex's Digital Asset RPC API. It is responsible for getting\ninformation out of the Solana validator and sending it to a message bus in a binary format. While this component was\nbuilt to serve the APIs, it was designed to allow any message bus tech to be used. That being said, it can be used for many use cases.\n\n## WARNING\n\n```\nSolana 1.10.41 or greater is required. Your calling components must support V2 GeyserPlugin types\n```\n\nIt is built on the following principles.\n\n- Do as little work in the validator process as possible.\n- Allow any message bus tech to work.\n- Opinionated and efficient Wire format as a standard.\n- Async first\n\n### Components\n\n1. Plerkle -\u003e Geyser Plugin that sends raw information to a message bus using Messenger\n2. Messenger -\u003e A message bus agnostic Messaging Library that sends Transaction, Account, Block and Slot updates in the Plerkle Serialization format.\n3. Plerkle Serialization -\u003e FlatBuffers based serialization code and schemas. This is the wire-format of Plerkle.\n4. Plerkle Snapshot -\u003e ETL for Solana accounts snapshot.\n\n## Developing\n\nIf you are building the Metaplex RPC API infrastructure please follow the instructions in [Metaplex RPC API infrastructure](https://github.com/metaplex-foundation/digital-asset-rpc-infrastructure).\n\nIf you are using this plugin for your bespoke use case then the build steps are below.\n\n#### A note on formatting\n\nSince `rustfmt.toml` uses unstable configuration options, it is required to run formatting with the nightly toolchain: `cargo +nightly fmt`.\n\n### Building Locally\n\n#### Linux\n\n`cargo build` for debug or\n`cargo build --release` for a release build.\n\nYou will now have a libplerkle.so file in the target folder.\n\n#### Mac\n\nBuilding is similar to Linux, except for the extension of the library produced.\nInstead of a `.so` file, look for `libplerkle.dylib`. The loader does not really care what extension to link, as long as it's a proper dynamically linked object, such as a `dylib`.\n\n### Configuration\n\n```bash\n--geyser-plugin-config plugin-config.json\n```\n\nThe plugin config for plerkle must have this format, but you can put whatever keys you want\n\n```json\nEXAMPLE PLEASE DONT CONSIDER THIS THE PERFECT CONFIG\n{\n  \"libpath\": \"/.../libplerkle.so\",\n  \"enable_metrics\": false,\n  \"env\": \"local\",\n  \"handle_startup\": true, // set to false if you dont want initial account flush\n  \"accounts_selector\": {\n    \"owners\": [\n      \"metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s\"\n    ]\n  },\n  \"transaction_selector\": {\n    \"mentions\": [\n      \"BGUMAp9Gq7iTEuizy4pqaxsTyUCBK68MDfK752saRPUY\"\n    ]\n  }\n}\n```\n\nThis config file points to where your plugin library file is, and what programs it is listening to.\nThis is the standard Solana geyser plugin config file that the validator reads.\n\nThere are some other bits of configuration needed. Environment Variables.\nThe process running the validator must have access to environment variables. Those variables are as follows\n\n```bash\nRUST_LOG=warn\nPLUGIN_MESSENGER_CONFIG={ messenger_type=\"Redis\", connection_config={ redis_connection_str=\"redis://redis\" } }\n```\n\nThe PLUGIN_MESSENGER_CONFIG determines which compiled messenger to select and a specific configuration for the messenger.\n\n#### Additional Configuration Examples\n\n**_Producer Configuration_**\n\n- \"pipeline_size_bytes\" - Maximum command size, roughly equates to the payload size. This setting locally buffers bytes in a queue to be flushed when the buffer grows past the desired amount. Default is 512MB (max redis command size) / 1000, maximum is 512MB (max redis command size) / 1000. You should test your optimal size to avoid high send latency and avoid RTT.\n- \"local_buffer_max_window\" - Maximum time to wait for the buffer to fill be for flushing. For lower traffic you dont want to be waiting around so set a max window and it will send at a minumum of every X milliseconds . Default 10\n- \"confirmation_level\" - Can be one of \"Processed\", \"Confirmed\", \"Rooted\". Defaults to Processed which is the level we wait for before sending. \"Processed\" is essentially when we first see it which can on rare cases be reverted. \"Confirmed\" has extremley low likley hood of being reverted but takes longer (~1k ms in our testing) to show up. \"Rooted\" is impossible to revert but takes the longest.\n- \"num_workers\" - This is the number of workers who will pickup notifications from the plugin and send them to the messenger. Default is 5\n- \"account_stream_size\" - default value 100_000_000\n- \"slot_stream_size\" - default value 100_000\n- \"transaction_stream_size\" - default value 10_000_000\n- \"block_stream_size\" - default value 100_000\n\n```\nLower Scale Low network latency\n\nPLUGIN_MESSENGER_CONFIG={pipeline_size_bytes=1000000,local_buffer_max_window=10, messenger_type=\"Redis\", connection_config={ redis_connection_str=\"redis://redis\" } }\n\nHigh Scale Higher latency\n\nPLUGIN_MESSENGER_CONFIG={pipeline_size_bytes=50000000,local_buffer_max_window=500, messenger_type=\"Redis\", connection_config={ redis_connection_str=\"redis://redis\" } }\n\n\n```\n\n**_Consumer Configuration_**\n\n- \"retries\" - Amount of times to deliver the message. If delivered this many times and not ACKed, then it is deleted\n- \"batch_size\" - Max Amout of messages to grab within the wait timeout window.\n- \"message_wait_timeout\" - Amount of time the consumer will keep the stream open and wait for messages\n- \"idle_timeout\" - Amount of time a consumer can have the message before it goes back on the queue\n- \"consumer_id\" - VERY important. This is used to scale horizontally so messages arent duplicated over instances.Make sure this is different per instance\n\n```\n\nPLUGIN_MESSENGER_CONFIG={batch_size=1000, message_wait_timeout=5, retries=5, consumer_id=\"random_string\",messenger_type=\"Redis\", connection_config={ redis_connection_str=\"redis://redis\" } }\nPLUGIN_ACCOUNT_STREAM_SIZE=250000000\nPLUGIN_SLOT_STREAM_SIZE=250000\nPLUGIN_TRANSACTION_STREAM_SIZE=25000000\nPLUGIN_BLOCK_STREAM_SIZE=250000\n\n```\n\nNOTE: in 1.4.0 we are not sending to slot status.\n\n### Metrics\n\nThe plugin exposes the following statsd metrics\n\n- count plugin.startup -\u003e times the plugin started\n- time message_send_queue_time -\u003e time spent on messenger internal buffer\n- time message_send_latency -\u003e rtt time to messenger bus\n- count account_seen_event , tags: owner , is_startup -\u003e number of account events filtered and seen\n- time startup.timer -\u003e startup flush timer\n- count transaction_seen_event tags slot-idx -\u003e number of filtered txns seen\n\n### Building With Docker\n\nThis repo contains a docker file that allows you to run and test the plerkle plugin using a test validator.\nTo test it you can build the container with`docker compose build` and run it with `docker compose up`.\n\nYou will want to change the programs you are listening to in `./docker/runs.sh`. Once you spin up the validator send your transactions to the docker host as you would a normal RPC.\n\nAny program .so files you add to the /so/ file upon running the docker compose system will be added to the local validator.\n\nYou need to name the so file what you want the public key to be:\n\n```bash\nmetaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s.so\n```\n\nThis is because of this line in the `docker-compose.yml` file.\n\n```yaml\n- ./programs:/so/:ro\n```\n\nYou can comment this out if you dont want it.\n\n### Using docker output for solana and geyser artifacts\n\nYou can run `./docker/build.sh` to create a build of the geyser plugin and solana, then output the artifacts\nto your local filesystem so you can run the validator.\n\nNote that differences in image distro could cause incompatible GLibc versions.\n\n### Crates\n\nNOTE WE DO NOT PUBLISH THE PLUGIN ANY MORE:\n\nplerkle_messenger-https://crates.io/crates/plerkle_messenger\nplerkle_serialization-https://crates.io/crates/plerkle_serialization\n\n## Snapshot ETL\n\nThe Plerkle snapshot tool can be used for parsing Solana account snapshots. The repository already includes a pre-configured `accounts-selector-config.json` file, which is ready to use. The only thing you might want to modify is the list of programs in `accounts-selector-config.json`; otherwise, you can leave the configurations as they are.\n\nBefore running the tool, it's important to create an .env file, modeled after .env.example. In this file, you should specify the path to the directory containing the snapshots as well as the snapshot redis connection details.\n\nOnce everything is set up, you can build the Docker container for ETL by running:\n\n```\nmake build\n```\n\nThis will create a Docker container with the Geyser plugin and ETL fully built and ready to use.\n\nThe next step is to run the ETL:\n\n```\nmake stream\n```\n\nThis command will launch the ETL Docker container. It will load the snapshot archives, the Geyser plugin binary, and stream all the accounts from the snapshot to the plugin.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetaplex-foundation%2Fdigital-asset-validator-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmetaplex-foundation%2Fdigital-asset-validator-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetaplex-foundation%2Fdigital-asset-validator-plugin/lists"}