{"id":22072562,"url":"https://github.com/broxus/ton-kafka-producer","last_synced_at":"2025-07-24T11:30:39.996Z","repository":{"id":66355214,"uuid":"447336030","full_name":"broxus/ton-kafka-producer","owner":"broxus","description":"A simple service to stream Everscale data into Kafka","archived":false,"fork":false,"pushed_at":"2024-09-07T14:26:04.000Z","size":1040,"stargazers_count":7,"open_issues_count":1,"forks_count":6,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-09-08T12:05:48.758Z","etag":null,"topics":["blockchain","everscale","kafka","rpc","venom-blockchain","venom-developer-program","venom-indexer"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/broxus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-12T18:55:58.000Z","updated_at":"2024-09-07T14:26:07.000Z","dependencies_parsed_at":"2023-10-03T02:56:41.824Z","dependency_job_id":"9cfa5182-455e-469f-995b-bc34e959eeb7","html_url":"https://github.com/broxus/ton-kafka-producer","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/broxus%2Fton-kafka-producer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/broxus%2Fton-kafka-producer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/broxus%2Fton-kafka-producer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/broxus%2Fton-kafka-producer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/broxus","download_url":"https://codeload.github.com/broxus/ton-kafka-producer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227432007,"owners_count":17775893,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blockchain","everscale","kafka","rpc","venom-blockchain","venom-developer-program","venom-indexer"],"created_at":"2024-11-30T21:13:44.259Z","updated_at":"2024-11-30T21:13:46.436Z","avatar_url":"https://github.com/broxus.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/venom-blockchain/developer-program\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/venom-blockchain/developer-program/main/vf-dev-program.png\" alt=\"Logo\" width=\"366.8\" height=\"146.4\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n# ton-kafka-producer\n\nThe indexing infrastructure for TVM-compatible blockchains includes a node\navailable via [jRPC](https://github.com/broxus/everscale-jrpc) and indexer\nservices with some high-level APIs for each dApp we want to index. The latter\ndoesn’t fetch needed messages from the former. Instead, we use Kafka to organize\na stable, consistent and ordered queue of records from the node and deliver it\nto arbitrary number of indexing services.\n\nThe Kafka producer is a software component that connects to the blockchain node\nand deliver data to Kafka brokers, which are responsible for storing and\nreplicating it across a Kafka cluster. Resulting stream includes information\nabout transactions, blocks, and other relevant data from the blockchain network.\n\nBy organizing data in Kafka topics, the system ensures that data is properly\nordered and available to indexer services. Thus, system can handle heavy loads,\nensuring that each indexer database is in sync with the blockchain network\n\nIt provides three different methods of scanning blockchain data:\n\n- `NetworkScanner` scans data from a running node. It uses Indexer to retrieve\n  the blockchain data and scans the data using various network protocols, such\n  as ADNL, RLD, and DHT. It then sends the scanned data to a Kafka broker. This\n  method requires a running TON node and access to its data.\n\n- `ArchivesScanner` scans data from local disk archives. It reads the blockchain\n  data from the archive files and sends the data to a Kafka broker. This method\n  requires a local copy of the blockchain archives.\n\n- `S3Scanner` scans data from S3 storage. It reads the blockchain data from the\n  specified S3 bucket and sends the data to a Kafka broker. This method requires\n  access to an S3 bucket containing blockchain data.\n\n### Runtime requirements\n\n- CPU: 4 cores, 2 GHz\n- RAM: 8 GB\n- Storage: 100 GB fast SSD\n- Network: 100 MBit/s\n\n### How to run\n\n1. Build all binaries and prepare services\n   ```bash\n   ./scripts/setup.sh\n   ```\n2. Edit `/etc/ton-kafka-producer/config.yaml`\n3. Enable and start the service:\n   ```bash\n   systemctl enable ton-kafka-producer\n   systemctl start ton-kafka-producer\n   ```\n\n### Config example\n\nThe example configuration includes settings that specify how the Kafka producer\nshould connect to Kafka brokers, as well as options for securing the connection\nusing SASL/SSL protocols. It also includes settings for the scan type, which\ndetermines how the producer retrieves data from the TON node.\n\n```yaml\n---\n# Optional states endpoint (see docs below)\nrpc_config:\n  # States RPC endpoint\n  listen_address: \"0.0.0.0:8081\"\n  generate_stub_keyblock: true\n  # Minimal JRPC API:\n  type: simple\n  # # Or full JRPC API:\n  # type: full\n  # persistent_db_path: \"/var/db/jrpc-storage\"\n  # # Virtual shards depth to use during shard state accounts processing\n  # shard_split_depth: 4\n  # # Specify options to enable the transactions GC (disabled by default)\n  # transactions_gc_options:\n  #   # For at least how long to store transactions (in seconds)\n  #   ttl_sec: 1209600\n  #   # GC invocation interval (in seconds)\n  #   interval_sec: 3600\n\nmetrics_settings:\n  # Listen address of metrics. Used by the client to gather prometheus metrics.\n  # Default: \"127.0.0.1:10000\"\n  listen_address: \"0.0.0.0:10000\"\n  # Metrics update interval in seconds. Default: 10\n  collection_interval_sec: 10\n\n# # Scan from local archives\n# scan_type:\n#   kind: FromArchives\n#   # Example how to prepare: `find path/to/archives \u003e path/to/archives_list`\n#   list_path: path/to/archives_list\n\nscan_type:\n  kind: FromNetwork\n  node_config:\n    # Root directory for node DB. Default: \"./db\"\n    db_path: \"/var/db/ton-kafka-producer\"\n\n    # UDP port, used for ADNL node. Default: 30303\n    adnl_port: 30000\n\n    # Path to temporary ADNL keys.\n    # NOTE: Will be generated if it was not there.\n    # Default: \"./adnl-keys.json\"\n    temp_keys_path: \"/etc/ton-kafka-producer/adnl-keys.json\"\n\n    # Archives map queue. Default: 16\n    parallel_archive_downloads: 32\n\n    # archive_options:\n    #   # Archives S3 uploader options\n    #   uploader_options:\n    #     name: \"\"\n    #     endpoint: \"http://127.0.0.1:9000\"\n    #     bucket: \"archives\"\n    #     credentials:\n    #       access_key: \"example_key\"\n    #       secret_key: \"example_password\"\n\n    # # Specific block from which to run the indexer\n    # start_from: 12365000\n\n    # Manual rocksdb memory options (will be computed from the\n    # available memory otherwise).\n    # db_options:\n    #   rocksdb_lru_capacity: \"512 MB\"\n    #   cells_cache_size: \"4 GB\"\n\n    # Everscale specific network settings\n    adnl_options:\n      use_loopback_for_neighbours: true\n      force_use_priority_channels: true\n    rldp_options:\n      force_compression: true\n    overlay_shard_options:\n      force_compression: true\n\nkafka_settings:\n  mode: broxus # it will try to write to 0..8 partition in topic, so, make sure that you have all set.\n  raw_transaction_producer:\n    topic: everscale-transactions\n    brokers:\n      \"kafka1.my.website:20001, kafka1.my.website:20002, kafka1.my.website:20003\"\n    attempt_interval_ms: 100\n    security_config:\n      Sasl:\n        security_protocol: \"SASL_SSL\"\n        ssl_ca_location: \"client.pem\"\n        sasl_mechanism: \"sasl mechanism\"\n        sasl_username: \"your sasl username\"\n        sasl_password: \"your sasl password\"\n# OR gql kafka producer if you want to get json messages\n#\n#kafka_settings:\n#  mode: gql\n#  requests_consumer:\n#    topic: gql.requests\n#    brokers: \"1.2.3.4:20001, 1.2.3.4:20002, 1.2.3.4:20003\"\n#    group_id: gql-mainnet\n#    session_timeout_ms: 6000\n#  block_producer:\n#    topic: gql.blocks\n#    brokers: \"1.2.3.4:20001, 1.2.3.4:20002, 1.2.3.4:20003\"\n#    attempt_interval_ms: 100\n#    message_max_size: 4000000\n#  message_producer:\n#    topic: gql.messages\n#    brokers: \"1.2.3.4:20001, 1.2.3.4:20002, 1.2.3.4:20003\"\n#    attempt_interval_ms: 100\n#    message_max_size: 4000000\n#  transaction_producer:\n#    topic: gql.transactions\n#    brokers: \"1.2.3.4:20001, 1.2.3.4:20002, 1.2.3.4:20003\"\n#    attempt_interval_ms: 100\n#    message_max_size: 4000000\n#  account_producer:\n#    topic: gql.accounts\n#    brokers: \"1.2.3.4:20001, 1.2.3.4:20002, 1.2.3.4:20003\"\n#    attempt_interval_ms: 100\n#    message_max_size: 4000000\n#  block_proof_producer:\n#    topic: gql.blocks_signatures\n#    brokers: \"1.2.3.4:20001, 1.2.3.4:20002, 1.2.3.4:20003\"\n#    attempt_interval_ms: 100\n#    message_max_size: 4000000\n```\n\n## Contributing\n\nWe welcome contributions to the project! If you notice any issues or errors, feel free to open an issue or submit a pull request.\n\n## License\n\nLicensed under GPL-3.0 license ([LICENSE](/LICENSE) or https://opensource.org/license/gpl-3-0/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbroxus%2Fton-kafka-producer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbroxus%2Fton-kafka-producer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbroxus%2Fton-kafka-producer/lists"}