{"id":18953162,"url":"https://github.com/0xpolygon/zkevm-data-streamer","last_synced_at":"2025-04-16T02:31:57.426Z","repository":{"id":191868085,"uuid":"685501024","full_name":"0xPolygon/zkevm-data-streamer","owner":"0xPolygon","description":"Go implementation of a generic fast data streamer library. ","archived":false,"fork":false,"pushed_at":"2025-02-05T19:27:11.000Z","size":1093,"stargazers_count":13,"open_issues_count":8,"forks_count":19,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-03-29T05:34:51.018Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/0xPolygon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-31T11:27:00.000Z","updated_at":"2025-02-05T19:26:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"8bc26742-d91e-42b4-b38d-262546bb526c","html_url":"https://github.com/0xPolygon/zkevm-data-streamer","commit_stats":null,"previous_names":["0xpolygonhermez/zkevm-data-streamer","0xpolygon/zkevm-data-streamer"],"tags_count":53,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xPolygon%2Fzkevm-data-streamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xPolygon%2Fzkevm-data-streamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xPolygon%2Fzkevm-data-streamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0xPolygon%2Fzkevm-data-streamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/0xPolygon","download_url":"https://codeload.github.com/0xPolygon/zkevm-data-streamer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249184542,"owners_count":21226398,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T13:36:53.842Z","updated_at":"2025-04-16T02:31:56.837Z","avatar_url":"https://github.com/0xPolygon.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# zkEVM DataStreamer\n\nGo implementation of a generic fast data streamer library. \n\n**Use case for the zkEVM Node**: Sequencer data stream service to stream L2 blocks and L2 txs.\n\n![zkEVM DataSreamer diagram](doc/data-streamer.png)\n\n## GENERAL DIAGRAM\nDescription of the diagram elements:\n\n- **Sequencer**: uses the data streamer library through the API to send L2 blocks and L2 transactions to the stream. It's an use case of the datastreamer component.\n\n- **Data Streamer**: creates the stream server, it manages the stream file and allows stream clients to connect using a TCP socket. For such stream clients, it offers a series of commands via the TCP port.\n\n- **Stream File**: stores all the streaming data. It is a custom binary file organized into data pages with a preceding header page.\n\n- **Bookmarks DB**: stores an index for the bookmarks that are available in the streaming file. It uses a LevelDB using the library [goleveldb](https://github.com/syndtr/goleveldb).\n\n- **Streamer Client**: creates a stream client, it manages the data streamer connection and allows to execute the TCP commands available in the server.\n\n## STREAM FILE\nThe binary file format is described below.\n\n### Header page\n- The first page is the header. \n- Header page size = 4096 bytes\n\n#### Magic numbers\nAt the beginning of the file there are the following magic bytes (file signature): `polygonDATSTREAM`\n\n#### HEADER ENTRY format (HeaderEntry)\n\u003eu8 packetType = 1 // 1:Header  \n\u003eu32 headerLength = 38 // Total length of header entry  \n\u003eu8 Version // Stream file version (starting from 1)  \n\u003eu64 SystemID // E.g.: ChainID  \n\u003eu64 streamType // 1:Sequencer  \n\u003eu64 TotalLength // Total bytes used in the file  \n\u003eu64 TotalEntries // Total number of data entries  \n\n### Data page\n- From the second page starts the data pages.  \n- Page size = 1 MB\n\n#### DATA ENTRY format (FileEntry)\n\u003eu8 packetType // 2:Data entry, 0:Padding  \n\u003eu32 Length // Total length of data entry (17 bytes + length(data))  \n\u003eu32 Type // 0xb0:Bookmark, 1:Event1, 2:Event2,...  \n\u003eu64 Number // Entry number (sequential starting with 0)  \n\u003eu8[] data  \n\nNOTE: If an entry does not fit in the remaining page space, the entry will be stored in the next page.\n\n### File diagram\n![Alt](doc/data-streamer-bin-file.drawio.png)\n\n## STREAM TCP COMMANDS\n- All the commands available for the stream clients return first a response, a `Result` entry defined in a later section.\n- Some commands like `Start` or `Header` may return more data.\n\nBelow is the detail of the available commands:\n\n### Start \nSyncs from the entry number (`fromEntryNumber`) and starts receiving data streaming from that entry. \n\nCommand format sent by the client:\n\u003eu64 command = 1  \n\u003eu64 streamType // e.g. 1:Sequencer  \n\u003eu64 fromEntryNumber  \n\nIf already started terminates the connection.\n\n### StartBookmark\nSyncs from the bookmark (`fromBookmark`) and starts receiving data streaming from the entry pointed by that bookmark.\n\nCommand format sent by the client:\n\u003eu64 command = 4  \n\u003eu64 streamType // e.g. 1:Sequencer  \n\u003eu32 bookmarkLength // Length of fromBookmark (Max bookmark length value is 16)  \n\u003eu8[] fromBookmark  \n\nIf already started or `bookmarkLength` exceeds the maximum length, terminates the connection.\n\n### Stop\nStops the reception of the streaming transmission.\n\nCommand format sent by the client:\n\u003eu64 command = 2  \n\u003eu64 streamType // e.g. 1:Sequencer  \n\nIf not started terminates the connection.\n\n### GetHeader \nGets the current stream file header (`HeaderEntry` format defined in the [STREAM FILE](#stream-file) section), so stream clients can know the total number of entries and the size of the stream file.\n\nCommand format sent by the client:\n\u003eu64 command = 3  \n\u003eu64 streamType // e.g. 1:Sequencer  \n\nIf streaming already started terminates the connection.\n\n### GetEntry\nGets the data from the entry (`entryNumber`) in the format `FileEntry` defined in the [STREAM FILE](#stream-file) section).\n\nCommand format sent by the client:\n\u003eu64 command = 4  \n\u003eu64 streamType // e.g. 1:Sequencer  \n\u003eu64 entryNumber  \n\nIf streaming already started terminates the connection.\n\n### GetBookmark\nGets the data from the entry pointed by the bookmark (`bookmark`) in the format `FileEntry` defined in the [STREAM FILE](#stream-file) section).\n\nCommand format sent by the client:\n\u003eu64 command = 5  \n\u003eu64 streamType // e.g. 1:Sequencer  \n\u003eu32 bookmarkLength // Length of bookmark (Max bookmark length value is 16)  \n\u003eu8[] bookmark  \n\nIf streaming already started or `bookmarkLength` exceeds the maximum length, terminates the connection.\n\n### RESULT FORMAT (ResultEntry)\nRemember that all these TCP commands firstly return a response in the following detailed format:\n\u003eu8 packetType // 0xff:Result  \n\u003eu32 length // Total length of the entry  \n\u003eu32 errorNum // Error code (0:OK)  \n\u003eu8[] errorStr\n\n## BOOKMARKS\nBookmarks make possible to the clients to sync the streaming from a business logic point.\n- No need to store the latest `stream entry number` received.\n- Using the API, bookmarks to business logic data are added in the send data to stream implementation.\n- e.g. zkEVM Sequencer streaming: each L2 block number has its own bookmark. Clients can request to start the stream from a L2 block number.\n\n## STREAM RELAY\nStream relay server included in the datastream library allows scaling the number of stream connected clients.\n\n![Datastream relay diagram](doc/data-streamer-relay.png)\n\n- **Data Streamer Relay** acts as a `stream client` towards the main data stream server, and also acts as a `stream server` towards the stream clients connected to it.\n\n\n## DATA STREAMER INTERFACE (API)\n### SERVER API\n- Create and start a datastream server (`StreamServer`) using the `NewServer` function followed by the `Start` function.\n- Send data to stream by starting an atomic operation through `StartAtomicOp`, adding entry events (`AddStreamEntry`) and bookmarks (`AddStreamBookmark`), and commit the operation `CommitAtomicOp`.\n\n#### Send data API\n- StartAtomicOp()  \n- AddStreamBookmark(u8[] bookmark) -\u003e returns u64 entryNumber  \n- AddStreamEntry(u32 entryType, u8[] data) -\u003e returns u64 entryNumber  \n- CommitAtomicOp()  \n- RollbackAtomicOp()  \n\n#### Query data API\n- GetHeader() -\u003e returns struct HeaderEntry\n- GetEntry(u64 entryNumber) -\u003e returns struct FileEntry\n- GetBookmark(u8[] bookmark) -\u003e returns u64 entryNumber\n- GetFirstEventAfterBookmark(u8[] bookmark) -\u003e returns struct FileEntry\n- GetDataBetweenBookmarks(bookmarkFrom []byte, bookmarkTo []byte) ([]byte, error) -\u003e returns the array of data, ignoring bookmarks, between the given ones\n\n#### Update data API\n- UpdateEntryData(u64 entryNumber, u32 entryType, u8[] newData)\n\n### CLIENT API\n- Create and start a datastream client (`StreamClient`) using the `NewClient` function followed by the `Start` function.\n- Executes server commands by calling `ExecCommandStart`, `ExecCommandStartBookmark`, `ExecCommandGetHeader`, `ExecCommandGetEntry`, `ExecCommandGetBookmark`, or `ExecCommandStop`.\n\n#### Streaming API\n- ExecCommandStart(fromEntry): Initiates the stream starting from the entry number specified in the parameter.\n- ExecCommandStartBookmark(fromBookmark): Initiates the stream starting from the entry pointed by the bookmark specified in the parameter.\n- ExecCommandStop(): Stops receiving stream.\n- SetProcessEntryFunc(f `ProcessEntryFunc`): Sets the callback function for each entry received. Overrides default function that just prints the entry fields.\n\n#### Query data API\n- ExecCommandGetHeader() -\u003e returns struct HeaderEntry: Fetches stream file header info and returns it.\n- ExecCommandGetEntry(fromEntry) -\u003e returns struct FileEntry: Fetches entry data from the specified entry number and returns it.\n- ExecCommandGetBookmark(fromBookmark) -\u003e returns struct FileEntry: Fetches entry data pointed by the specified bookmark and returns it.\n\n## DATASTREAM CLI DEMO APP\nBuild the binary datastream demo app (`dsapp`):\n```\nmake build-dsapp\n```\nRun the app without parameters to see the available commands:\n```\n./dsapp\n```\n```\nNAME:\n   dsapp - Run a datastream server/client/relay demo cli app\n\nUSAGE:\n   dsapp [global options] command [command options] [arguments...]\n\nCOMMANDS:\n   server   Run datastream server\n   client   Run datastream client\n   relay    Run datastream relay\n   help, h  Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --help, -h  show help\n```\n### SERVER\nUse the help option to check available parameters for the server command:\n```\n./dsapp help server\n```\n```\nNAME:\n   dsapp server - Run datastream server\n\nUSAGE:\n   dsapp server [command options] [arguments...]\n\nOPTIONS:\n   --port value   exposed port for clients to connect (default: 6900)\n   --file value   datastream data file name (*.bin) (default: datastream.bin)\n   --log value    log level (debug|info|warn|error) (default: info)\n   --sleep value  initial sleep and sleep between atomic operations in ms (default: 0)\n   --opers value  number of atomic operations (server will terminate after them) (default: 1000000)\n   --help, -h     show help\n```\nRun a datastream server with default parameters (port: `6900`, file: `datastream.bin`, log: `info`):\n```\n./dsapp server\n```\nOr run a datastream server with custom parameters:\n```\n./dsapp server --port 6969 --file seqstream.bin --log warn\n```\n### CLIENT\nUse the help option to check available parameters for the client command:\n```\n./dsapp help client\n```\n```\nNAME:\n   dsapp client - Run datastream client\n\nUSAGE:\n   dsapp client [command options] [arguments...]\n\nOPTIONS:\n   --server value        datastream server address to connect (IP:port) (default: 127.0.0.1:6900)\n   --from value          entry number to start the sync/streaming from (latest|0..N) (default: latest)\n   --frombookmark value  bookmark to start the sync/streaming from (0..N) (has preference over --from parameter)\n   --header              query file header information (default: false)\n   --entry value         entry number to query data (0..N)\n   --bookmark value      entry bookmark to query entry data pointed by it (0..N)\n   --log value           log level (debug|info|warn|error) (default: info)\n   --help, -h            show help\n```\nRun a datastream client with default parameters (server: `127.0.0.1:6900`, from: `latest`, log: `info`)\n```\n./dsapp client\n```\nOr run a datastream client with custom parameters:\n```\n./dsapp client --server 127.0.0.1:6969 --from 0 --log debug\n```\nOr just get the current stream header file information:\n```\n./dsapp client --server 127.0.0.1:6969 --header\n```\n### RELAY\nUse the help option to check available parameters for the relay command:\n```\n./dsapp help relay\n```\n```\nNAME:\n   dsapp relay - Run datastream relay\n\nUSAGE:\n   dsapp relay [command options] [arguments...]\n\nOPTIONS:\n   --server value  datastream server address to connect (IP:port) (default: 127.0.0.1:6900)\n   --port value    exposed port for clients to connect (default: 7900)\n   --file value    relay data file name (*.bin) (default: datarelay.bin)\n   --log value     log level (debug|info|warn|error) (default: info)\n   --help, -h      show help\n```\nRun a datastream relay with default parameters (server: `127.0.0.1:6900`, port: `7900`, file: `datarelay.bin`, log: `info`)\n```\n./dsapp relay\n```\n\n## USE CASE: zkEVM SEQUENCER ENTRIES\nSequencer data stream service to stream L2 blocks and L2 txs\n\nList of events (entry types):\n\n### BatchStart:\nuint64 number\nBatchType type\nuint64 fork_id\nuint64 chain_id\nDebug debug\n\n### BatchEnd:\nuint64 number\nbytes local_exit_root\nbytes state_root\nDebug debug\n\n### L2Block:\nuint64 number\nuint64 batch_number\nuint64 timestamp\nuint32 delta_timestamp\nuint64 min_timestamp\nbytes l1_blockhash\nuint32 l1_infotree_index\nbytes hash\nbytes state_root\nbytes global_exit_root\nbytes coinbase\nuint64 block_gas_limit\nbytes block_info_root\nDebug debug\n\n### L2BlockEnd:\nuint64 number\n\n### Transaction:\nuint64 l2block_number\nuint64 index\nbool is_valid\nbytes encoded\nuint32 effective_gas_price_percentage\nbytes im_state_root\nDebug debug\n\n### UpdateGER:\nuint64 batch_number\nuint64 timestamp\nbytes global_exit_root\nbytes coinbase\nuint64 fork_id\nuint64 chain_id\nbytes state_root\nDebug debug\n\n### BookMark:\nBookmarkType type\nuint64 value\n\n### Debug:\nstring message\n\n### BookmarkType:\nBOOKMARK_TYPE_UNSPECIFIED = 0;\nBOOKMARK_TYPE_BATCH = 1;\nBOOKMARK_TYPE_L2_BLOCK = 2;\n\n### EntryType:\nENTRY_TYPE_UNSPECIFIED = 0;\nENTRY_TYPE_BATCH_START = 1;\nENTRY_TYPE_L2_BLOCK = 2;\nENTRY_TYPE_TRANSACTION = 3;\nENTRY_TYPE_BATCH_END = 4;\nENTRY_TYPE_UPDATE_GER = 5;\nENTRY_TYPE_L2_BLOCK_END = 6;\n\n### BatchType:\nBATCH_TYPE_UNSPECIFIED = 0;\nBATCH_TYPE_REGULAR = 1;\nBATCH_TYPE_FORCED = 2;\nBATCH_TYPE_INJECTED = 3;\nBATCH_TYPE_INVALID = 4;","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0xpolygon%2Fzkevm-data-streamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F0xpolygon%2Fzkevm-data-streamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0xpolygon%2Fzkevm-data-streamer/lists"}