{"id":30677268,"url":"https://github.com/sitano/mdbutil","last_synced_at":"2025-10-23T19:56:20.103Z","repository":{"id":303780898,"uuid":"1016631767","full_name":"sitano/mdbutil","owner":"sitano","description":"MariaDB Experimental tools in Rust. Parse Redo Log or create a fake checkpoint.","archived":false,"fork":false,"pushed_at":"2025-08-29T18:17:57.000Z","size":205,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-29T20:51:12.119Z","etag":null,"topics":["database","mariadb","redo","rust","storage"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sitano.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-09T09:45:01.000Z","updated_at":"2025-08-29T18:18:00.000Z","dependencies_parsed_at":"2025-07-09T11:53:14.794Z","dependency_job_id":"b100cc49-1b9a-4cbc-bddc-0931cf1966a1","html_url":"https://github.com/sitano/mdbutil","commit_stats":null,"previous_names":["sitano/mdbutil"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sitano/mdbutil","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sitano%2Fmdbutil","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sitano%2Fmdbutil/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sitano%2Fmdbutil/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sitano%2Fmdbutil/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sitano","download_url":"https://codeload.github.com/sitano/mdbutil/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sitano%2Fmdbutil/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273114756,"owners_count":25048253,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","mariadb","redo","rust","storage"],"created_at":"2025-09-01T11:36:05.363Z","updated_at":"2025-10-23T19:56:20.097Z","avatar_url":"https://github.com/sitano.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"MariaDB experimental utilities for testing and development purposes.\n===\n\nRead page\n---\n\n```\n$ cargo run read-page --file-path ./undo002 --page 10\nOpened tablespace file: ./undo002 with size: 17825792 bytes, page size: 16384 bytes, num pages: 1088, flags: FULL_CRC32|PAGE_SSIZE=5|POST_ANTELOPE|RAW=0x00000015\nTablespace(space_id=2, flags=0x15, page_size=16384, order=0)\nPageBuf { space_id: 2, page_no: 10, prev_page: None, next_page: None, page_lsn: 11787325, page_type: Sys, checksum: 1559159512 }\ntrx_rseg_t {\n    format: 0,\n    history_size: 0,\n    history: flst_base_node_t { len: 0 },\n    fseg_header: fseg_header_t { space: 2, page_no: 2, offset: 1394 },\n    undo_slots: [],\n    max_trx_id: 70,\n    mysql_log: None,\n    wsrep_xid: None,\n}\n\n$ cargo run read-page --file-path ./undo002 --page 50\nOpened tablespace file: ./undo002 with size: 17825792 bytes, page size: 16384 bytes, num pages: 1088, flags: FULL_CRC32|PAGE_SSIZE=5|POST_ANTELOPE|RAW=0x00000015\nTablespace(space_id=2, flags=0x15, page_size=16384, order=0)\nPageBuf { space_id: 2, page_no: 50, prev_page: None, next_page: None, page_lsn: 181119183, page_type: UndoLog, checksum: 4168205373 }\ntrx_undo_page_t {\n    page_type: 0,\n    start: 56,\n    free: 15866,\n    node: flst_node_t { prev: fil_addr_t { page: 49, boffset: 44 }, next: fil_addr_t { page: 51, boffset: 44 } },\n}\n\n$ cargo run read-page --file-path ./undo002 --page 50 --hex | head\n00000000: 00 00 00 00 00 00 00 32 ff ff ff ff ff ff ff ff |.......2........|\n00000010: 00 00 00 00 0a cb a8 cf 00 02 00 00 00 00 00 00 |................|\n00000020: 00 00 00 00 00 02 00 00 00 38 3d fa 00 00 00 31 |.........8=....1|\n00000030: 00 2c 00 00 00 33 00 2c 04 56 1c 4b 1b 00 00 00 |.,...3.,.V.K....|\n00000040: 00 00 00 e0 80 00 00 00 00 00 00 04 80 00 00 97 |................|\n00000050: 01 03 84 00 41 41 41 41 41 41 41 41 41 41 41 41 |....AAAAAAAAAAAA| \u003c-- user's data A*\n00000060: 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 |AAAAAAAAAAAAAAAA|\n00000070: 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 |AAAAAAAAAAAAAAAA|\n00000080: 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 |AAAAAAAAAAAAAAAA|\n00000090: 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 |AAAAAAAAAAAAAAAA|\n```\n\nRead tablespace\n---\n\nOn Undo Log structure [link](https://sitano.github.io/mariadb/innodb/undolog/recovery/2025/08/08/notes-on-mariadb-undo-log/).\n\n\n```\n$ cargo run read-tablespace --file-path ./ibdata1 --undo-log-dir ./\nOpened tablespace file: ./ibdata1 with size: 12582912 bytes, page size: 16384 bytes, num pages: 768, flags: FULL_CRC32|PAGE_SSIZE=5|POST_ANTELOPE|RAW=0x00000015\nTablespace(space_id=0, flags=0x15, page_size=16384, order=0)\nPageBuf { space_id: 0, page_no: 0, prev_page: None, next_page: None, page_lsn: 44158, page_type: FspHdr, checksum: 1379061894 }\nFSP header: fsp_header_t {\n    space_id: 0,\n    not_used: 0,\n    space_pages: 768,\n    free_limit: 320,\n    flags: 21,\n    free_frag_pages: 50,\n    free_extens: flst_base_node_t { len: 2, first: fil_addr_t { page: 0, boffset: 278 }, last: fil_addr_t { page: 0, boffset: 318 } },\n    free_frag: flst_base_node_t { len: 1, first: fil_addr_t { page: 0, boffset: 158 }, last: fil_addr_t { page: 0, boffset: 158 } },\n    full_frag: flst_base_node_t { len: 0 },\n    seg_id: 26,\n    seg_inodes_full: flst_base_node_t { len: 0 },\n    seg_inodes_free: flst_base_node_t { len: 1, first: fil_addr_t { page: 2, boffset: 38 }, last: fil_addr_t { page: 2, boffset: 38 } },\n}\nPageBuf { space_id: 0, page_no: 5, prev_page: None, next_page: None, page_lsn: 40054, page_type: TrxSys, checksum: 162806086 }\nTRX_SYS header: trx_sys_t {\n    id_store: 0,\n    fseg_header: fseg_header_t {\n        space: 0,\n        page_no: 2,\n        offset: 242,\n    },\n    rsegs: [\n        (space_id: 0, page_no: 6),\n        (space_id: 1, page_no: 3),\n        (space_id: 2, page_no: 3),\n        (space_id: 3, page_no: 3),\n        ...\n        (space_id: 1, page_no: 44),\n        (space_id: 2, page_no: 44),\n        (space_id: 3, page_no: 44),\n    ],\n    wsrep_xid: None,\n    mysql_log: None,\n    doublewrite: trx_sys_doublewrite_t {\n        fseg: fseg_header_t {\n            space: 0,\n            page_no: 2,\n            offset: 2738,\n        },\n        magic: 536853855,\n        block1: 64,\n        block2: 128,\n        magic_repeat: 536853855,\n        block1_repeat: 64,\n        block2_repeat: 128,\n    },\n}\nRSEG page: PageBuf { space_id: 0, page_no: 6, prev_page: None, next_page: None, page_lsn: 269027095, page_type: Sys, checksum: 3066088388 }\nRSEG page: PageBuf { space_id: 1, page_no: 3, prev_page: None, next_page: None, page_lsn: 66000, page_type: Sys, checksum: 2212095732 }\ntrx_rseg_t { max_trx_id: 12 }\n...\nRSEG page: PageBuf { space_id: 1, page_no: 6, prev_page: None, next_page: None, page_lsn: 76648, page_type: Sys, checksum: 2692835315 }\ntrx_rseg_t {\n    format: 0,\n    history_size: 0,\n    history: flst_base_node_t { len: 0 },\n    fseg_header: fseg_header_t { space: 1, page_no: 2, offset: 626 },\n    undo_slots: [],\n    max_trx_id: 44,\n    mysql_log: Some(\n        mysql_log_t {\n            log_offset: 7441,\n            log_name: \"/.../binlog.000001\",\n        },\n    ),\n    wsrep_xid: None,\n}\n...\n```\n\nRedo log parser for 11.8.x:\n---\n\nOn Redo Log structure [link](https://sitano.github.io/mariadb/innodb/redolog/recovery/2025/07/07/notes-on-mariadb-redo-log/).\n\n```\n$ scripts/mariadb-install-db --datadir ./data --innodb-log-file-size=10M\n$ bin/mariadbd --datadir ./data --innodb_fast_shutdown=0 --innodb-log-file-size=10M\n\n$ mycli -S /tmp/mysql.sock\n\u003e CREATE TABLE a (id int not null auto_increment primary key, t TEXT);\n\u003e SET max_recursive_iterations = 1000000;\n\u003e INSERT INTO a (t)\n  WITH RECURSIVE fill(n) AS (\n    SELECT 1 UNION ALL SELECT n + 1 FROM fill WHERE n \u003c 60500\n  )\n  SELECT RPAD(CONCAT(FLOOR(RAND()*1000000)), 64, 'x') FROM fill;\n$ pkill mariadbd\n$ cargo run -- read-redo --log-file-path data/ib_logfile0\n\nHeader block: 12288\nSize: 100663296, Capacity: 100651008\nRedoHeader {\n    version: 1349024115,\n    first_lsn: 12288,\n    creator: \"WorkatoDB Controller\",\n    crc: 509551572,\n}\nRedoCheckpointCoordinate {\n    checkpoints: [\n        RedoHeaderCheckpoint {\n            checkpoint_lsn: 84875,\n            end_lsn: 84875,\n            checksum: 2243572435,\n        },\n        RedoHeaderCheckpoint {\n            checkpoint_lsn: 84793,\n            end_lsn: 84793,\n            checksum: 3358478536,\n        },\n    ],\n    checkpoint_lsn: Some(\n        84875,\n    ),\n    checkpoint_no: Some(\n        1,\n    ),\n    end_lsn: 84875,\n    encrypted: false,\n    version: 1349024115,\n    start_after_restore: false,\n}\n1: MTR Chain count=1, len=16, lsn=84875\n  1: [84875..84886) Mtr { space_id: 0, page_no: 0, op: FileCheckpoint } at (84875+11)\nCheckpoint LSN/1: RedoHeaderCheckpoint { checkpoint_lsn: 84875, end_lsn: 84875, checksum: 2243572435 }\nCheckpoint LSN/2: RedoHeaderCheckpoint { checkpoint_lsn: 84793, end_lsn: 84793, checksum: 3358478536 }\nFile checkpoint chain: Some(MtrChain { lsn: 84875, len: 16, marker: 1, checksum: 3078137627, mtr: [Mtr { lsn: 84875, len: 11, space_id: 0, page_no: 0, op: FileCheckpoint, file_checkpoint_lsn: Some(84875) }] })\nFile checkpoint LSN: 84875\n```\n\nor without the checkpoint:\n\n```\n$ scripts/mariadb-install-db --datadir ./data --innodb-log-file-size=10M\n$ bin/mariadbd --datadir ./data --innodb_fast_shutdown=0 --innodb-log-file-size=10M\n\n$ mycli -S /tmp/mysql.sock\n\u003e CREATE TABLE a (id int not null auto_increment primary key, t TEXT);\n\u003e SET max_recursive_iterations = 1000000;\n\u003e INSERT INTO a (t)\n  WITH RECURSIVE fill(n) AS (\n    SELECT 1 UNION ALL SELECT n + 1 FROM fill WHERE n \u003c 60500\n  )\n  SELECT RPAD(CONCAT(FLOOR(RAND()*1000000)), 64, 'x') FROM fill;\n$ pkill -9 mariadbd\n$ cargo run -- read-redo --log-file-path data/ib_logfile0\n\nHeader block: 12288\nSize: 100663296, Capacity: 100651008\nRedoHeader {\n    version: 1349024115,\n    first_lsn: 12288,\n    creator: \"WorkatoDB Controller\",\n    crc: 509551572,\n}\nRedoCheckpointCoordinate {\n    checkpoints: [\n        RedoHeaderCheckpoint {\n            checkpoint_lsn: 83365,\n            end_lsn: 83365,\n            checksum: 694933498,\n        },\n        RedoHeaderCheckpoint {\n            checkpoint_lsn: 83267,\n            end_lsn: 83267,\n            checksum: 3695364396,\n        },\n    ],\n    checkpoint_lsn: Some(\n        83365,\n    ),\n    checkpoint_no: Some(\n        1,\n    ),\n    end_lsn: 83365,\n    encrypted: false,\n    version: 1349024115,\n    start_after_restore: false,\n}\n1: MTR Chain count=1, len=16, lsn=83365\n  1: [83365..83376) Mtr { space_id: 0, page_no: 0, op: FileCheckpoint } at (83365+11)\n2: MTR Chain count=37, len=289, lsn=83381\n  1: [83381..83386) Mtr { space_id: 3, page_no: 0, op: Write } at (83381+5)\n  2: [83386..83392) Mtr { space_id: 3, page_no: 2, op: Write } at (83386+6)\n  3: [83392..83396) Mtr { space_id: 3, page_no: 2, op: Memset } at (83392+4)\n  4: [83396..83400) Mtr { space_id: 3, page_no: 2, op: Memmove } at (83396+4)\n...\n21: MTR Chain count=13, len=89, lsn=84704\n  1: [84704..84714) Mtr { space_id: 3, page_no: 4, op: Write } at (84704+10)\n...\n  13: [84780..84788) Mtr { space_id: 3, page_no: 0, op: Option } at (84780+8)\nCheckpoint LSN/1: RedoHeaderCheckpoint { checkpoint_lsn: 83365, end_lsn: 83365, checksum: 694933498 }\nCheckpoint LSN/2: RedoHeaderCheckpoint { checkpoint_lsn: 83267, end_lsn: 83267, checksum: 3695364396 }\nFile checkpoint chain: Some(MtrChain { lsn: 83365, len: 16, marker: 1, checksum: 3479569512, mtr: [Mtr { lsn: 83365, len: 11, space_id: 0, page_no: 0, op: FileCheckpoint, file_checkpoint_lsn: Some(83365) }] })\nFile checkpoint LSN: 83365\n\n```\n\nto craft fake redo log file checkpoint use `write`. MariaDB ensures that:\n\n- file checkpoint is the latest entry in redo log by comparing its LSN to the\n  end of the file LSN (redo log LSN).\n- file checkpoint LSN is not less than the pages LSN in the tablespaces.\n\n```\n$ cargo run -- write-redo --log-file-path data/ib_logfile0 --size 100663296 --lsn 8336\n\nWriting file checkpoint: [fa, 0, 0, 0, 0, 0, 0, 0, 1, 45, a6, 1, dc, 36, f7, 9c, 0] at pos: 83366 (0x145a6)\nTarget header block: 12288\nSize: 100663296, Capacity: 0x5ffd000\nRedoHeader {\n    version: 1349024115,\n    first_lsn: 12288,\n    creator: \"test_creator\",\n    crc: 2774233419,\n}\nRedoCheckpointCoordinate {\n    checkpoints: [\n        RedoHeaderCheckpoint {\n            checkpoint_lsn: 83366,\n            end_lsn: 83366,\n            checksum: 3290552678,\n        },\n        RedoHeaderCheckpoint {\n            checkpoint_lsn: 83366,\n            end_lsn: 83366,\n            checksum: 3290552678,\n        },\n    ],\n    checkpoint_lsn: Some(\n        83366,\n    ),\n    checkpoint_no: Some(\n        0,\n    ),\n    end_lsn: 83366,\n    encrypted: false,\n    version: 1349024115,\n    start_after_restore: false,\n}\n  [83366..83377) Mtr { space_id: 0, page_no: 0, op: FileCheckpoint } at (83366+11)\nTarget checkpoint LSN/1: RedoHeaderCheckpoint { checkpoint_lsn: 83366, end_lsn: 83366, checksum: 3290552678 }\nTarget checkpoint LSN/2: RedoHeaderCheckpoint { checkpoint_lsn: 83366, end_lsn: 83366, checksum: 3290552678 }\nTarget file checkpoint LSN: 83366\n\n# and now we can start mariadbd with the new redo log file\n\n$ mariadbd --datadir ./data --innodb_fast_shutdown=0\n\n2025-07-22 17:51:49 0 [Warning] Setting lower_case_table_names=2 because file system for ./data/ is case insensitive\n2025-07-22 17:51:49 0 [Note] Starting MariaDB 11.6.2-MariaDB-debug source revision d8dad8c3b54cd09fefce7bc3b9749f427eed9709 server_uid jrmwW5r3Tn164Vhvku7bB+z6nV4= as process 13591\n2025-07-22 17:51:49 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!\n2025-07-22 17:51:49 0 [Note] InnoDB: Compressed tables use zlib 1.3.1\n2025-07-22 17:51:49 0 [Note] InnoDB: Number of transaction pools: 1\n2025-07-22 17:51:49 0 [Note] InnoDB: Using generic crc32 instructions\n2025-07-22 17:51:49 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB\n2025-07-22 17:51:49 0 [Note] InnoDB: Completed initialization of buffer pool\n2025-07-22 17:51:49 0 [Note] InnoDB: End of log at LSN=83366\n2025-07-22 17:51:49 0 [Note] InnoDB: Opened 3 undo tablespaces\n2025-07-22 17:51:49 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.\n2025-07-22 17:51:49 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...\n2025-07-22 17:51:49 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.\n2025-07-22 17:51:49 0 [Note] InnoDB: log sequence number 83366+16; transaction id 26\n2025-07-22 17:51:49 0 [Note] InnoDB: Loading buffer pool(s) from ./data/ib_buffer_pool\n2025-07-22 17:51:49 0 [Note] Plugin 'FEEDBACK' is disabled.\n2025-07-22 17:51:49 0 [Note] Plugin 'wsrep-provider' is disabled.\n2025-07-22 17:51:49 0 [Note] InnoDB: Buffer pool(s) load completed at 250722 17:51:49\n2025-07-22 17:51:50 0 [Note] Server socket created on IP: '::'.\n2025-07-22 17:51:50 0 [Note] Server socket created on IP: '0.0.0.0'.\n2025-07-22 17:51:50 0 [Note] mariadbd: Event Scheduler: Loaded 0 events\n2025-07-22 17:51:50 0 [Note] mariadbd: ready for connections\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsitano%2Fmdbutil","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsitano%2Fmdbutil","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsitano%2Fmdbutil/lists"}