{"id":13316280,"url":"https://github.com/Shelnutt2/crunch","last_synced_at":"2025-03-10T22:32:57.560Z","repository":{"id":137741528,"uuid":"96634989","full_name":"Shelnutt2/crunch","owner":"Shelnutt2","description":"MariaDB storage engine based on cap'n proto storage","archived":false,"fork":false,"pushed_at":"2018-04-17T11:07:09.000Z","size":442,"stargazers_count":3,"open_issues_count":20,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-12T10:20:40.701Z","etag":null,"topics":["capnp","capnproto","mariadb","mariadb-storage-engine","mysql"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Shelnutt2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-08T17:50:30.000Z","updated_at":"2022-10-27T21:12:00.000Z","dependencies_parsed_at":null,"dependency_job_id":"3261b9df-e555-4fa9-9340-56d434a91d1b","html_url":"https://github.com/Shelnutt2/crunch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shelnutt2%2Fcrunch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shelnutt2%2Fcrunch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shelnutt2%2Fcrunch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shelnutt2%2Fcrunch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Shelnutt2","download_url":"https://codeload.github.com/Shelnutt2/crunch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242939420,"owners_count":20209874,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["capnp","capnproto","mariadb","mariadb-storage-engine","mysql"],"created_at":"2024-07-29T18:21:31.377Z","updated_at":"2025-03-10T22:32:57.545Z","avatar_url":"https://github.com/Shelnutt2.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Crunch\n\n[![Build Status](https://travis-ci.org/Shelnutt2/crunch.svg?branch=master)](https://travis-ci.org/Shelnutt2/crunch)\n\nMariaDB storage engine based on cap'n proto storage\n\n## Requirements\n\nRequires MariaDB 10.2 or newer. It is untested on older versions.\n\n## Installation\n\nThere are two methods to building crunch.\n\n### Inside MariaDB Source Tree (Recommended)\nThe first is inside a MariaDB source tree (recommended).\n\n```bash\ngit clone git@github.com:MariaDB/server.git\ncd server\ngit submodule add git@github.com:Shelnutt2/crunch.git storage/crunch\nmkdir build \u0026\u0026 cd build\ncmake ..\nmake -j4\n```\n\n### Standalone compilation\n\nThe second method is building it standalone and\ninstalling the library to the mariadb plugin folder.\nThis method is not recommended, as the compiler and\ncompiler flags must match for mariadb to load the plugin.\n\n```bash\ngit clone git@github.com:Shelnutt2/crunch.git\ncd crunch\nmkdir build \u0026\u0026 cd build\ncmake .. -DCRUNCH_COMPILE_STANDALONE=ON\nmake -j4\n```\n\n## Features\n\n### Inplace Alter Table\n\nIn place alter tables are supported for column renames, column additions and\nsome column data type changes.\nAdding of columns is supported at any position (LAST or AFTER supported).\n\nWhen altering tables, the on disk data format is not changed. A new schema is\nproduced and when reading data it is converted on read to the new schema.\n\nIf a default value is set for a column and the column is also set to null\nthen the default value is not used, but the column will return null for\nexisting data. This is a limitation that will be addressed in the future.\n\nFor column data type changes, if the underlying type is similar, int16-\u003eint32\nor float64 -\u003e float32, the change can be made online. There is no protection\nagainst decreased percision or overflow. Changing from a int64 to a int16\nis allowed, but could result in data loss if any values are larger than an\nint16. This is how mariadb normally works, so we've allowed it with online\nalters.\n\n## Architecture\n\n### On Disk Format\n\nThe ondisk format is based on cap'n proto. Each table gets a cap'n proto schema\nfile which represent a row in the table.\nSee [examples/t1.capnp](examples/t1.capnp) for a sample cap'n proto schema\nrepresenting the following table:\n\n```sql\nCREATE TABLE t1 (\n  column1 integer,\n  column2 varchar(64)\n) ENGINE=crunch;\n```\n\n### On Disk File Hierarchy\n\nBelow is a hierarchy of the ondisk structure,\nassuming the database is test and table is t1.\n```\nmysql_datadir\n└── test\n    ├── t1\n    │   ├── 1515252170775990244-e8e7a69e-923d-471f-b0ca-8e54568a3fef.capnpd\n    │   ├── t1.capnp\n    │   ├── t1.capnpd\n    │   ├── t1.deleted.capnpd\n    │   └── transactions\n    │       ├── 1515251992233237213-b0929f33-a67c-4d7e-83e1-594b983ce299.capnpd\n    │       └── 1515251992233237213-b0929f33-a67c-4d7e-83e1-594b983ce299.deleted.capnpd\n    └── t1.frm\n```\n\n### Transactions\n\nTransactions are supported and handled with the crunchTxn class.\nEach transaction will perform it's operations in a \"transactions\" folder,\nwith dedicated files per table/transactions.\nThis allows for ondisk isolation for of transactions before commit.\nIt also removes the immediate need for an undo log.\n\nA rollback simply deletes the transaction files and its done.\nWhen a commit is made, the transaction files are renamed and moved to the\nmain table folder. The rename function call is atomic according to the\n[ISO c standard](http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html)\nassuming the transaction folder lies on the same filesystem as the main\ntable folder.\n\nThe transaction folder is a subdirectory in the table directory.\nThe transaction files are in the format of\n**unix_timestamp_nanoseconds**-**uuid**.ext\n\nThe use of unix timestamp is to order the files on disk so full table scans\ncan sequentially read all files and keep the ordering the same as original\ninserts.\n\nA consequence of using independent files for each transaction is that the\nnumber of data files will grow without bounds. During a full table scan\neach data file must be opened and mmaped one at a time. Performance\ndegrades in a linear manor. To combat this a consolidation of datafiles\nhas been implemented. Currently this only acts on table close.\nThe long term goal is to implement a daemon plugin which will implement\nconsolidation in the background independent of the mysql server. See #44.\n\nConsolidation can also be manually forced by running `OPTIMIZE TABLE tbl_name`\n\n## System and Table Variables\n\nBelow are a list of server and table configuration variables.\n\n| Variable Bame | description | Data Type | Default Value | Range | System Variable | Table Variable |\n| ------------- | ----------- | --------- | ------------- | ----- | --------------- | -------------- |\n| consolidation_threshold | Threshold for number of data files to start consolidation | Integer | 100 | 0 to MAX_INT | Yes | Yes |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FShelnutt2%2Fcrunch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FShelnutt2%2Fcrunch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FShelnutt2%2Fcrunch/lists"}