{"id":16216023,"url":"https://github.com/tangledbytes/use","last_synced_at":"2026-01-12T02:43:54.060Z","repository":{"id":62058195,"uuid":"555996417","full_name":"tangledbytes/use","owner":"tangledbytes","description":"USE - USE Storage Engine. Playing with storage engines.","archived":false,"fork":false,"pushed_at":"2023-01-29T14:37:18.000Z","size":129,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-13T09:53:51.339Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tangledbytes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-22T20:38:17.000Z","updated_at":"2023-01-10T15:18:33.000Z","dependencies_parsed_at":"2023-02-15T23:10:14.014Z","dependency_job_id":null,"html_url":"https://github.com/tangledbytes/use","commit_stats":null,"previous_names":["tangledbytes/use","utkarsh-pro/use"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tangledbytes%2Fuse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tangledbytes%2Fuse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tangledbytes%2Fuse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tangledbytes%2Fuse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tangledbytes","download_url":"https://codeload.github.com/tangledbytes/use/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243982177,"owners_count":20378604,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-10T11:17:47.642Z","updated_at":"2026-01-12T02:43:54.024Z","avatar_url":"https://github.com/tangledbytes.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# USE\n\nUSE is an insanely stupid storage engine. The sole purpose of the existence of this storage engine is so that I can play with multiple things in a single project. Plan for iteration 1 is to have:\n1. A stupid simple storage engine, which stores key value pairs in a single file. The data is stored in TLV format. Data file is nothing but an append only log. Writes are fast, reads are excrutiatingly slow. No plans to improve any of this in this iteration.\n2. A simple HTTP based transporation layer for the storage engine to expose its API. No TLS, no complex query language, nothing. Just a simple HTTP query based API.\n3. Single master replication. Nothing fancy here either. Follower nodes will use a join token to join the cluster. Once the token is verified, they can choose either asynchronous or synchronous mode of replication.\n\nThat's it for the first iteration. For the future iteration (if that ever happens):\n1. I might attempt to improve the core storage engine. Something based on the LSM tree. I also plan to have multiple implementation of the core storage engine, some based on simple SSTable, some on LSM Tree, some on B Tree, some on B+Tree, etc.\n2. I would want to have a decent enough query parser (not SQL for this). Maybe a compiled or JIT based.\n3. For replication, I don't know man. Multi-leader and leaderless are scary as f**k. I may not be able to muster enough courage to implement it in a learning project.\n4. Sharding? Don't know yet.\n\nWhta does USE stand for? It's a recursive name U -\u003e USE, S -\u003e Storage, E -\u003e Engine\n\n\n\n### Benchmarks\n**Set Operation - No Sync**\n```\ngoos: darwin\ngoarch: arm64\npkg: github.com/utkarsh-pro/use/pkg/storage/stupid\nBenchmarkStupidSetNoSync/128B-10         \t  510524\t      2940 ns/op\t  43.54 MB/s\t     528 B/op\t      14 allocs/op\nBenchmarkStupidSetNoSync/256B-10         \t  337562\t      3178 ns/op\t  80.56 MB/s\t     968 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/1K-10           \t  331254\t      3702 ns/op\t 276.60 MB/s\t    3560 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/2K-10           \t  277218\t      4347 ns/op\t 471.09 MB/s\t    7272 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/4K-10           \t  247633\t      4940 ns/op\t 829.10 MB/s\t   14440 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/8K-10           \t  189014\t      6698 ns/op\t1223.11 MB/s\t   28008 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/16K-10          \t  118286\t      9638 ns/op\t1699.89 MB/s\t   59496 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/32K-10          \t   81962\t     15055 ns/op\t2176.56 MB/s\t  114792 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/64K-10          \t   48633\t     25361 ns/op\t2584.15 MB/s\t  213097 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/128K-10         \t   26365\t     54351 ns/op\t2411.60 MB/s\t  409707 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/256K-10         \t    8853\t    128416 ns/op\t2041.36 MB/s\t  802924 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/512K-10         \t    6817\t    194201 ns/op\t2699.72 MB/s\t 1589360 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/1M-10           \t    3225\t    425407 ns/op\t2464.88 MB/s\t 3162225 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/2M-10           \t    1670\t    661528 ns/op\t3170.16 MB/s\t 6307950 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/4M-10           \t     760\t   1405073 ns/op\t2985.11 MB/s\t12599407 B/op\t      15 allocs/op\nBenchmarkStupidSetNoSync/8M-10           \t     434\t   2839935 ns/op\t2953.80 MB/s\t25182318 B/op\t      15 allocs/op\nPASS\nok  \tgithub.com/utkarsh-pro/use/pkg/storage/stupid\t27.126s\n```\n\n**Set Operation - Sync**\n```\ngoos: darwin\ngoarch: arm64\npkg: github.com/utkarsh-pro/use/pkg/storage/stupid\nBenchmarkStupidSetSync/128B-10         \t      55\t  18468191 ns/op\t   0.01 MB/s\t     528 B/op\t      14 allocs/op\nBenchmarkStupidSetSync/256B-10         \t      63\t  18528600 ns/op\t   0.01 MB/s\t     968 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/1K-10           \t      62\t  18497663 ns/op\t   0.06 MB/s\t    3560 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/2K-10           \t      63\t  19417550 ns/op\t   0.11 MB/s\t    7272 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/4K-10           \t      63\t  19855822 ns/op\t   0.21 MB/s\t   14440 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/8K-10           \t      63\t  18257138 ns/op\t   0.45 MB/s\t   28008 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/16K-10          \t      63\t  18745929 ns/op\t   0.87 MB/s\t   59496 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/32K-10          \t      64\t  18535575 ns/op\t   1.77 MB/s\t  114792 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/64K-10          \t      64\t  18512802 ns/op\t   3.54 MB/s\t  213096 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/128K-10         \t      62\t  19197916 ns/op\t   6.83 MB/s\t  409706 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/256K-10         \t      63\t  18500382 ns/op\t  14.17 MB/s\t  802956 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/512K-10         \t      58\t  18841219 ns/op\t  27.83 MB/s\t 1589357 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/1M-10           \t      58\t  19661541 ns/op\t  53.33 MB/s\t 3162267 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/2M-10           \t      58\t  19847116 ns/op\t 105.67 MB/s\t 6307953 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/4M-10           \t      56\t  22466036 ns/op\t 186.70 MB/s\t12599409 B/op\t      15 allocs/op\nBenchmarkStupidSetSync/8M-10           \t      48\t  24237001 ns/op\t 346.11 MB/s\t25182323 B/op\t      15 allocs/op\nPASS\nok  \tgithub.com/utkarsh-pro/use/pkg/storage/stupid\t19.562s\n```\n\n**Set Operation - Async Syncing**\n```\ngoos: darwin\ngoarch: arm64\npkg: github.com/utkarsh-pro/use/pkg/storage/stupid\nBenchmarkStupidSetAsyncSync/128B-10         \t     100\t  12918243 ns/op\t   0.01 MB/s\t     673 B/op\t      15 allocs/op\nBenchmarkStupidSetAsyncSync/256B-10         \t     104\t  12936466 ns/op\t   0.02 MB/s\t    1028 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/1K-10           \t      91\t  14281883 ns/op\t   0.07 MB/s\t    3626 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/2K-10           \t      79\t  15608401 ns/op\t   0.13 MB/s\t    7324 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/4K-10           \t      98\t  14038768 ns/op\t   0.29 MB/s\t   14489 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/8K-10           \t      90\t  14180032 ns/op\t   0.58 MB/s\t   28051 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/16K-10          \t      81\t  14509683 ns/op\t   1.13 MB/s\t   59523 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/32K-10          \t      68\t  16905240 ns/op\t   1.94 MB/s\t  114808 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/64K-10          \t      64\t  16283159 ns/op\t   4.02 MB/s\t  213112 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/128K-10         \t      67\t  16624864 ns/op\t   7.88 MB/s\t  409727 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/256K-10         \t      64\t  16924340 ns/op\t  15.49 MB/s\t  802960 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/512K-10         \t      73\t  17693342 ns/op\t  29.63 MB/s\t 1589403 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/1M-10           \t      64\t  19745871 ns/op\t  53.10 MB/s\t 3162248 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/2M-10           \t      70\t  20500542 ns/op\t 102.30 MB/s\t 6307972 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/4M-10           \t     118\t  10283815 ns/op\t 407.85 MB/s\t12599434 B/op\t      16 allocs/op\nBenchmarkStupidSetAsyncSync/8M-10           \t      68\t  16451892 ns/op\t 509.89 MB/s\t25182336 B/op\t      16 allocs/op\nPASS\nok  \tgithub.com/utkarsh-pro/use/pkg/storage/stupid\t27.076s\n```\n\n**Get Operation**\n```\ngoos: darwin\ngoarch: arm64\npkg: github.com/utkarsh-pro/use/pkg/storage/stupid\nBenchmarkStupidGet/128B-10         \t  248076\t      4941 ns/op\t  25.91 MB/s\t     360 B/op\t      20 allocs/op\nBenchmarkStupidGet/256B-10         \t  229862\t      4884 ns/op\t  52.42 MB/s\t     488 B/op\t      20 allocs/op\nBenchmarkStupidGet/1K-10           \t  222211\t      5003 ns/op\t 204.69 MB/s\t    1256 B/op\t      20 allocs/op\nBenchmarkStupidGet/2K-10           \t  218511\t      5123 ns/op\t 399.79 MB/s\t    2280 B/op\t      20 allocs/op\nBenchmarkStupidGet/4K-10           \t  211711\t      5297 ns/op\t 773.32 MB/s\t    4328 B/op\t      20 allocs/op\nBenchmarkStupidGet/8K-10           \t  192913\t      5811 ns/op\t1409.83 MB/s\t    8424 B/op\t      20 allocs/op\nBenchmarkStupidGet/16K-10          \t  169510\t      6750 ns/op\t2427.38 MB/s\t   16616 B/op\t      20 allocs/op\nBenchmarkStupidGet/32K-10          \t  135506\t      8380 ns/op\t3910.19 MB/s\t   33000 B/op\t      20 allocs/op\nBenchmarkStupidGet/64K-10          \t   99961\t     11797 ns/op\t5555.51 MB/s\t   65768 B/op\t      20 allocs/op\nBenchmarkStupidGet/128K-10         \t   66978\t     17901 ns/op\t7322.18 MB/s\t  131305 B/op\t      20 allocs/op\nBenchmarkStupidGet/256K-10         \t   39435\t     29523 ns/op\t8879.21 MB/s\t  262377 B/op\t      20 allocs/op\nBenchmarkStupidGet/512K-10         \t   21183\t     56949 ns/op\t9206.26 MB/s\t  524523 B/op\t      20 allocs/op\nBenchmarkStupidGet/1M-10           \t   10000\t    104689 ns/op\t10016.09 MB/s\t 1048813 B/op\t      20 allocs/op\nBenchmarkStupidGet/2M-10           \t    6278\t    187423 ns/op\t11189.42 MB/s\t 2097390 B/op\t      20 allocs/op\nBenchmarkStupidGet/4M-10           \t    2946\t    407927 ns/op\t10281.99 MB/s\t 4194539 B/op\t      20 allocs/op\nBenchmarkStupidGet/8M-10           \t    1182\t   1018820 ns/op\t8233.65 MB/s\t 8388843 B/op\t      20 allocs/op\nPASS\nok  \tgithub.com/utkarsh-pro/use/pkg/storage/stupid\t23.241s\n```\n\n## Updates\n### 23rd October 2022\n- Improved \"stupid\" storage type reading. Now supports parallel reads without locking.\n- TLV reader and writer decoupled for ergonomics reason.\n- Add HTTP transport layer.\n- Add physical snapshot API without locking writes.\n- Add IDs to the DB writes.\n\n### 25th October 2022\n- Add custom logger.\n- Add support for data corruption detection and auto fixing during the startup.\n- Add `ForEach` method to minimize code duplication and better control over the iteration.\n\n### 4th November 2022\n- Add bitset data structure.\n\n### 12th November 2022\n- Add a standard bloom filter implementation.\n- Add DIBF implementation.\n\n### 17th November 2022\n- Integrated DIBF into stupid storage engine. No filter regeneration however.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftangledbytes%2Fuse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftangledbytes%2Fuse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftangledbytes%2Fuse/lists"}