{"id":18547723,"url":"https://github.com/johnramsden/znsccache","last_synced_at":"2025-05-15T07:34:28.795Z","repository":{"id":223992951,"uuid":"760239664","full_name":"johnramsden/znsccache","owner":"johnramsden","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-28T05:39:28.000Z","size":195,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-14T15:06:44.306Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/johnramsden.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-20T03:29:27.000Z","updated_at":"2024-04-28T05:39:31.000Z","dependencies_parsed_at":"2024-04-21T03:01:29.884Z","dependency_job_id":"b3a7ba22-6661-40cb-9002-b3e8e7c55484","html_url":"https://github.com/johnramsden/znsccache","commit_stats":null,"previous_names":["johnramsden/znsccache"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnramsden%2Fznsccache","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnramsden%2Fznsccache/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnramsden%2Fznsccache/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/johnramsden%2Fznsccache/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/johnramsden","download_url":"https://codeload.github.com/johnramsden/znsccache/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254296692,"owners_count":22047332,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T20:30:53.496Z","updated_at":"2025-05-15T07:34:28.764Z","avatar_url":"https://github.com/johnramsden.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"## libzbd\n\n```\ncd /data/john\nmkdir -p libs\n\ngit clone https://github.com/westerndigitalcorporation/libzbd.git\ncd libzbd\nsh ./autogen.sh\n./configure --prefix=/data/john/libs\nmake\nmake install\n```\n\n## libgcrypt\n\n```\napt install libgcrypt20-dev\n```\n\n## uuid\n\n```\napt install uuid-dev\n```\n\n## Compiling\n\n```\nexport LD_LIBRARY_PATH=/data/john/libs/lib\nmake clean \u0026\u0026 make\n```\n\n## Run\n\n```\nsudo LD_LIBRARY_PATH=/data/john/libs/lib ./znsccache -d /dev/nvme0n2 -s 4096 -c ../credentials.json\nsudo LD_LIBRARY_PATH=/data/john/libs/lib ./znsccache -d /dev/nvme0n2 -s 32768 -c ../credentials.json -f /home/john/ssd/zns/disk\\=33554432KB\\,obj\\=32KB\\,uuids\\=1048576iter\\=2097152.txt | tee /data/john/zns.log\n```\n\n## GDB\n\n```\nsudo LD_LIBRARY_PATH=/data/john/libs/lib gdb --args ./znsccache -d /dev/nvme0n2 -s 4096 -c ../credentials.json\n```\n\n## GDB\n\n```\nsudo valgrind env LD_LIBRARY_PATH=/data/john/libs/lib ./znsccache -d /dev/nvme0n2 -s 131072 -c ../credentials.json -f /home/john/ssd/zns/disk=268435456KB,obj=32KB,uuids=16777216iter=33554432.txt\n```\n\n## Tidy\n\n```\nclang-format -i src/*.c include/*.h\n```\n\n## AWS libs3\n\n```\napt install libs3-dev\n```\n\n## cJSON\n\n```\napt install libcjson-dev\n```\n\n## Data\n\n### znscache.small\n\n```\nUploading objects of size 268435456 random bytes to znscache.small\n6c228e61-abb7-4100-a5df-417f25b47c36, 's'\ncf752bc3-b33f-435f-ac30-178e971f54a2, 't'\n7c968f5a-0c97-4cd8-960b-1c1bf6ab67ee, 'u'\nc3f16f76-8f94-40f1-950d-695bb226fe43, 'v'\n54b4afb0-7624-448d-9c52-ddcc56e56bf2, 'w'\n3f5d58a4-1780-46b5-bd96-d4c5b4f55b2b, 'x'\ne5a19178-6b77-40b7-9bf8-f45f70ea19a5, 'y'\n13f9c273-3f2f-486a-8003-ba2fba58b258, 'z'\nc75b2775-f71e-47f9-9500-530c8cf9f344, 'a'\ne5be69c3-6e22-4c2c-aa09-0bff9417c361, 'b'\n982aff04-b7b6-4060-a141-241af8c46375, 'c'\n62c0ae68-e637-4ddc-a0aa-aa1151f88595, 'd'\n7fc3ca1b-3968-4274-9a5b-688f01dca81b, 'e'\nb9cece7d-50ce-4f00-9abb-4e9c07ed68fd, 'f'\n7446cbb0-430b-44c4-9da9-443a044c14d1, 'g'\n8c0630ac-c4b5-4bf1-a7a3-13a7e95032dd, 'h'\nFinished uploading. Total uploaded size: 4294967296 bytes.\n```\n\n## Benchmarks\n\nSet:\n\n* xf\n* zones\n* file\n* count\n* -s bytes\n* -m output\n\n```\ntimestamp,type,time (ms)\n```\n\nRecord:\n* hit rate vs time\n* evictions vs time\n* total get latency vs time\n* Read disk latency vs time\n* Write disk latency vs time\n\nBench:\n* 32K obj   - chunk sz 16K, 32K   - evict, no evict - S3 geomean 32K\n\u003c!-- * 512K obj  - chunk sz 256K, 512K - evict, no evict - S3 geomean 512K --\u003e\n* 1M obj    - chunk sz 512K, 1M   - evict, no evict - S3 geomean 1M\n\u003c!-- * 256M obj  - chunk sz 128M 256M  - evict, no evict - S3 geomean 256M --\u003e\n* 1G obj    - chunk sz 512M 1G    - evict, no evict - S3 geomean 1G\n\n* 32K obj   - chunk sz 16K, 32K   - evict, no evict - S3 geomean 32K\n\nUsing 57ms for bench\n\n```c\n#define EVICT_THRESH_ZONES_LEFT 1\n#define EVICT_THRESH_ZONES_REMOVE 2\n```\n\nEvict\n\n'disk=10485760KB,obj=32KB,uuids=26214400iter=2621440.txt'\n\nNo Evict:\n\n'disk=41943040KB,obj=1024KB,uuids=40960iter=327680.txt'\n\n```\nmake clean \u0026\u0026 DEBUG=0 EMULATE_S3=1 S3_DELAY=57 make\n```\n\n144.738501,\n98.838001,\n107.908126,\n98.918494,\n118.180804,\n116.020368,\n60.175368,\n37.280651,\n58.458616,\n35.930335,\n42.303642,\n31.601407,\n36.354112,\n30.871992,\n100.580836,\n53.483931,\n36.369907,\n34.819342,\n32.351520,\n31.446523,\n36.295667,\n41.905635,\n33.363852,\n29.932280,\n36.685492,\n34.027327,\n33.672402,\n33.987916,\n38.897166,\n34.277723,\n34.772344,\n36.018225,\n35.033560,\n32.214941,\n36.892214,\n33.702793,\n34.758242,\n35.443970,\n67.553659,\n36.191228,\n36.559560,\n38.053496,\n34.650326,\n31.218108,\n31.639598,\n38.053472,\n39.905578,\n37.811444,\n33.288950,\n32.699743,\n34.919856,\n34.030839,\n33.912846,\n32.659927,\n34.068531,\n37.473022,\n32.584065,\n34.812547,\n32.297873,\n30.739199,\n37.814750,\n58.374286,\n44.341046,\n30.071477,\n37.168953,\n38.416765,\n41.087754,\n33.353333,\n29.139559,\n40.512590,\n34.713584,\n36.020293,\n33.374929,\n32.113547,\n38.850767,\n37.482747,\n30.991440,\n38.285367,\n29.306384,\n59.654151,\n36.210850,\n35.197976,\n32.719741,\n35.170658,\n31.322904,\n35.549491,\n30.584413,\n34.622259,\n37.709964,\n34.463742,\n34.518399,\n34.468402,\n36.018905,\n33.288947,\n36.133134,\n34.596191,\n33.574634,\n37.633029,\n40.193240,\n31.660826,\n\ngeomean=39.08033715816217\nstdev=20.75347930627147\navg=66.007688\n\n\u003c!-- * 512K obj  - chunk sz 256K, 512K - evict, no evict - S3 geomean 512K\n\nUsing 198ms for bench\n\n```\nmake clean \u0026\u0026 DEBUG=0 EMULATE_S3=1 S3_DELAY=198 make\n```\n\nEvict\n\ndisk=16GB\nobj=512KB\nuuids=2621440\niter=262144\n\nNo Evict:\n\ndisk=16GB\nobj=512KB\nuuids=32768\niter=262144\n\n\n0=299.297865\n1=190.926778\n2=228.409959\n3=235.568989\n4=210.793632\n5=131.004768\n6=112.267369\n7=140.005505\n8=166.498834\n9=200.929175\n10=221.518356\n11=222.765190\n12=306.344046\n13=224.442063\n14=185.046569\nstdev=52.94\ngeomean=197.93\navg=205.054607 --\u003e\n\n* 1M obj    - chunk sz 512K, 1M   - evict, no evict - S3 geomean 1M\n\nEvict:\n\n'disk=41943040KB,obj=1024KB,uuids=3276800iter=327680.txt'\n\nNo Evict:\n\n'disk=41943040KB,obj=1024KB,uuids=40960iter=327680.txt'\n\nUsing 123ms for bench\n\n```c\n#define EVICT_THRESH_ZONES_LEFT 1\n#define EVICT_THRESH_ZONES_REMOVE 8\n#define ZONES_USED 40\n```\n\n```shell\nmake clean \u0026\u0026 DEBUG=0 EMULATE_S3=1 S3_DELAY=123 make\ns=1048576\nfor f in 'disk=41943040KB,obj=1024KB,uuids=3276800iter=327680' 'disk=41943040KB,obj=1024KB,uuids=40960iter=327680'; do time sudo LD_LIBRARY_PATH=/data/john/libs/lib ./znsccache -d /dev/nvme0n2 -s $s -c ../credentials.json -f ../\"$f\".txt -m /data/john/metric/\"${f}\".csv; done\n```\n\n379.407507,\n324.466045,\n304.762546,\n261.600677,\n239.067244,\n197.643438,\n193.684361,\n180.012219,\n162.592104,\n165.724330,\n267.777171,\n163.222187,\n168.060735,\n163.174782,\n165.704061,\n204.354201,\n162.527558,\n148.731770,\n149.756350,\n144.010244,\n151.974617,\n132.640099,\n136.106890,\n148.159048,\n164.235069,\n164.391175,\n146.178410,\n151.505337,\n151.179584,\n152.841972,\n242.451406,\n129.444552,\n133.178374,\n131.760989,\n145.595123,\n152.420751,\n115.732278,\n116.811031,\n116.184848,\n114.289081,\n123.797955,\n121.154719,\n131.107601,\n118.613192,\n115.533624,\n105.778838,\n102.630257,\n111.097905,\n103.145199,\n96.450548,\n111.991947,\n101.450951,\n106.658781,\n101.503723,\n102.448760,\n102.562838,\n97.514699,\n100.894683,\n101.198875,\n102.536089,\n108.910146,\n89.400979,\n87.905842,\n86.216650,\n93.507178,\n87.545124,\n91.324082,\n85.415193,\n84.636622,\n84.046763,\n87.196268,\n84.326796,\n86.099441,\n86.810106,\n86.173009,\n88.940620,\n92.195960,\n86.064256,\n81.124127,\n100.756241,\n176.491150,\n99.678376,\n100.785342,\n97.823395,\n103.273104,\n103.159409,\n106.751627,\n100.171662,\n100.848039,\n100.186507,\n130.413496,\n99.540712,\n99.922896,\n95.540788,\n97.713666,\n198.523299,\n85.312468,\n88.189138,\n88.695380,\n83.824218,\ngeomean=123.24246908284057\nstdev=54.36151694114781\n\u003c!-- * 256M obj  - chunk sz 128M 256M  - evict, no evict - S3 geomean 256M\n\nUsing 5974ms for bench\n\ndisk=16777216KB,obj=262144KB,uuids=64iter=512\ndisk=16777216KB,obj=262144KB,uuids=5120iter=512\n\n0=19432.277805\n1=9693.144624\n2=8815.460454\n3=6778.249581\n4=6186.889682\n5=5873.107870\n6=5501.407963\n7=4484.963589\n8=4233.923278\n9=3979.102497\n10=4844.022035\n11=6341.245626\n12=4723.135366\n13=4376.366384\n14=4086.082339\nstdev=3794.30\ngeomean=5974.46\navg=6623.291940 --\u003e\n\n* 1G obj    - chunk sz 512M 1G    - evict, no evict - S3 geomean 1G\n\nUsing 24343ms for bench\n\n```c\n#define EVICT_THRESH_ZONES_LEFT 1\n#define EVICT_THRESH_ZONES_REMOVE 20\n#define ZONES_USED 100\n```\n\n```shell\nmake clean \u0026\u0026 DEBUG=0 EMULATE_S3=1 S3_DELAY=16705 make\ns=1073741824\nfor f in 'disk=104857600KB,obj=1048576KB,uuids=8000iter=800' 'disk=104857600KB,obj=1048576KB,uuids=100iter=800'; do time sudo LD_LIBRARY_PATH=/data/john/libs/lib ./znsccache -d /dev/nvme0n2 -s $s -c ../credentials.json -f ../\"$f\".txt -m /data/john/metric/\"${f}\".csv; done\n```\n\nEvict:\n\n'disk=104857600KB,obj=1048576KB,uuids=8000iter=800.txt'\n\nNo Evict:\n\n'disk=104857600KB,obj=1048576KB,uuids=100iter=800.txt'\n\n\n31742.873934,\n20459.921846,\n16902.393337,\n13546.954396,\n15273.340304,\n11994.791415,\n11758.152278,\n10777.139775,\n10871.532194,\n11983.420855,\n10886.705776,\n11063.549937,\n10945.401134,\n13324.133361,\n11112.815445,\n11251.658074,\n13924.043155,\n11014.023071,\n37683.249850,\n24814.537191,\n20653.686123,\n19499.157451,\n18427.312826,\n15931.091545,\n14961.991679,\n13972.097067,\n12936.662582,\n11780.197856,\n10959.263361,\n12327.458443,\n12680.273007,\n10940.531734,\n11299.680058,\n37958.165823,\n33148.809639,\n26662.552847,\n23368.945497,\n19471.527772,\n18066.624907,\n17507.943025,\n13863.598802,\n13308.492780,\n11170.789201,\n10898.509202,\n11076.848490,\n15592.741362,\n37776.855549,\n24481.304154,\n17931.751409,\n18058.231281,\n12805.444865,\n30234.241833,\n39333.221743,\n36547.493431,\n20231.357210,\n38013.027083,\n40259.754038,\n21216.849075,\n15809.681471,\n15897.299951,\n18149.367138,\n14628.377541,\n12057.099581,\n12114.405607,\n14343.916081,\n11683.515276,\n11607.731590,\n11627.871372,\n10970.413344,\n10857.167010,\n13122.530736,\n11257.146840,\n41844.450488,\n32068.518909,\n30608.224092,\n22599.389567,\n22018.496450,\n25119.057508,\n20138.893412,\n21157.045457,\n19701.652088,\n13252.647489,\n39225.065400,\n24026.787373,\n23220.207190,\n17358.841635,\n15225.278076,\n13179.112845,\n13834.529151,\n10915.075062,\n11492.634303,\n10939.197258,\n10934.731912,\n12756.264925,\n10784.303746,\n11050.952335,\n11087.191851,\n14068.162675,\n40537.651630,\n24682.300882,\nstdev=8768.10166286344\ngeomean=16705.160649196725\n\n```\nfor s in hitrate evictions get read write freezones cached uncach finishzone; do echo \"timestamp,type,timems\" \u003e $s.csv; grep $s obj* \u003e\u003e $s.csv; done\n```\n\n---\n\n            Runtime,        gets\n32K,noevict: 11002092.938818, 3000000\n32K,evict: 14401014.269159, 912859\n1M,noevict: 7515494.630467, 3000000\n1M,evict: 14401063.336236, 256022\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnramsden%2Fznsccache","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjohnramsden%2Fznsccache","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjohnramsden%2Fznsccache/lists"}