{"id":13667869,"url":"https://github.com/yangcancai/big_data","last_synced_at":"2025-04-10T04:10:52.883Z","repository":{"id":42191471,"uuid":"361482341","full_name":"yangcancai/big_data","owner":"yangcancai","description":"Safe Rust code for creating Erlang NIF and Redis module to store big data ","archived":false,"fork":false,"pushed_at":"2022-04-11T13:18:04.000Z","size":2337,"stargazers_count":6,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T05:26:31.181Z","etag":null,"topics":["erlang","redis-module","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yangcancai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-25T16:38:43.000Z","updated_at":"2023-02-10T10:22:45.000Z","dependencies_parsed_at":"2022-08-12T09:00:49.924Z","dependency_job_id":null,"html_url":"https://github.com/yangcancai/big_data","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yangcancai%2Fbig_data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yangcancai%2Fbig_data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yangcancai%2Fbig_data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yangcancai%2Fbig_data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yangcancai","download_url":"https://codeload.github.com/yangcancai/big_data/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248154987,"owners_count":21056543,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["erlang","redis-module","rust"],"created_at":"2024-08-02T07:00:52.795Z","updated_at":"2025-04-10T04:10:52.860Z","avatar_url":"https://github.com/yangcancai.png","language":"Rust","readme":"# big_data\nSafe Rust code for creating Erlang NIF to store big data, supported redis module  \n\n![CI](https://github.com/yangcancai/big_data/actions/workflows/ci.yml/badge.svg)\n\n## Features\n\n* Sorted\n* Range query data by time\n* Range query row_ids by time\n* Update element/counter by positon\n* Lookup element by position\n* Remove a row\n* Remove Range row by time\n* Clear Bucket\n* Query all data of the Bucket\n* Safe thread\n* Supported redis module(supoorted aof,rdb_save,rdb_load and aofrewrite)\n* Erlang External Term Format to communicate with redis\n\n## Supported erlang type\n* float(f64)\n* integer(i64)\n* atom\n* string\n* list\n* binary\n* tuple\n* map\n* pid\n* function\n* reference\n\n## Required \n- cargo 1.52.0 (32da9eaa5 2021-03-13) or later\n- rebar 3.14.4 on Erlang/OTP 22 Erts 10.7.2.1\n- redis v4.0 or greater\n\n## Comand \n```shell\n## suite test\n$ make ct  \nsh crates/build_crates.sh clippy\n    Finished dev [unoptimized + debuginfo] target(s) in 0.02s\nsh crates/build_crates.sh test\n    Finished test [unoptimized + debuginfo] target(s) in 0.02s\n     Running unittests (crates/big_data/target/debug/deps/big_data-f3da29e6da47f8c3)\n\nrunning 0 tests\n\ntest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s\n\n     Running tests/big_data_test.rs (crates/big_data/target/debug/deps/big_data_test-f8d535b5018e2ef3)\n\nrunning 14 tests\ntest clear ... ok\ntest get ... ok\ntest get_time_index ... ok\ntest get_row_ids ... ok\ntest get_range ... ok\ntest get_range_row_ids ... ok\ntest insert ... ok\ntest len_range_row_ids ... ok\ntest len_row_ids ... ok\ntest remove ... ok\ntest to_list ... ok\ntest remove_row_ids ... ok\ntest update_counter ... ok\ntest update_elem ... ok\n\ntest result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s\n\n   Doc-tests big_data\n\nrunning 12 tests\ntest src/big_data.rs - big_data::BigData::get (line 387) ... ok\ntest src/big_data.rs - big_data::BigData::insert (line 90) ... ok\ntest src/big_data.rs - big_data::BigData::len_range_row_ids (line 192) ... ok\ntest src/big_data.rs - big_data::BigData::get_time_index (line 405) ... ok\ntest src/big_data.rs - big_data::BigData::get_range (line 464) ... ok\ntest src/big_data.rs - big_data::BigData::clear (line 129) ... ok\ntest src/big_data.rs - big_data::BigData::get_range_row_ids (line 491) ... ok\ntest src/big_data.rs - big_data::BigData::get_row_ids (line 429) ... ok\ntest src/big_data.rs - big_data::BigData::len_row_ids (line 165) ... ok\ntest src/big_data.rs - big_data::BigData::remove (line 556) ... ok\ntest src/big_data.rs - big_data::BigData::remove_row_ids (line 591) ... ok\ntest src/big_data.rs - big_data::BigData::to_list (line 526) ... ok\n\ntest result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.51s\n\n./rebar3 do ct --dir test/ct -v --config test/ct/ct.config --sys_config config/test.config\n===\u003e Verifying dependencies...\n    Finished release [optimized] target(s) in 0.02s\n===\u003e Analyzing applications...\n===\u003e Compiling big_data_nif\n===\u003e Running Common Test suites...\n\nCommon Test starting (cwd is /Users/admin/proj/rust/big_data_nif)\n\n\n\nCWD set to: \"/Users/admin/proj/rust/big_data_nif/_build/test/logs/ct_run.nonode@nohost.2021-05-13_15.54.32\"\n\nTEST INFO: 1 test(s), 11 case(s) in 1 suite(s)\n\nTesting test.ct: Starting test, 11 test cases\n%%% big_data_SUITE: ...........\nTesting test.ct: TEST COMPLETE, 11 ok, 0 failed of 11 test cases\n\nUpdating /Users/admin/proj/rust/big_data_nif/_build/test/logs/index.html ... done\nUpdating /Users/admin/proj/rust/big_data_nif/_build/test/logs/all_runs.html ... done\n$ make shell\n# ./tool.sh replace_config\n./rebar3 as test shell\n===\u003e Verifying dependencies...\n    Finished release [optimized] target(s) in 0.04s\n===\u003e Analyzing applications...\n===\u003e Compiling big_data_nif\nErlang/OTP 22 [erts-10.7.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]\n\nEshell V10.7.2.1  (abort with ^G)\n1\u003e ===\u003e Booted big_data_nif\n\n\n```\n## Redis Example(BigData)\n\n### Redis Command\n```rust\n        [\"big_data.set\", big_data_set, \"write\", 1, 1, 1],\n        [\"big_data.update_elem\", big_data_update_elem, \"write\", 1, 1, 1],\n        [\"big_data.update_counter\", big_data_update_counter, \"write\", 1, 1, 1],\n        [\"big_data.remove\", big_data_remove, \"write\", 1, 1, 1],\n        [\"big_data.remove_row\", big_data_remove_row, \"write\", 1, 1, 1],\n        [\"big_data.remove_row_ids\", big_data_remove_row_ids, \"write\", 1, 1, 1],\n        [\"big_data.get_row\", big_data_get_row, \"readonly\", 1, 1, 1],\n        [\"big_data.get\", big_data_get, \"readonly\", 1, 1, 1],\n        [\"big_data.get_range\", big_data_get_range, \"readonly\", 1, 1, 1],\n        [\"big_data.get_range_row_ids\", big_data_get_range_row_ids, \"readonly\", 1, 1, 1],\n        [\"big_data.get_row_ids\", big_data_get_row_ids, \"readonly\", 1, 1, 1],\n        [\"big_data.get_time_index\", big_data_get_time_index, \"readonly\", 1, 1, 1],\n        [\"big_data.lookup_elem\", big_data_lookup_elem, \"readonly\", 1, 1, 1],\n\n```\n### Erlang Command(big_data_redis)\n```erlang\n158\u003e {ok,P} = big_data_redis:new().\n{ok,\u003c0.410.0\u003e}\n159\u003e big_data_redis:insert(P,\u003c\u003c\"a\"\u003e\u003e,\u003c\u003c\"rowid1\"\u003e\u003e,erlang:system_time(1),{a,8,\u003c\u003c\"a\"\u003e\u003e,\"d\",[{a,b}]}).\nok\n160\u003e big_data_redis:get_row(P,\u003c\u003c\"a\"\u003e\u003e,\u003c\u003c\"rowid1\"\u003e\u003e).\n[{row_data,\u003c\u003c\"rowid1\"\u003e\u003e,\n           {a,8,\u003c\u003c\"a\"\u003e\u003e,\"d\",[{a,b}]},\n           1641522837}]\n161\u003e big_data_redis:get(P,\u003c\u003c\"a\"\u003e\u003e).\n[\n {row_data,\u003c\u003c\"rowid1\"\u003e\u003e,\n           {a,8,\u003c\u003c\"a\"\u003e\u003e,\"d\",[{a,b}]},\n           1641522837}]\n163\u003e big_data_redis:update_elem(P,\u003c\u003c\"a\"\u003e\u003e,\u003c\u003c\"rowid1\"\u003e\u003e,{0,{1,a,\u003c\u003c\"a\"\u003e\u003e}}).\n[true]\n164\u003e big_data_redis:get_row(P,\u003c\u003c\"a\"\u003e\u003e,\u003c\u003c\"rowid1\"\u003e\u003e).\n[{row_data,\u003c\u003c\"rowid1\"\u003e\u003e,\n           {{1,a,\u003c\u003c\"a\"\u003e\u003e},8,\u003c\u003c\"a\"\u003e\u003e,\"d\",[{a,b}]},\n           1641522837}]\n166\u003e big_data_redis:update_counter(P,\u003c\u003c\"a\"\u003e\u003e,\u003c\u003c\"rowid1\"\u003e\u003e,[{1,1}]).\n[true]\n167\u003e big_data_redis:get_row(P,\u003c\u003c\"a\"\u003e\u003e,\u003c\u003c\"rowid1\"\u003e\u003e).\n[{row_data,\u003c\u003c\"rowid1\"\u003e\u003e,\n           {{1,a,\u003c\u003c\"a\"\u003e\u003e},9,\u003c\u003c\"a\"\u003e\u003e,\"d\",[{a,b}]},\n           1641522837}]\n168\u003e big_data_redis:update_counter(P,\u003c\u003c\"a\"\u003e\u003e,\u003c\u003c\"rowid1\"\u003e\u003e,[{1,10}]).\n[true]\n169\u003e big_data_redis:get_row(P,\u003c\u003c\"a\"\u003e\u003e,\u003c\u003c\"rowid1\"\u003e\u003e).\n[{row_data,\u003c\u003c\"rowid1\"\u003e\u003e,\n           {{1,a,\u003c\u003c\"a\"\u003e\u003e},19,\u003c\u003c\"a\"\u003e\u003e,\"d\",[{a,b}]},\n           1641522837}]\n170\u003e big_data_redis:lookup_elem(P,\u003c\u003c\"a\"\u003e\u003e,\u003c\u003c\"rowid1\"\u003e\u003e,{0,1}).\n{{1,a,\u003c\u003c\"a\"\u003e\u003e},19}\n171\u003e big_data_redis:lookup_elem(P,\u003c\u003c\"a\"\u003e\u003e,\u003c\u003c\"rowid1\"\u003e\u003e,{0}).\n{{1,a,\u003c\u003c\"a\"\u003e\u003e}}\n172\u003e big_data_redis:lookup_elem(P,\u003c\u003c\"a\"\u003e\u003e,\u003c\u003c\"rowid1\"\u003e\u003e,{10}).\n{}\n174\u003e big_data_redis:lookup_elem(P,\u003c\u003c\"a\"\u003e\u003e,\u003c\u003c\"rowid1\"\u003e\u003e,[4]).\n{[{a,b}]}\n```\n\n## Nif Example\n### Insert and Get\n```erlang\n%% create a player bucket\n3\u003e {ok, Ref} = big_data_nif:new().\n{ok,#Ref\u003c0.2349964349.40239108.14596\u003e}\n%% insert a row to bucket\n4\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"1\"\u003e\u003e,{a,1},1}).\nok\n%% query all data of player bucket\n5\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e).\n[{row_data,\u003c\u003c\"1\"\u003e\u003e,{a,1},1}]\n%% query a row \n6\u003e big_data_nif:get_row(Ref,\u003c\u003c\"player\"\u003e\u003e, \u003c\u003c\"1\"\u003e\u003e).\n{row_data,\u003c\u003c\"1\"\u003e\u003e,{a,1},1}\n%% insert other row\n7\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"2\"\u003e\u003e,{a,1},1}).\nok\n8\u003e big_data_nif:get_row(Ref,\u003c\u003c\"player\"\u003e\u003e, \u003c\u003c\"2\"\u003e\u003e).                   \n{row_data,\u003c\u003c\"2\"\u003e\u003e,{a,1},1}\n9\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e).                                \n[{row_data,\u003c\u003c\"1\"\u003e\u003e,{a,1},1},{row_data,\u003c\u003c\"2\"\u003e\u003e,{a,1},1}]\n```\n\n### Range Query\n\n```erlang\n1\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e).\n[]\n12\u003e \n12\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"2\"\u003e\u003e,{a,1},1}).\nok\n13\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"1\"\u003e\u003e,{a,1},10}).\nok\n14\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1},0}). \nok\n%% query all data which will be sorted by time \n15\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e).                                 \n[{row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1},0},\n {row_data,\u003c\u003c\"2\"\u003e\u003e,{a,1},1},\n {row_data,\u003c\u003c\"1\"\u003e\u003e,{a,1},10}]\n16\u003e big_data_nif:get_range(Ref,\u003c\u003c\"player\"\u003e\u003e,0,1).\n[{row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1},0},{row_data,\u003c\u003c\"2\"\u003e\u003e,{a,1},1}]\n17\u003e big_data_nif:get_range(Ref,\u003c\u003c\"player\"\u003e\u003e,0,0).\n[{row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1},0}]\n18\u003e big_data_nif:get_range(Ref,\u003c\u003c\"player\"\u003e\u003e,0,10).\n[{row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1},0},\n {row_data,\u003c\u003c\"2\"\u003e\u003e,{a,1},1},\n {row_data,\u003c\u003c\"1\"\u003e\u003e,{a,1},10}]\n\n%% query the row_ids by range time\n%% included 0 and included 10\n%% result will be sorted by time\n 19\u003e big_data_nif:get_range_row_ids(Ref,\u003c\u003c\"player\"\u003e\u003e,0,10).\n[\u003c\u003c\"3\"\u003e\u003e,\u003c\u003c\"2\"\u003e\u003e,\u003c\u003c\"1\"\u003e\u003e]\n%% query the time by row_id\n22\u003e big_data_nif:get_time_index(Ref,\u003c\u003c\"player\"\u003e\u003e,\u003c\u003c\"1\"\u003e\u003e).\n10\n```\n### Lookup Elem\n```erlang\n33\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1,{a,b},\u003c\u003c\"hello\"\u003e\u003e},0}).\nok\n34\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e).                                                  \n[{row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1,{a,b},\u003c\u003c\"hello\"\u003e\u003e},0}]\n35\u003e big_data_nif:lookup_elem(Ref,\u003c\u003c\"player\"\u003e\u003e,\u003c\u003c\"3\"\u003e\u003e,0).\n{a}\n36\u003e big_data_nif:lookup_elem(Ref,\u003c\u003c\"player\"\u003e\u003e,\u003c\u003c\"3\"\u003e\u003e,[0,1]).\n{a,1}\n37\u003e big_data_nif:lookup_elem(Ref,\u003c\u003c\"player\"\u003e\u003e,\u003c\u003c\"3\"\u003e\u003e,[0,1,3]).\n{a,1,\u003c\u003c\"hello\"\u003e\u003e}\n```\n\n### Update Elem\n\n```erlang\n24\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1},0}).\nok\n25\u003e big_data_nif:update_elem(Ref,\u003c\u003c\"player\"\u003e\u003e,\u003c\u003c\"3\"\u003e\u003e,[{0,b},{1,2}]).  \n[true,true]\n26\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e).\n[{row_data,\u003c\u003c\"3\"\u003e\u003e,{b,2},0}]\n```\n### Update Counter\n```erlang\n28\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1,3,{4,5}},0}).\nok\n29\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e).                                        \n[{row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1,3,{4,5}},0}]\n30\u003e big_data_nif:update_counter(Ref,\u003c\u003c\"player\"\u003e\u003e,\u003c\u003c\"3\"\u003e\u003e,[{1,1},{2,5}]).       \n[true,true]\n31\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e).                                 \n[{row_data,\u003c\u003c\"3\"\u003e\u003e,{a,2,8,{4,5}},0}]\n```\n\n### Remove \n\n```erlang\n2\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1,3,{4,5}},0}).\nok\n3\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"4\"\u003e\u003e,{a,1,3,{4,5}},0}).\n4\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e).\n[{row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1,3,{4,5}},0},\n {row_data,\u003c\u003c\"4\"\u003e\u003e,{a,1,3,{4,5}},0}]\n %% remove player bucket\n6\u003e big_data_nif:remove(Ref,\u003c\u003c\"player\"\u003e\u003e).\nok\n7\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e).   \n[]\n8\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"4\"\u003e\u003e,{a,1,3,{4,5}},0}).\nok\n9\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1,3,{4,5}},0}).\nok\n%% remove a row which belong to player bucket\n10\u003e big_data_nif:remove_row(Ref,\u003c\u003c\"player\"\u003e\u003e,\u003c\u003c\"3\"\u003e\u003e).\nok\n11\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e).                                        \n[{row_data,\u003c\u003c\"4\"\u003e\u003e,{a,1,3,{4,5}},0}]\n12\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1,3,{4,5}},0}).\nok\n%% remove all rows which time between 0 and 0\n13\u003e big_data_nif:remove_row_ids(Ref,\u003c\u003c\"player\"\u003e\u003e,0,0).                         \nok\n14\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e).                                        \n[]\n15\u003e big_data_nif:insert(Ref, \u003c\u003c\"player\"\u003e\u003e, {row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1,3,{4,5}},0}).\nok\n16\u003e big_data_nif:insert(Ref, \u003c\u003c\"player1\"\u003e\u003e, {row_data,\u003c\u003c\"4\"\u003e\u003e,{a,1,3,{4,5}},0}).\nok\n17\u003e big_data_nif:get(Ref,\u003c\u003c\"player1\"\u003e\u003e).\n[{row_data,\u003c\u003c\"4\"\u003e\u003e,{a,1,3,{4,5}},0}]\n18\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e). \n[{row_data,\u003c\u003c\"3\"\u003e\u003e,{a,1,3,{4,5}},0}]\n%% clear all buckets\n19\u003e big_data_nif:clear(Ref).\nok\n20\u003e big_data_nif:get(Ref,\u003c\u003c\"player1\"\u003e\u003e).\n[]\n21\u003e big_data_nif:get(Ref,\u003c\u003c\"player\"\u003e\u003e). \n[]\n```\n\n## Bench\n### cargo bench\n```shell\n$ sh crates/build_crates.sh bench\n    Finished bench [optimized] target(s) in 0.02s\n     Running unittests (crates/big_data/target/release/deps/big_data-a47418a78ef79a70)\n\nrunning 0 tests\n\ntest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s\n\n     Running unittests (crates/big_data/target/release/deps/bench-2cb028b9ea72486c)\n\nrunning 2 tests\ntest get    ... bench:         216 ns/iter (+/- 133)\ntest insert ... bench:       1,160 ns/iter (+/- 141)\n\ntest result: ok. 0 passed; 0 failed; 0 ignored; 2 measured\n```\n### FlameGraph\n\n#### MacOS start dtrace to set probe\n```shell\n$ ./flamegraph.sh\ndtrace: system integrity protection is on, some features will not be available\n\ndtrace: description 'profile-997 ' matched 1 probe\n```\n#### Start redis-server\nFirst of all build redis_api `make redis_api` then output library to `target/debug/libredis_api.dylib` or \n`target/release/libredis_api.dylib`. The below command to load redis module\n```shell\n$ redis-server --loadmodule target/debug/libredis_api.dylib\n42527:C 11 Jan 2022 17:24:07.942 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo\n42527:C 11 Jan 2022 17:24:07.942 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=42527, just started\n42527:C 11 Jan 2022 17:24:07.942 # Configuration loaded\n42527:M 11 Jan 2022 17:24:07.943 * Increased maximum number of open files to 10032 (it was originally set to 256).\n42527:M 11 Jan 2022 17:24:07.943 * monotonic clock: POSIX clock_gettime\n                _._\n           _.-``__ ''-._\n      _.-``    `.  `_.  ''-._           Redis 6.2.6 (00000000/0) 64 bit\n  .-`` .-```.  ```\\/    _.,_ ''-._\n (    '      ,       .-`  | `,    )     Running in standalone mode\n |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379\n |    `-._   `._    /     _.-'    |     PID: 42527\n  `-._    `-._  `-./  _.-'    _.-'\n |`-._`-._    `-.__.-'    _.-'_.-'|\n |    `-._`-._        _.-'_.-'    |           https://redis.io\n  `-._    `-._`-.__.-'_.-'    _.-'\n |`-._`-._    `-.__.-'    _.-'_.-'|\n |    `-._`-._        _.-'_.-'    |\n  `-._    `-._`-.__.-'_.-'    _.-'\n      `-._    `-.__.-'    _.-'\n          `-._        _.-'\n              `-.__.-'\n\n42527:M 11 Jan 2022 17:24:07.944 # Server initialized\n42527:M 11 Jan 2022 17:24:07.946 * \u003cbig_data\u003e Created new data type 'big_data1'\n42527:M 11 Jan 2022 17:24:07.946 * Module 'big_data' loaded from target/debug/libredis_api.dylib\n```\n#### Start big_data erlang client\n```shell\n$ make shell\n./rebar3 as test shell\n===\u003e Verifying dependencies...\n/Users/admin/proj/erlang/big_data/target\n   Compiling big_data v0.1.2 (/Users/admin/proj/erlang/big_data/crates/big_data)\n    Finished release [optimized] target(s) in 2.70s\n===\u003e Analyzing applications...\n===\u003e Compiling big_data\nErlang/OTP 22 [erts-10.7.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]\n\nEshell V10.7.2.1  (abort with ^G)\n1\u003e =INFO REPORT==== 11-Jan-2022::17:26:46.165809 ===\nSetup running ...\n=INFO REPORT==== 11-Jan-2022::17:26:46.175540 ===\nDirectories verified. Res = ok\n=INFO REPORT==== 11-Jan-2022::17:26:46.175985 ===\nSetup finished processing hooks (Mode=normal)...\n=DEBUG REPORT==== 11-Jan-2022::17:26:46.641428 ===\nWaiting wal recover...\n=DEBUG REPORT==== 11-Jan-2022::17:26:46.641624 ===\nRecover: ..#{dir =\u003e \"data\",waiting_pid =\u003e \u003c0.393.0\u003e}\n=DEBUG REPORT==== 11-Jan-2022::17:26:46.648339 ===\nCheckpointSeq = 0, id_seq = 0, walfiles = [\"data/1_00000001.wal\",\n                                           \"data/1_00000002.wal\"]\n=DEBUG REPORT==== 11-Jan-2022::17:26:46.648581 ===\nCheckpoint sync timeout:600000\n=DEBUG REPORT==== 11-Jan-2022::17:26:46.648733 ===\nFinish checkpoint id_seq = 0, checkpointseq = 0\n=DEBUG REPORT==== 11-Jan-2022::17:26:46.660010 ===\nNotify recover finished, Pid = \u003c0.393.0\u003e\n=DEBUG REPORT==== 11-Jan-2022::17:26:46.660359 ===\nWal recover finished, Recover total_time = 19 ms\n===\u003e Booted recon\n===\u003e Booted observer_cli\n===\u003e Booted hut\n===\u003e Booted setup\n===\u003e Booted bear\n===\u003e Booted folsom\n===\u003e Booted syntax_tools\n===\u003e Booted parse_trans\n===\u003e Booted exometer_core\n===\u003e Booted pa\n===\u003e Booted quickrand\n===\u003e Booted uuid\n===\u003e Booted xmerl\n===\u003e Booted jiffy\n===\u003e Booted flatlog\n===\u003e Booted cool_tools\n===\u003e Booted big_data\n1\u003ebd_bench:run(500,500).\n......\nBech done pid = \u003c0.857.0\u003e\nOverview = #{aver =\u003e 20,sum_time =\u003e 1488897303,total_command =\u003e 72868,\n             total_time =\u003e 4684,tps =\u003e 18217}\n```\n![FlameGraph](pretty-graph.svg)\n\n## Reference\n* [redismodule-rs](https://github.com/RedisLabsModules/redismodule-rs.git)\n* [rabbitmq/ra](https://github.com/rabbitmq/ra.git)","funding_links":[],"categories":["库"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyangcancai%2Fbig_data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyangcancai%2Fbig_data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyangcancai%2Fbig_data/lists"}