{"id":15049206,"url":"https://github.com/juminiy/wal","last_synced_at":"2026-01-02T02:21:55.134Z","repository":{"id":247569066,"uuid":"822624074","full_name":"Juminiy/wal","owner":"Juminiy","description":"GNU99 C Utils Collections","archived":false,"fork":false,"pushed_at":"2024-07-14T11:12:54.000Z","size":793,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-14T01:16:15.900Z","etag":null,"topics":["c99","json","json-flatten","map","memory-layout","mutex","optimized-memory","thread","wal","workgroup"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Juminiy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-01T13:45:45.000Z","updated_at":"2024-07-14T11:12:57.000Z","dependencies_parsed_at":"2024-11-19T17:42:36.608Z","dependency_job_id":"128bcea4-0f5d-40e0-848e-39fc147c7902","html_url":"https://github.com/Juminiy/wal","commit_stats":null,"previous_names":["juminiy/wal"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Juminiy%2Fwal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Juminiy%2Fwal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Juminiy%2Fwal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Juminiy%2Fwal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Juminiy","download_url":"https://codeload.github.com/Juminiy/wal/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243505960,"owners_count":20301619,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c99","json","json-flatten","map","memory-layout","mutex","optimized-memory","thread","wal","workgroup"],"created_at":"2024-09-24T21:19:05.548Z","updated_at":"2026-01-02T02:21:55.101Z","avatar_url":"https://github.com/Juminiy.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# wal\n\u003e all components either memory/disk \n\u003e 1. copied\n\u003e 2. duplicated\n\u003e 3. mutex, rwmutex protected\n\u003e 4. not in resource raced status\n\u003e ...\n\u003e anyway must be MTS\n\u003e MTS: multiple threads safe\n1. log\n   1. macro log type: info, warn, error, debug, fatal\n   2. SAFE: fmt log msg length is restricted\n   3. BUG: msg truncated and overflow\n      1. (buffer BUG in windows msys2 BUG, not mine)\n      2. MacOS-Arm64 and Ubuntu-Amd64 is OK\n      3. has been solved\n   4. TODO: enable color\n   5. hasDone: ILOG, MTS, Buffer Not Safe\n2. coworker\n   1. concurrency in multithreads\n   2. sync, async, time record\n   3. mutex, cond signal, progress record\n   4. param_list, return values set\n   5. TODO: error cancel, timeout control\n3. wal\n   1. normal log in app\n   2. fd,write fd,fsync\n   3. write ahead logging\n4. wal in coworker\n5. yyjson parse json to accelerate json parser\n   1. yyjson read jsonstring not need to be escaping, only json format is ok\n   2. struct of binding, serialize, unserialize\n   3. easy search key: full_path, short_key, value\n   4. json flatten: full_path, short_key, array_value\n   5. TODO: how to switch tag to get type and value\n6. TODO: find a c lib of map any_val -\u003e any_val\n7. all thirdparty library can be found on Github search by package name LIKE: yyjson\n8. TODO: micorsoft/mimalloc\n9. report BUG of opensource lib: uthash, c_hashmap\n10. anyway: the API design is:\n    1.  Go Server Loading a plugin of gnuc99 target .o file called plugin_json_flatten.\n    2.  go program send a original json binary bytestream to plugin_json_flatten.\n    3.  gnuc99 parse and flatten json to new json format, write a new json to a new bytestream.\n    4.  go receive a flatten json binary bytestream to parse it use predefined format in go.\n    5.  now go only to recycle garbage of original bytestream and new bytestream and new struct of.\n11. internal:\n    1.  use sc_map to store string to array of value by map[char*]void* -\u003e void* reinterpret to array of real value.\n    2.  ~~use bidirectional linked list to store the order of key.~~\n    3.  map[char*]void* to map full_path -\u003e void* reinterpret to {uint64_t tag; void * arr;}\n    4.  write back to json_string use of mut_json\n    5.  receive from Go bytestream to char *\n    6.  send char * to Go bytestream\n    7.  free gnuc99 alloc to use mimalloc\n12. memory leak saved method:\n    1.  record for all alloc but maynot be freed memory\n    2.  use freelist or mimalloc in application level mimalloc and mifree for reused and preheet\n    3.  memory leak tools like: valgrind \n    4.  try to manage memory leak\n        1.  loop for collect and signal in global obj like yyjson_doc\n        2.  change the memory management thought, convert small alloc/free to middle/PAGE_SIZE block to manage\n13. performance-test:\n    1. 100MB JSON: https://github.com/seductiveapps/largeJSON/blob/master/100mb.json\n    2. 15GB JSON: curl -L http://openlibrary.org/data/ol_cdump_latest.txt.gz  | gunzip | cut -f 5 \u003e ol_cdump.json\n14. load of .so module plugin in golang\n    1.  .so dynamic linked failed both in GNUC99 and GoRuntime1.22\n    2.  success both in GNUC99 and GoRuntime1.22 ON ARM64-Darwin\n15. use microsoft/mimalloc, OK.\n16. find a high performance xml.Decoder and xml.Encoder\n17. CGO cgo -\n18. Linux kernel -\n19. CMake Cmakelists.txt-\n20. dynamic linked how it memory layout\n21. make a json_flatten allocator learned by yyjson_doc","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuminiy%2Fwal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjuminiy%2Fwal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuminiy%2Fwal/lists"}