{"id":20806334,"url":"https://github.com/nodesource/poblado","last_synced_at":"2025-05-11T17:31:45.820Z","repository":{"id":57326515,"uuid":"114388645","full_name":"nodesource/poblado","owner":"nodesource","description":"Parses and processes JSON that is written to it in chunks on a background thread.","archived":true,"fork":false,"pushed_at":"2017-12-22T17:24:30.000Z","size":22,"stargazers_count":1,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-18T09:19:25.619Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nodesource.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-15T16:09:52.000Z","updated_at":"2025-02-01T13:59:41.000Z","dependencies_parsed_at":"2022-09-21T02:01:25.552Z","dependency_job_id":null,"html_url":"https://github.com/nodesource/poblado","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodesource%2Fpoblado","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodesource%2Fpoblado/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodesource%2Fpoblado/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nodesource%2Fpoblado/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nodesource","download_url":"https://codeload.github.com/nodesource/poblado/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253604857,"owners_count":21934910,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T19:19:21.783Z","updated_at":"2025-05-11T17:31:45.548Z","avatar_url":"https://github.com/nodesource.png","language":"C++","readme":"# poblado [![build status](https://secure.travis-ci.org/nodesource/poblado.svg?branch=master)](http://travis-ci.org/nodesource/poblado)\n\nParses and processes JSON that is written to it in chunks on a background thread.\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*\n\n- [Dependencies](#dependencies)\n- [API](#api)\n  - [`void onparsedToken(rapidjson_writable::SaxHandler\u0026 handler)`](#void-onparsedtokenrapidjson_writablesaxhandler-handler)\n  - [`void processParsedData()`](#void-processparseddata)\n  - [`void onprocessingComplete()`](#void-onprocessingcomplete)\n  - [Overriding `ASSERT`](#overriding-assert)\n  - [Logging Diagnostics via `POBLADO_LOG`](#logging-diagnostics-via-poblado_log)\n- [Example](#example)\n  - [Output](#output)\n- [LICENSE](#license)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n\n## Dependencies\n\nPoblado depends on the following Headers\n\n- [rapidjson](https://github.com/Tencent/rapidjson)\n- [rapidjson-writable](https://github.com/nodesource/rapidjson-writable)\n- [libuv](https://github.com/libuv/libuv)\n\nMake sure you are including all those headers when building and in the case of libuv you need to either link to the\nlibuv library as well or make it part of your build.\n\n## API\n\nIn order to use poblado you'll extend the `Poblado` class and override the following methods.\n\n```cc\n#include \"poblado.h\"\n\nclass MyPoblado : public Poblado {\n  void onparsedToken(rapidjson_writable::SaxHandler\u0026 handler);\n  void processParsedData();\n  void onprocessingComplete();\n};\n```\n\nThe methods are invoked either on the _parser/processor (background) thread_ or the _main thread_ (same as libuv's event\nloop).\n\npoblado guarantees that it will not invoke any _background thread_ methods after invoking `onprocessingComplete` on the\n_main thread_. Therefore users of the API don't have to use any thread synchronization as it is all handled under the\nhood by poblado.\n\n### `void onparsedToken(rapidjson_writable::SaxHandler\u0026 handler)`\n\n- invoked on background thread.\n- compare the `handler.type` to a `rapidjson_writable::JsonType` and from that decide how to process the respective values.\n- for more information have a look at the\n  [rapidjson_writable::SaxHandler](https://github.com/nodesource/rapidjson-writable/blob/master/include/rapidjson_saxhandler.h)\n  implementation\n- store the extracted data in fields on your class so they are available in the following processing steps\n\n### `void processParsedData()`\n\n- invoked on background thread.\n- here the user can perform further processing on the data collected via `onparsedToken`\n\n### `void onprocessingComplete()`\n\n- invoked on main thread.\n- this method is invoked when both _background_ processing methods have completed\n- at this point the processing methods will not be called anymore and thus the process result can safely be accessed\n  on the main thread\n\n### Overriding `ASSERT`\n\npoblado uses a built in `ASSERT` method which can be overridden.\n\n```cc\n#define POBLADO_ASSERT MY_ASSERT\n#include \"poblado.h\"\n```\n\n### Logging Diagnostics via `POBLADO_LOG`\n\nBy default poblado is quiet, but has built in support to log diagnostic messages. All you have to do is supply a\n`POBLADO_LOG` method that takes a `char*` as an input.\n\n```cc\n#define POBLADO_LOG MY_LOG\n#include \"poblado.h\"\n```\n\nHave a look at the top of [test/test_dump.cc](test/test_dump.cc) to see an example of that feature.\n\n## Example\n\nBelow is a sample implementation of the `Poblado` class. For a more complete example please read\n[test/test_dump.cc](test/test_dump.cc) or [test/test_capitalize.cc](test/test_capitalize.cc).\n\n```cc\nclass ExtractKeysProcessor : public Poblado {\n  public:\n    ExtractKeysProcessor(uv_loop_t\u0026 loop) : Poblado(loop) {}\n\n  private:\n// background thread {\n\n    // @override\n    void onparsedToken(rapidjson_writable::SaxHandler\u0026 handler) {\n      TEST_LOG(\"[processor] parsed a token\");\n      if (handler.type != rapidjson_writable::JsonType::Key) return;\n      keys_.push_back(scopy(handler.stringVal.c_str()));\n    }\n\n    // @override\n    void processParsedData() {\n      for (auto\u0026 key : keys_) {\n        TEST_LOG(key);\n        processedKeys_.push_back(string_toupper(key));\n\n        // simulate that processing keys is slow\n        uv_sleep(200);\n      }\n    }\n\n// } background thread\n\n// main thread {\n\n    // @override\n    void onprocessingComplete() {\n      if (hasError()) {\n        TEST_LOG(\"[collector] found error\");\n        fprintf(stderr, \"%s\\n\", \u0026error());\n        return;\n      }\n\n      for (auto\u0026 key : processedKeys_) {\n        TEST_LOG(\"[collector] printing key\");\n        TEST_LOG(key);\n      }\n    }\n\n// } main thread\n\n    std::vector\u003cconst char*\u003e keys_;\n    std::vector\u003cconst char*\u003e processedKeys_;\n};\n```\n\n### Output\n\nNote that the output is using a log method to provide diagnostic messages in order to demonstrate how things work under\nthe hood.\n\nThe output format is as follows:\n\n```\n[time](file:line)[thread id] [topic] message\n```\n\n```\n➝  ./bin/test_dump test/fixtures/capitals.json\n[main] processing test/fixtures/capitals.json\n[00000](test_dump.cc:157)[0x7fff88639340] [main] starting loop\n[00604](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token\n[00604](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token\n[00604](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token\n[00604](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token\n[00604](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token\n[01226](test_dump.cc:85)[0x7fff88639340] [ticker] tick\n[01226](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token\n[01226](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token\n[01226](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token\n[01226](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token\n[01226](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token\n[01226](poblado.h:89)[0x700004f9f000] [processor] parser complete, processing ...\n[01226](test_dump.cc:47)[0x700004f9f000] colombia\n[01429](test_dump.cc:47)[0x700004f9f000] australia\n[01629](test_dump.cc:47)[0x700004f9f000] germany\n[01832](test_dump.cc:47)[0x700004f9f000] united states\n[02037](poblado.h:141)[0x700004f9f000] [processor] signaling processing is complete\n[02037](poblado.h:123)[0x7fff88639340] [collector] found processing is complete, reading results\n[02037](test_dump.cc:68)[0x7fff88639340] [collector] printing key\n[02037](test_dump.cc:69)[0x7fff88639340] COLOMBIA\n[02037](test_dump.cc:68)[0x7fff88639340] [collector] printing key\n[02037](test_dump.cc:69)[0x7fff88639340] AUSTRALIA\n[02037](test_dump.cc:68)[0x7fff88639340] [collector] printing key\n[02037](test_dump.cc:69)[0x7fff88639340] GERMANY\n[02037](test_dump.cc:68)[0x7fff88639340] [collector] printing key\n[02037](test_dump.cc:69)[0x7fff88639340] UNITED STATES\n[02457](test_dump.cc:85)[0x7fff88639340] [ticker] tick\n```\n\n## LICENSE\n\nMIT\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnodesource%2Fpoblado","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnodesource%2Fpoblado","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnodesource%2Fpoblado/lists"}