{"id":20398053,"url":"https://github.com/zgbjgg/kaa","last_synced_at":"2025-04-12T13:09:53.543Z","repository":{"id":57511245,"uuid":"97528432","full_name":"zgbjgg/kaa","owner":"zgbjgg","description":"KAA - support to execute jun tasks using protocol buffers","archived":false,"fork":false,"pushed_at":"2020-02-28T16:38:31.000Z","size":837,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-12T13:09:49.883Z","etag":null,"topics":["dataframe","elixir","erlang","jun","numpy","pandas","plotly","proto-messages","protocol-buffers","python"],"latest_commit_sha":null,"homepage":null,"language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zgbjgg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-07-17T22:38:36.000Z","updated_at":"2020-11-17T14:33:53.000Z","dependencies_parsed_at":"2022-09-26T16:31:09.287Z","dependency_job_id":null,"html_url":"https://github.com/zgbjgg/kaa","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zgbjgg%2Fkaa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zgbjgg%2Fkaa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zgbjgg%2Fkaa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zgbjgg%2Fkaa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zgbjgg","download_url":"https://codeload.github.com/zgbjgg/kaa/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248571884,"owners_count":21126522,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataframe","elixir","erlang","jun","numpy","pandas","plotly","proto-messages","protocol-buffers","python"],"created_at":"2024-11-15T04:18:05.477Z","updated_at":"2025-04-12T13:09:53.520Z","avatar_url":"https://github.com/zgbjgg.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kaa\n![Kaa](https://user-images.githubusercontent.com/1471055/28333979-4c770d16-6bbf-11e7-90b1-a394d6e7e414.png)\n\n**KAA** - support to execute jun tasks using protocol buffers\n\n[![Hex.pm](https://img.shields.io/hexpm/v/kaa.svg)](https://hex.pm/packages/kaa) \n[![Build Status](https://travis-ci.org/zgbjgg/kaa.svg?branch=master)](https://travis-ci.org/zgbjgg/kaa)\n[![Codecov](https://img.shields.io/codecov/c/github/zgbjgg/kaa.svg)](https://codecov.io/gh/zgbjgg/kaa)\n[![License: MIT](https://img.shields.io/github/license/zgbjgg/kaa.svg)](https://raw.githubusercontent.com/zgbjgg/kaa/master/LICENSE)\n[![Hex.pm](https://img.shields.io/hexpm/dt/kaa.svg)](https://hex.pm/packages/kaa)\n[![Hex.pm](https://img.shields.io/hexpm/dw/kaa.svg)](https://hex.pm/packages/kaa)\n\nKAA is a library written in erlang to send and receive tasks to a `JUN` worker through a protocol buffers, so then any client implementing the proto files can send messages to `JUN`.\n\nThis project is under development and should not be used in production, it's not ready for that.\n\n### Creating the environment\n\nTo create a new kaa worker (that also creates a jun worker onto it):\n\n```erlang\n(kaa@hurakann)1\u003e {ok, Key} = kaa_main_worker:start_link().\n14:28:02.689 [info] initialized default modules for py pid \u003c0.422.0\u003e\n14:28:03.042 [info] initialized jun worker pid \u003c0.421.0\u003e\n{ok,\u003c\u003c\"ngioehzlskpwopeypenolaztsalgerupwlnimzzxxtfwdvcfatevctjogkdhmzxatjljzarvgavmjhlciwfvuastmkyqctkpffmv\"\u003e\u003e}\n```\n\nThis will create and hold the environment for kaa and jun, so kaa can now receive messages encoded into a protocol buffers and pass to the jun worker, receive the response and delivered as proto message.\n\n### Building a proto messages\n\nNow we are executing some tasks into jun, but using the proto messages in kaa format, so for example, let's generate a message to read a csv into dataframe:\n\n```erlang\n(kaa@hurakann)2\u003e ReadCsvInstruction = kaa_proto:read_csv_instruction(\"\u003c0.421.0\u003e\", \"/Users/zgbjgg/jun-nogit/file.csv\").    \n\u003c\u003c8,0,16,0,26,9,60,48,46,52,50,49,46,48,62,10,32,47,85,\n  115,101,114,115,47,122,103,98,106,103,...\u003e\u003e\n```\n\nAs you can see, the returned value is a binary in a protocol buffers format (for kaa), so this can be decoded using the proto file. Let's send the proto message to the kaa worker:\n\n```erlang\n(kaa@hurakann)3\u003e {ok, Result} = kaa_main_worker:kaa_proto_in(Key, ReadCsvInstruction).\n{ok,\u003c\u003c10,2,111,107,26,234,6,131,104,3,100,0,15,36,101,\n      114,108,112,111,114,116,46,111,112,97,113,117,...\u003e\u003e}\n```\n\nNow we have a `Result` variable with a proto message in a binary format, let's decode the message (this can be done by any client):\n\n```erlang\n(kaa@hurakann)4\u003e rr(\"/Users/zgbjgg/kaa/include/kaa_result.hrl\").\n['KaaResult']\n(kaa@hurakann)5\u003e KaaResult = kaa_result:decode_msg(Result, 'KaaResult'). \n#'KaaResult'{ok = \"ok\",\n             result = {dataframe,\u003c\u003c131,104,3,100,0,15,36,101,114,108,\n                                   112,111,114,116,46,111,112,97,113,\n                                   117,101,100,0,6,...\u003e\u003e}}\n```\n\n### Working with dataframes\n\nNow we have a decoded result, the dataframe read by the read_csv function, lets play with the dataframe, first to all get only the dataframe value:\n\n```erlang\n(kaa@hurakann)6\u003e {dataframe, DataFrame} = KaaResult#'KaaResult'.result. \n{dataframe,\u003c\u003c131,104,3,100,0,15,36,101,114,108,112,111,\n             114,116,46,111,112,97,113,117,101,100,0,6,\n             112,121,116,...\u003e\u003e}\n```\n\nGenerate another instruction, this time we are going to sum all values of column age from dataframe:\n\n```erlang\n(kaa@hurakann)7\u003e SumAgeInstruction = kaa_proto:sum_age_instruction(\"\u003c0.421.0\u003e\", DataFrame, sum, \"age\"). \n\u003c\u003c8,0,16,1,26,9,60,48,46,52,50,49,46,48,62,18,242,6,10,\n  234,6,131,104,3,100,0,15,36,101,...\u003e\u003e\n```\n\nWe have now the sum instruction into a proto message format, let's send to kaa and finally parse the result from proto result message:\n\n```erlang\n(kaa@hurakann)8\u003e {ok, Result2} = kaa_main_worker:kaa_proto_in(Key, SumAgeInstruction).                 \n{ok,\u003c\u003c10,2,111,107,8,13\u003e\u003e}\n(kaa@hurakann)9\u003e kaa_result:decode_msg(Result2, 'KaaResult').   \n#'KaaResult'{ok = \"ok\",result = {inumber,13}}\n```\n\nReady!, if check the result we have an inumber with a value of 13 which is the correct sum of all columns!\n\n### See also\n\n[JUN: python pandas support for dataframes manipulation over erlang](https://github.com/zgbjgg/jun)\n\n### Authors\n\n@zgbjgg Jorge Garrido \u003czgbjgg@gmail.com\u003e\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzgbjgg%2Fkaa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzgbjgg%2Fkaa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzgbjgg%2Fkaa/lists"}