{"id":26600802,"url":"https://github.com/scinim/datamancer","last_synced_at":"2025-04-06T20:08:53.927Z","repository":{"id":39582842,"uuid":"377584345","full_name":"SciNim/Datamancer","owner":"SciNim","description":"A dataframe library with a dplyr like API","archived":false,"fork":false,"pushed_at":"2024-10-13T14:10:54.000Z","size":983,"stargazers_count":139,"open_issues_count":8,"forks_count":7,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-30T19:05:08.683Z","etag":null,"topics":["dataframe","dplyr","hacktoberfest","nim","nim-lang"],"latest_commit_sha":null,"homepage":"https://scinim.github.io/Datamancer/datamancer.html","language":"Nim","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SciNim.png","metadata":{"files":{"readme":"README.org","changelog":"changelog.org","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-16T17:58:42.000Z","updated_at":"2025-03-30T02:34:11.000Z","dependencies_parsed_at":"2024-01-06T11:49:23.812Z","dependency_job_id":"a5eeb9a4-b4ca-48e1-87dd-c585de416bd1","html_url":"https://github.com/SciNim/Datamancer","commit_stats":null,"previous_names":[],"tags_count":47,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SciNim%2FDatamancer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SciNim%2FDatamancer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SciNim%2FDatamancer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SciNim%2FDatamancer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SciNim","download_url":"https://codeload.github.com/SciNim/Datamancer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246365644,"owners_count":20765546,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataframe","dplyr","hacktoberfest","nim","nim-lang"],"created_at":"2025-03-23T18:35:27.039Z","updated_at":"2025-03-30T19:05:13.676Z","avatar_url":"https://github.com/SciNim.png","language":"Nim","funding_links":[],"categories":[],"sub_categories":[],"readme":"* Datamancer\n[[https://github.com/SciNim/datamancer/workflows/datamancer%20CI/badge.svg]]\n[[https://matrix.to/#/#nim-science:envs.net][https://img.shields.io/static/v1?message=join%20chat\u0026color=blue\u0026label=nim-science\u0026logo=matrix\u0026logoColor=gold\u0026style=flat-square\u0026.svg]]\n[[https://discord.gg/f5hA9UK3dY][https://img.shields.io/discord/371759389889003530?color=blue\u0026label=nim-science\u0026logo=discord\u0026logoColor=gold\u0026style=flat-square\u0026.svg]]\n\n~Datamancer~ is a DataFrame library for Nim, which is heavily inspired\nby [[https://dplyr.tidyverse.org/][dplyr]].\n\n** Comparison to other dataframe libraries\n\nCheck out the following gist for a comparison of this library with\ndplyr (R) and pandas (Python):\n\nhttps://gist.github.com/Vindaar/6908c038707c7d8293049edb3d204f84\n\n** Documentation\n\nThe documentation is found at:\n\nhttps://scinim.github.io/Datamancer/\n\nwith a short introduction under:\n\nhttps://scinim.github.io/Datamancer/datamancer.html\n\nand a tutorial for data wrangling with Datamancer available at:\n\nhttps://scinim.github.io/getting-started/basics/data_wrangling.html\n\n** Installation \u0026 dependencies\n\nInstallation should be just a \n#+BEGIN_SRC sh\nnimble install datamancer\n#+END_SRC\naway.\n\n** Backend targets\n\nThe library supports both Nim's C/C++ backends as well as the\nJavaScript target. The latter was added in ~v0.4.2~ and is still\nexperimental. Certain features are currently not supported (reading\nfiles from disk, reading CSVs from a URL).\n\nNote also that on older versions than current devel (as of\n\u003c2024-02-19 Mon 14:47\u003e) including the current stable, in certain\napplications the formula macro ~f{}~ (see below) does not resolve\ntypes in the way it should on the JS backend. You might need to give\nexplicit type hints in more cases than usual (also see below).\n\n** Features and formulas\n\nThe data frame provides the \"5 verbs\" of [[https://dplyr.tidyverse.org/][dplyr]] and more. Main implemented functions:\n- =filter=\n- =mutate=, =transmute=\n- =select=, =rename=\n- =arrange=\n- =summarize=\n- =group_by=\n- =arrange=\n- =inner_join=\n- =set_diff=\n- =count=\n- =bind_rows=\n- =gather=\n- =unique=,\nwhich are all based on the =FormulaNode= object. Basically they all\nreceive =varargs[FormulaNode]=, which is evaluated in context of the\ngiven dataframe.\n\n** Supported types\n\nBy default the =DataFrame= type supports these types as builtins:\n- =float=\n- =int=\n- =string=\n- =bool=\n- =Value= (a variant object that can store either of the above)\n\nAs of version =v0.3.0= very experimental support to extend the\nsupported types has landed. See the [[changelog.org]] for an explanation or\nthe [[playground/non_generic_generics.nim]] example to get an idea.\n\nEssentially any arbitrary type can be used in a DF by extending the\n=Column= type using some macro magic.\n\n** A few words on the =f{}= macro to create formulas\n\nUse:\n- no infix symbol and only code, which does not involve a column in\n  the sense defined below in [[Column access]]:\n  #+BEGIN_SRC nim\n  f{1 + 2}\n  f{\"foo\"}\n  f{true}\n  #+END_SRC\n  a =FormulaNode= of kind =fkVariable=. Stores the values as a =Value=\n  variant object.\n- =\u003c-= for assignment\n  #+BEGIN_SRC nim\n  f{\"newName\" \u003c- \"oldName\"}\n  #+END_SRC\n  a =FormulaNode= of kind =fkAssign=.\n  This does not involve a closure and is just a simple object storing\n  a LHS as a string and the RHS as a =Value= (to also support constant\n  columns via =f{\"constantCol\" \u003c- 5}=).\n  Typically used for =rename= or as an argument for =transmute= and\n  =mutate= to just rename a column or to assign a constant column.\n- =\u003c\u003c= for reduce operations\n  #+BEGIN_SRC nim\n  f{\"meanHwy\" \u003c\u003c mean(`hwy`)}\n  #+END_SRC\n  a =FormulaNode= of kind =fkScalar=.\n  Used only for =summarize= and means we reduce a full column to a\n  single =Value=. This generates a closure, which computes the RHS and\n  assigns it to a result variable of type =Value=. Type hints are\n  required (for now) if only a single proc call is involved on the\n  RHS to tell the macro as what to read the column \"hwy\" and what the\n  result variable is.\n- =~= for vector like proc\n  #+BEGIN_SRC nim\n  f{\"xSquared\" ~ `x` * `x`}\n  #+END_SRC\n  a =FormulaNode= of kind =fkVector=.\n  Used in =mutate=, =transmute= to calculate a full column. This also\n  generates a closure as the reduce operations =\u003c\u003c= does, except here\n  we loop over the length of the DF and access each read tensor via =[idx]=.\n- a formula without any infix symbols will be considered:\n  - =fkVariable= if no column involved\n  - =fkVector= else\n\n*** Column access\nTo access columns in the context of formula, the biggest change\noccured. In the old formula system, a literal string was attempted to\nbe resolved as a DF column dynamically. Since the new formulas are\ncompiled to closures, this would involve overhead and is thus avoided\nfor clearer separation between columns and real strings. This also\nhelps readers of a formula.\n\nThis means:\n- =`columnName`=: accented quotes refer to a DF column. Be careful to\n  only use this for simple letters (no non letter characters or spaces).\n- =c\"columnName\"= : call string literals (by convention use a =c=\n  before the string) are interpreted as a column in the same way as\n  accented quotes, but allow for column names with spaces / non letter\n  characters.\n- =idx(\"columnName\"), idx(`columnName`), idx(nimExpressionReturningString)=:\n  refers to a specific element of the referred column\n- =col(\"columnName\"), col(`columnName`), col(nimExpressionReturningString)=:\n  refers to a the full tensor of the referred column\n- or directly via: =df[nimExpressionReturningString] /\n  df[nimExpressionReturningString][idx]=: to access columns / indices using\n  identifiers / symbols / general expressions that return a string\n  quotes, call string literals or just string literals). This is\n  equivalent to =idx= / =col=, so the latter are preferred.\n\nThe closures take a data frame as an argument, which is named\n=df=. The =df[\"columnName\"]= refers to that argument, although not\nliterally (it is gen'symmed and =df[\"columnName\"]= refers to a\n=Column=). From that column we get the underlying =Tensor=.\n\nIn the context of calling procedures, e.g.:\n#+BEGIN_SRC nim\nf{someProc(`columnName`)}\n#+END_SRC\nit may not be clear whether the procedure is supposed to take the\nwhole tensor as an argument or hand each element of the tensor in a\nloop. Internally the macro tries to determine a suitable call for\neither a scalar or tensor argument. If the called procedure is unique\nthis will likely succeed. In case of heavily overloaded symbols\n(e.g. =max=) it also tries to determine a match from (if any)\nadditional arguments given to that procedure (and uses their types if\nthey are not column references).\n\nIn case at cannot be resolved, you will get an error at compile time\nto specify =idx= (per index access) or =col= (full column access) of the column.\n\nSo for example:\n#+BEGIN_SRC nim\nf{\"asFloat\" ~ parseFloat(idx(\"colName\"))}\n#+END_SRC\nwhere =parseFloat= acts on each element individually. If there is only\na single overload (as in case of =parseFloat=), the input and output\ntypes are inferred automatically to be:\n- read tensor =colName= as a =string=\n- result type is =float=\n\n*** Type information for columns\n\nWhen accessing columns sometimes it may be necessary or desired to\nspecify the type with which a column should be read. To achieve this,\nthe =idx= and =col= helpers mentioned in the previous section can be\ngiven an explicit type:\n#+begin_src nim\nidx(\"foo\", int) # \u003c- reads \"foo\" as a `Tensor[int]`\ncol(\"bar\", float) # \u003c- read \"baz\" as a `Tensor[float]`)\n#+end_src\n\nThese can be mixed and matched to read differently typed columns in\nthe same formula:\n#+begin_src nim\nf{\"foo\" ~ idx(\"x\", int) + idx(\"yStr\", string).parseInt}\n#+end_src\nfor an example.\n\n*** Type hints\n\nTo avoid specifying the types of every input column (as mentioned above\nusing =idx= and =col=) and to force a specific return type, type hints\ncan be used.\n\nType hints are required if the formula macro cannot determine the type\nrequired, either input or output. This is usually the case for\nambiguous operations (overloaded procedures, only a single column\nwithout any operations, etc.). They are of\nthe form:\n- =\u003ctype\u003e: \u003cactualFormula\u003e=: simple type hint for the type of the\n  underlying tensor of the columns involved in the formula.  \n- =\u003ctype\u003e -\u003e \u003cresDtype\u003e: \u003cactualFormula\u003e=: full type for closure.\n  =\u003ctype\u003e= is the dtype used for input tensors, =\u003cresDtype\u003e= the resulting\n  type.\nFor example:\n#+begin_src nim\nf{int -\u003e int: `x` * `y`}\n# ^--- type of the tensors involved on the RHS. Will be read as integers\n#        ^--- type of the resulting tensor\n#+end_src\nIn this case the type would be determined to be float by the macro, so\ntype hints are required in case we need them to be integers.\n\nIn addition to looking at symbols in the scope, there is a step\ninvolving some simple heuristic rules, e.g. if =*=, =/= is involved, it's\nassumed that the input tensors are floats and the output as well. If\n=\u0026= or =$= is involved, it's assumed to be strings. \nFinally if =and= and other logic keywords are used, the result is\nassumed to be =bool= (not the input thought!). \n\n#+BEGIN_SRC nim\n    const floatSet = toSet(@[\"+\", \"-\", \"*\", \"/\", \"mod\"])\n    const stringSet = toSet(@[\"\u0026\", \"$\"])\n    const boolSet = toSet(@[\"and\", \"or\", \"xor\", \"\u003e\", \"\u003c\", \"\u003e=\", \"\u003c=\", \"==\", \"!=\",\n                            \"true\", \"false\", \"in\", \"notin\"])\n#+END_SRC\n\n*** Notes on formula macro internals\n\nFor an insight into the implementation details, ideas and development\nnotes, check out the following document:\n\nhttps://github.com/SciNim/Datamancer/blob/master/notes/formula_dev_notes.org\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscinim%2Fdatamancer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscinim%2Fdatamancer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscinim%2Fdatamancer/lists"}