{"id":13707434,"url":"https://github.com/gdemin/maditr","last_synced_at":"2025-07-22T05:04:28.550Z","repository":{"id":60722838,"uuid":"129561860","full_name":"gdemin/maditr","owner":"gdemin","description":"Fast Data Aggregation, Modification, and Filtering","archived":false,"fork":false,"pushed_at":"2024-11-10T14:39:29.000Z","size":1456,"stargazers_count":61,"open_issues_count":1,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-06-19T00:54:12.672Z","etag":null,"topics":["data-table","magrittr","pipes","r"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gdemin.png","metadata":{"files":{"readme":"README.MD","changelog":"NEWS","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-14T23:26:58.000Z","updated_at":"2024-11-14T03:38:35.000Z","dependencies_parsed_at":"2023-12-28T00:15:27.725Z","dependency_job_id":"83394377-1cf3-41ac-9cfa-6198e9f435ca","html_url":"https://github.com/gdemin/maditr","commit_stats":{"total_commits":187,"total_committers":3,"mean_commits":"62.333333333333336","dds":"0.010695187165775444","last_synced_commit":"ea3ee62fdc9392a2bd335fd0695d491cf34f6449"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/gdemin/maditr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gdemin%2Fmaditr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gdemin%2Fmaditr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gdemin%2Fmaditr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gdemin%2Fmaditr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gdemin","download_url":"https://codeload.github.com/gdemin/maditr/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gdemin%2Fmaditr/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266430672,"owners_count":23927167,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-table","magrittr","pipes","r"],"created_at":"2024-08-02T22:01:31.199Z","updated_at":"2025-07-22T05:04:28.478Z","avatar_url":"https://github.com/gdemin.png","language":"HTML","funding_links":[],"categories":["HTML"],"sub_categories":[],"readme":"# maditr: Fast Data Aggregation, Modification, and Filtering\n\n[![CRAN\\_Status\\_Badge](http://www.r-pkg.org/badges/version/maditr)](https://cran.r-project.org/package=maditr)\n[![](https://cranlogs.r-pkg.org/badges/maditr)](https://cran.rstudio.com/web/packages/maditr/index.html)\n[![](https://cranlogs.r-pkg.org/badges/grand-total/maditr)](https://cran.rstudio.com/web/packages/maditr/index.html)\n[![Coverage Status](https://img.shields.io/codecov/c/github/gdemin/maditr/master.svg)](https://codecov.io/github/gdemin/maditr?branch=master)\n\n### Links\n\n- [maditr on CRAN](https://cran.r-project.org/package=maditr)\n- [maditr on Github](https://github.com/gdemin/maditr)\n- [Issues](https://github.com/gdemin/maditr/issues)\n\n### Installation\n\n`maditr` is on CRAN, so for installation you can print in the console\n`install.packages(\"maditr\")`.\n\n## Overview\n\nPackage provides pipe-style interface for [data.table](https://cran.r-project.org/package=data.table) package. It preserves all data.table features without significant impact on performance. `let` and `take` functions are simplified interfaces for most common data manipulation tasks.\n\n- To select rows from data: `rows(mtcars, am==0)`\n- To select columns from data: `columns(mtcars, mpg, vs:carb)`\n- To aggregate data: `take(mtcars, mean_mpg = mean(mpg), by = am)`\n- To aggregate all non-grouping columns: `take_all(mtcars, mean, by = am)`\n- To aggregate several columns with one summary: `take(mtcars, mpg, hp, fun = mean, by = am)`\n- To get total summary skip `by` argument: `take_all(mtcars, mean)`\n- Use magrittr pipe `%\u003e%` to chain several operations: \n```R\n     mtcars %\u003e%\n        let(mpg_hp = mpg/hp) %\u003e%\n        take(mean(mpg_hp), by = am)\n```\n- To modify variables or add new variables: \n```R\n      mtcars %\u003e%\n         let(new_var = 42,\n             new_var2 = new_var*hp) %\u003e%\n         head()\n```          \n- To drop variable assign NULL: `let(mtcars, am = NULL) %\u003e% head()`\n- To modify all non-grouping variables:\n```R\n    iris %\u003e%\n      let_all(\n          scaled = (.x - mean(.x))/sd(.x),\n          by = Species) %\u003e%\n       head()\n``` \n- To aggregate all variables conditionally on name:\n```R\n    iris %\u003e%\n      take_all(\n          mean = if(startsWith(.name, \"Sepal\")) mean(.x),\n          median = if(startsWith(.name, \"Petal\")) median(.x),\n          by = Species\n      )\n```\n- For parametric assignment use `:=`: \n```R\n    new_var = \"my_var\"\n    old_var = \"mpg\"\n    mtcars %\u003e%\n        let((new_var) := get(old_var)*2) %\u003e%\n        head()\n     \n    # or,  \n    expr = quote(mean(cyl))\n    mtcars %\u003e% \n        let((new_var) := eval(expr)) %\u003e% \n        head()\n    \n    # the same with `take` \n    by_var = \"vs,am\"\n    take(mtcars, (new_var) := eval(expr), by = by_var)\n```      \n\n`query_if` function translates its arguments one-to-one to `[.data.table` method. Additionally there are some conveniences such as automatic `data.frame` conversion to `data.table`.\n\n## vlookup \u0026 xlookup\n\nLet's make datasets for lookups:\n```{r include=FALSE}\nlibrary(maditr)\n```\n\n```{r}\n\nworkers = fread(\"\n    name company\n    Nick Acme\n    John Ajax\n    Daniela Ajax\n\")\n\npositions = fread(\"\n    name position\n    John designer\n    Daniela engineer\n    Cathie manager\n\")\n\n# xlookup\nworkers = let(workers,\n  position = xlookup(name, positions$name, positions$position)\n)\n\n# vlookup\n# by default we search in the first column and return values from second column\nworkers = let(workers,\n  position = vlookup(name, positions, no_match = \"Not found\")\n)\n\n# the same \nworkers = let(workers,\n  position = vlookup(name, positions, \n                     result_column = \"position\", \n                     no_match = \"Not found\") # or, result_column = 2 \n)\n\nhead(workers)\n```\n\n### More examples\n\nWe will use for demonstartion well-known `mtcars` dataset and some examples from `dplyr` package. \n\n```R\nlibrary(maditr)\n\ndata(mtcars)\n\n# Newly created variables are available immediately\nmtcars %\u003e%\n    let(\n        cyl2 = cyl * 2,\n        cyl4 = cyl2 * 2\n    ) %\u003e% head()\n\n# You can also use let() to remove variables and\n# modify existing variables\nmtcars %\u003e%\n    let(\n        mpg = NULL,\n        disp = disp * 0.0163871 # convert to litres\n    ) %\u003e% head()\n\n\n# window functions are useful for grouped computations\nmtcars %\u003e%\n    let(rank = rank(-mpg, ties.method = \"min\"),\n        by = cyl) %\u003e%\n    head()\n\n# You can drop variables by setting them to NULL\nmtcars %\u003e%\n    let(cyl = NULL) %\u003e%\n    head()\n\n# keeps all existing variables\nmtcars %\u003e%\n    let(displ_l = disp / 61.0237) %\u003e%\n    head()\n\n# keeps only the variables you create\nmtcars %\u003e%\n    take(displ_l = disp / 61.0237) %\u003e% \n    head()\n\n\n# can refer to both contextual variables and variable names:\nvar = 100\nmtcars %\u003e%\n    let(cyl = cyl * var) %\u003e%\n    head()\n\n# select rows\nmtcars %\u003e%\n    rows(am==0) %\u003e% \n    head()\n\n# select rows with compound condition\nmtcars %\u003e%\n    rows(am==0 \u0026 mpg\u003emean(mpg))\n\n# select columns\nmtcars %\u003e% \n    columns(vs:carb, cyl)\n    \nmtcars %\u003e% \n    columns(-am, -cyl)    \n\n# regular expression pattern\ncolumns(iris, \"^Petal\") # variables which start from 'Petal'\ncolumns(iris, \"Width$\") # variables which end with 'Width'\n\n# move Species variable to the front\n# pattern \"^.\" matches all variables\ncolumns(iris, Species, \"^.\")\n\n# pattern \"^.*al\" means \"contains 'al'\"\ncolumns(iris, \"^.*al\")\n\n# numeric indexing - all variables except Species\ncolumns(iris, 1:4) \n\n# A 'take' with summary functions applied without 'by' argument returns an aggregated data\nmtcars %\u003e%\n    take(mean = mean(disp), n = .N)\n\n# Usually, you'll want to group first\nmtcars %\u003e%\n    take(mean = mean(disp), n = .N, by = am)\n\n# grouping by multiple variables\nmtcars %\u003e%\n    take(mean = mean(disp), n = .N, by = list(am, vs))\n\n# You can group by expressions:\nmtcars %\u003e%\n    take_all(\n        mean,\n        by = list(vsam = vs + am)\n    )\n\n# modify all non-grouping variables in-place\nmtcars %\u003e%\n    let_all((.x - mean(.x))/sd(.x), by = am) %\u003e%\n    head()\n\n# modify all non-grouping variables to new variables\nmtcars %\u003e%\n    let_all(scaled = (.x - mean(.x))/sd(.x), by = am) %\u003e%\n    head()\n\n# conditionally modify all variables\niris %\u003e%\n    let_all(mean = if(is.numeric(.x)) mean(.x)) %\u003e%\n    head()\n\n# modify all variables conditionally on name\niris %\u003e%\n    let_all(\n        mean = if(startsWith(.name, \"Sepal\")) mean(.x),\n        median = if(startsWith(.name, \"Petal\")) median(.x),\n        by = Species\n    ) %\u003e%\n    head()\n\n# aggregation with 'take_all'\nmtcars %\u003e%\n    take_all(mean = mean(.x), sd = sd(.x), n = .N, by = am)\n\n# conditionally aggregate all variables\niris %\u003e%\n    take_all(mean = if(is.numeric(.x)) mean(.x))\n\n# aggregate all variables conditionally on name\niris %\u003e%\n    take_all(\n        mean = if(startsWith(.name, \"Sepal\")) mean(.x),\n        median = if(startsWith(.name, \"Petal\")) median(.x),\n        by = Species\n    )\n\n# parametric evaluation:\nvar = quote(mean(cyl))\nmtcars %\u003e% \n    let(mean_cyl = eval(var)) %\u003e% \n    head()\ntake(mtcars, eval(var))\n\n# all together\nnew_var = \"mean_cyl\"\nmtcars %\u003e% \n    let((new_var) := eval(var)) %\u003e% \n    head()\ntake(mtcars, (new_var) := eval(var))\n\n```\n\n## Variable selection in the expressions\n\nYou can use 'columns' inside expression in the 'take'/'let'. 'columns' will\nbe replaced with data.table with selected columns. In 'let' in the\nexpressions with ':=', 'cols' or '%to%' can be placed in the left part of the\nexpression. It is usefull for multiple assignment.\nThere are four ways of column selection:\n\n1. Simply by column names\n2. By variable ranges, e. g. vs:carb. Alternatively, you can use '%to%'\ninstead of colon: 'vs %to% carb'.\n3. With regular expressions. Characters which start with '^' or end with $\nconsidered as Perl-style regular expression patterns. For example, '^Petal'\nreturns all variables started with 'Petal'. 'Width$' returns all variables\nwhich end with 'Width'. Pattern '^.' matches all variables and pattern\n'^.*my_str' is equivalent to contains \"my_str\"'.\n4. By character variables with interpolated parts. Expression in the curly\nbrackets inside characters will be evaluated in the parent frame with\n'text_expand' function. For example, `a{1:3}` will be transformed to the names 'a1',\n'a2', 'a3'. 'cols' is just a shortcut for 'columns'.\n\n```R\n# range selection\niris %\u003e% \n    let(\n        avg = rowMeans(Sepal.Length %to% Petal.Width)\n    ) %\u003e% \n    head()\n\n# multiassignment\niris %\u003e% \n    let(\n        # starts with Sepal or Petal\n        multipled1 %to% multipled4 := cols(\"^(Sepal|Petal)\")*2\n    ) %\u003e% \n    head()\n\n\nmtcars %\u003e% \n    let(\n        # text expansion\n        cols(\"scaled_{names(mtcars)}\") := lapply(cols(\"{names(mtcars)}\"), scale)\n    ) %\u003e% \n    head()\n\n# range selection in 'by'\n# selection of range + additional column\nmtcars %\u003e% \n    take(\n        res = sum(cols(mpg, disp %to% drat)),\n        by = vs %to% gear\n    )\n```\n\n## Joins\n\nHere we use the same datasets as with lookups:\n\n```R\nworkers = fread(\"\n    name company\n    Nick Acme\n    John Ajax\n    Daniela Ajax\n\")\n\npositions = fread(\"\n    name position\n    John designer\n    Daniela engineer\n    Cathie manager\n\")\n\nworkers\npositions\n```\n\nDifferent kinds of joins:\n\n```R\nworkers %\u003e% dt_inner_join(positions)\nworkers %\u003e% dt_left_join(positions)\nworkers %\u003e% dt_right_join(positions)\nworkers %\u003e% dt_full_join(positions)\n\n# filtering joins\nworkers %\u003e% dt_anti_join(positions)\nworkers %\u003e% dt_semi_join(positions)\n```\n\nTo suppress the message, supply `by` argument:\n```R\nworkers %\u003e% dt_left_join(positions, by = \"name\")\n```\n\nUse a named `by` if the join variables have different names:\n```R\npositions2 = setNames(positions, c(\"worker\", \"position\")) # rename first column in 'positions'\nworkers %\u003e% dt_inner_join(positions2, by = c(\"name\" = \"worker\"))\n```\n\n## 'dplyr'-like interface for data.table.\n\nThere are a small subset of 'dplyr' verbs to work with data.table. Note that there is no `group_by`\nverb - use `by` or `keyby` argument when needed.\n\n- `dt_mutate` adds new variables or modify existing variables. If data is data.table then it modifies in-place.\n- `dt_summarize` computes summary statistics. Splits the data into subsets, computes summary statistics for each, and returns the result in the \"data.table\" form.\n- `dt_summarize_all` the same as `dt_summarize` but work over all non-grouping variables.\n- `dt_filter` Selects rows/cases where conditions are true. Rows where the condition evaluates to NA are dropped.\n- `dt_select` Selects column/variables from the data set. Range of variables are supported, e. g. `vs:carb`. Characters which start with `^` or end with `\\$` considered as Perl-style regular expression patterns. For example, `'^Petal'`\nreturns all variables started with 'Petal'. `'Width\\$'` returns all variables which end with 'Width'. Pattern `^.` matches all variables and pattern `'^.*my_str'` is equivalent to contains `\"my_str\"`. See examples.\n\n```R\n# examples from 'dplyr'\n# newly created variables are available immediately\nmtcars  %\u003e%\n    dt_mutate(\n        cyl2 = cyl * 2,\n        cyl4 = cyl2 * 2\n    ) %\u003e%\n    head()\n\n\n# you can also use dt_mutate() to remove variables and\n# modify existing variables\nmtcars %\u003e%\n    dt_mutate(\n        mpg = NULL,\n        disp = disp * 0.0163871 # convert to litres\n    ) %\u003e%\n    head()\n\n\n# window functions are useful for grouped mutates\nmtcars %\u003e%\n    dt_mutate(\n        rank = rank(-mpg, ties.method = \"min\"),\n        keyby = cyl) %\u003e%\n    print()\n\n\n# You can drop variables by setting them to NULL\nmtcars %\u003e% dt_mutate(cyl = NULL) %\u003e% head()\n\n# A summary applied without by returns a single row\nmtcars %\u003e%\n    dt_summarise(mean = mean(disp), n = .N)\n\n# Usually, you'll want to group first\nmtcars %\u003e%\n    dt_summarise(mean = mean(disp), n = .N, by = cyl)\n\n\n# Multiple 'by' - variables\nmtcars %\u003e%\n    dt_summarise(cyl_n = .N, by = list(cyl, vs))\n\n# Newly created summaries immediately\n# doesn't overwrite existing variables\nmtcars %\u003e%\n    dt_summarise(disp = mean(disp),\n                  sd = sd(disp),\n                  by = cyl)\n\n# You can group by expressions:\nmtcars %\u003e%\n    dt_summarise_all(mean, by = list(vsam = vs + am))\n\n# filter by condition\nmtcars %\u003e%\n    dt_filter(am==0)\n\n# filter by compound condition\nmtcars %\u003e%\n    dt_filter(am==0,  mpg\u003emean(mpg))\n\n\n# select\nmtcars %\u003e% dt_select(vs:carb, cyl)\nmtcars %\u003e% dt_select(-am, -cyl)\n\n# regular expression pattern\ndt_select(iris, \"^Petal\") # variables which start from 'Petal'\ndt_select(iris, \"Width$\") # variables which end with 'Width'\n# move Species variable to the front\n# pattern \"^.\" matches all variables\ndt_select(iris, Species, \"^.\")\n# pattern \"^.*al\" means \"contains 'al'\"\ndt_select(iris, \"^.*al\")\ndt_select(iris, 1:4) # numeric indexing - all variables except Species\n\n```\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgdemin%2Fmaditr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgdemin%2Fmaditr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgdemin%2Fmaditr/lists"}