{"id":16516819,"url":"https://github.com/adamspannbauer/datatable_and_dplyr","last_synced_at":"2026-03-18T20:09:56.454Z","repository":{"id":83432072,"uuid":"91441602","full_name":"AdamSpannbauer/datatable_and_dplyr","owner":"AdamSpannbauer","description":"Comparison of operations done in `dplyr` and `data.table`","archived":false,"fork":false,"pushed_at":"2017-05-16T13:58:05.000Z","size":7127,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-13T05:28:58.954Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AdamSpannbauer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-16T09:39:43.000Z","updated_at":"2022-12-28T03:43:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"2bb01bbf-07b9-4d9c-8f3b-554b0d2eba38","html_url":"https://github.com/AdamSpannbauer/datatable_and_dplyr","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdamSpannbauer%2Fdatatable_and_dplyr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdamSpannbauer%2Fdatatable_and_dplyr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdamSpannbauer%2Fdatatable_and_dplyr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AdamSpannbauer%2Fdatatable_and_dplyr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AdamSpannbauer","download_url":"https://codeload.github.com/AdamSpannbauer/datatable_and_dplyr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241549117,"owners_count":19980476,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T16:27:00.902Z","updated_at":"2026-02-01T00:03:06.681Z","avatar_url":"https://github.com/AdamSpannbauer.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"``` r\nlibrary(data.table)\nlibrary(tidyverse)\n```\n\n#### resources\n\nfree intro to data manipulation with `dplyr` and `data.table` through [datacamp](https://www.datacamp.com/home) (more than intro costs $)\n\n-   [dplyr](https://www.datacamp.com/courses/dplyr-data-manipulation-r-tutorial)\n-   [data.table](https://campus.datacamp.com/courses/data-table-data-manipulation-r-tutorial)\n\nCreating/Inspecting dataframes\n==============================\n\nCreating a dataframe in `dplyr` or `data.table` is very similiar to creating a dataframe in base `R`. An advantage of dataframe creation for both packages is that `stringAsFactors` defaults to `FALSE`.\n\nExamining the structure of both df/dt we can start to see some differences. The `dplyr` dataframe has classes `tbl_df`, `tbl`, \u0026 `data.frame`. The `data.table` has classes `data.table` \u0026 `data.frame`. They both have class `data.frame` so we'll be able to call functions written for the base `R` class.\n\nThe last difference we can see in the sturcture is that `data.table` has a pointer attribute named `.internal.selfref`. This attribute will allow us to modify the `data.table` by reference for some operations and avoid copy-on-modify.\n\nLastly, we can print the two dataframes and see that they both have console friendly print methods for large datasets.\n\n``` r\nlen      \u003c- 1e5\ncol_inds \u003c- 1:len\ncol_grps \u003c- sample(letters[1:3], len, replace=TRUE)\ncol_vals \u003c- rnorm(len)\n\n#df creation-----------------------------------------\ndf \u003c- dplyr::data_frame(ind = col_inds,\n                        grp = col_grps,\n                        val = col_vals)\n\ndt \u003c- data.table::data.table(ind = col_inds,\n                             grp = col_grps,\n                             val = col_vals)\n#df structure-----------------------------------------\nstr(df)\n```\n\n    ## Classes 'tbl_df', 'tbl' and 'data.frame':    100000 obs. of  3 variables:\n    ##  $ ind: int  1 2 3 4 5 6 7 8 9 10 ...\n    ##  $ grp: chr  \"b\" \"a\" \"c\" \"b\" ...\n    ##  $ val: num  -0.5453 -1.4466 0.0808 -0.6143 -1.5132 ...\n\n``` r\nstr(dt)\n```\n\n    ## Classes 'data.table' and 'data.frame':   100000 obs. of  3 variables:\n    ##  $ ind: int  1 2 3 4 5 6 7 8 9 10 ...\n    ##  $ grp: chr  \"b\" \"a\" \"c\" \"b\" ...\n    ##  $ val: num  -0.5453 -1.4466 0.0808 -0.6143 -1.5132 ...\n    ##  - attr(*, \".internal.selfref\")=\u003cexternalptr\u003e\n\n``` r\n#print methods----------------------------------------\ndf\n```\n\n    ## # A tibble: 100,000 × 3\n    ##      ind   grp         val\n    ##    \u003cint\u003e \u003cchr\u003e       \u003cdbl\u003e\n    ## 1      1     b -0.54528051\n    ## 2      2     a -1.44659698\n    ## 3      3     c  0.08082766\n    ## 4      4     b -0.61431850\n    ## 5      5     c -1.51320799\n    ## 6      6     b -0.32294980\n    ## 7      7     c  0.40916513\n    ## 8      8     c  0.74801291\n    ## 9      9     b  0.62383372\n    ## 10    10     a  0.19813085\n    ## # ... with 99,990 more rows\n\n``` r\ndt\n```\n\n    ##            ind grp         val\n    ##      1:      1   b -0.54528051\n    ##      2:      2   a -1.44659698\n    ##      3:      3   c  0.08082766\n    ##      4:      4   b -0.61431850\n    ##      5:      5   c -1.51320799\n    ##     ---                       \n    ##  99996:  99996   b  0.38383444\n    ##  99997:  99997   c  0.63775732\n    ##  99998:  99998   a  1.58754273\n    ##  99999:  99999   c -1.76046752\n    ## 100000: 100000   a -0.36833525\n\nWriting/Reading csvs\n====================\n\n**note: these timings were performed on windows 64bit (8core, 32gb ram); `readr::write_csv` performs much better writing this 100000 row df on a non windows machine**\n\nWriting\n-------\n\nAgain, the syntax for writing csvs in both frameworks is very similiar to writing a csv in base. An advantage of both over base, in my opinion, is the omission of writing row.names by default. The `fwrite` function has the ability to set `row.names=TRUE` while the `readr` implementation does not have an argument for rownames.\n\n`fwrite` stands for 'fast write' and it lives up to the hype. In our example 100,000 rows the `readr` implentation is decidely slower (16375% slower for this knit). From `?fwrite` documentation: 'Modern machines almost surely have more than one CPU so fwrite uses them.'\n\n``` r\nsystem.time(readr::write_csv(df, \"readr_out.csv\"))\n```\n\n    ##    user  system elapsed \n    ##    0.15    0.82    6.55\n\n``` r\nsystem.time(data.table::fwrite(dt, \"fwrite_out.csv\"))\n```\n\n    ##    user  system elapsed \n    ##    0.02    0.00    0.04\n\nReading\n-------\n\nJust for fun we'll switch read in the file that the opposing package wrote out (WoOoOoOoO!)\n\nWith the current size of our data the read times for `readr` and `data.table` are very similiar; both are typically executing in under a second on my machine. In both functions we can specify the column classes. Using `readr` we can specify the `col_types` using shorthand or full names.\n\nBoth functions, of course, read the csv into the two different structures that we saw above.\n\nThis example doesn't show it, but `fread` scales to much larger data sets better than `readr`. On a 40 million row x 3 column data set `readr` completed the read in a little over 2.5 minutes, while `fread` completed the job in 14 seconds.\n\n``` r\nsystem.time(readr::read_csv(\"fwrite_out.csv\", col_types = \"icn\"))\n```\n\n    ##    user  system elapsed \n    ##    0.02    0.03    0.14\n\n``` r\nsystem.time(data.table::fread(\"readr_out.csv\", colClasses = c(\"integer\", \"character\", \"numeric\")))\n```\n\n    ##    user  system elapsed \n    ##    0.10    0.00    0.17\n\nData Manipulation\n=================\n\nThis is mostly going to be a collection of example syntax for performing operations in the different frameworks. Commentary on the code chunks will be limited.\n\n*(note: this doc is geared towards `dplyr` users that are less familiar with `data.table`.)*\n\n### data.table syntax intro\n\nOperations in `data.table` primarily use `[`. In base `[` typically is used for subsetting and given just 2 arguments when used with a dataframe: rows (`i`) and columns (`j`). When used with a `data.table` the brackets assume new functionality. The `[` still take `i` \u0026 `j` like arguments but a 3rd argument (`by`) is now assumed to be a grouping variable (`dt[i, j, by]`). Other differences between `data.table` and include: ability to reference column names without `df$` syntax increased computational functionality in the `j` argument.\n\nFiltering\n---------\n\n``` r\n#dplyr\ndf %\u003e% \n  filter(grp == \"a\")\n\n#data.table\ndt[grp==\"a\",]\n\n#base\ndf[df$grp==\"a\",]\n```\n\nSorting\n-------\n\nSort by `grp`.\n\n``` r\n#dplyr\ndf %\u003e% \n  arrange(grp)\n\n#data.table\ndt[order(grp),]\n#setkey(dt, grp)\n\n#base\ndf[order(df$grp),]\n```\n\nCreating/Storing/Deleting a column\n==================================\n\nCreate \u0026 store new column\n-------------------------\n\nHere we see a few new items in the `data.table` methodology.\n\nIn `dplyr` in order to create a new column and store the result we will have to create the column and then copy our data.frame to a new address for storage (this move can be seen by the change in `address(df)`). However, the operator `:=` from `data.table` creates the new column by reference so we do not copy our table to a new address; this can be a big performance boost for large datasets.\n\nThe next thing we see in `data.table` is the introduction of `.N`, which evaluates to `nrow(dt)` when called from `` `[.data.table` ``. In the `dplyr` pipeline we get the number of rows using the `.` sytnax associated with `` `%\u003e%` ``.\n\n``` r\n#dplyr\naddress(df) #\"00000000691018D0\"\ndf \u003c- df %\u003e% \n  mutate(new_col = runif(nrow(.)))\naddress(df) #\"000000002A3825A8\"\n\n#data.table\naddress(dt) #\"00000000117125C0\"\ndt[,new_col := runif(.N)]\naddress(dt) #\"00000000117125C0\"\n#dt[,.(ind, grp, val, new_col = runif(.N))]\n```\n\nDelete column\n-------------\n\n``` r\ndf \u003c- df %\u003e% \n  select(-new_col)\n\ndt[,new_col := NULL]\n```\n\nCreate column without storing\n-----------------------------\n\nSome new shorthand is again introced in the `data.table` syntax used here. The `.SD` references all columns not included in the grouping `by` argument of `` `[.data.table` ``. This evaluates to a list of columns (i.e. a dataframe), and we can add columns by `c`ombining a new list of columns to `.SD`. If `j` evaluates to a list of equal lenght columns `` `[.data.table` `` will interpret it as a `data.table`.\n\n``` r\ndf %\u003e% \n  mutate(new_col = runif(nrow(.)))\n```\n\n    ## # A tibble: 100,000 × 4\n    ##      ind   grp         val   new_col\n    ##    \u003cint\u003e \u003cchr\u003e       \u003cdbl\u003e     \u003cdbl\u003e\n    ## 1      1     b -0.54528051 0.4302584\n    ## 2      2     a -1.44659698 0.6446282\n    ## 3      3     c  0.08082766 0.5114203\n    ## 4      4     b -0.61431850 0.9204234\n    ## 5      5     c -1.51320799 0.6493419\n    ## 6      6     b -0.32294980 0.5688315\n    ## 7      7     c  0.40916513 0.1372726\n    ## 8      8     c  0.74801291 0.8770839\n    ## 9      9     b  0.62383372 0.7419305\n    ## 10    10     a  0.19813085 0.2833246\n    ## # ... with 99,990 more rows\n\n``` r\ndt[,c(.SD, list(newnew = runif(.N)))]\n```\n\n    ##            ind grp         val     newnew\n    ##      1:      1   b -0.54528051 0.54285282\n    ##      2:      2   a -1.44659698 0.96687137\n    ##      3:      3   c  0.08082766 0.94399398\n    ##      4:      4   b -0.61431850 0.06784121\n    ##      5:      5   c -1.51320799 0.99183871\n    ##     ---                                  \n    ##  99996:  99996   b  0.38383444 0.20356066\n    ##  99997:  99997   c  0.63775732 0.22234155\n    ##  99998:  99998   a  1.58754273 0.74215794\n    ##  99999:  99999   c -1.76046752 0.13666365\n    ## 100000: 100000   a -0.36833525 0.95060147\n\nSummarising and Grouping\n========================\n\n*note: `dplyr` applies sorting by the grouping variable when summarising; `data.table` orders the summarization by the first appearance of each distinct value in the grouping variable*\n\n``` r\n#using default naming\ndf %\u003e% \n  group_by(grp) %\u003e% \n  summarise(mean(val))\n```\n\n    ## # A tibble: 3 × 2\n    ##     grp   `mean(val)`\n    ##   \u003cchr\u003e         \u003cdbl\u003e\n    ## 1     a  0.0002478248\n    ## 2     b -0.0086373767\n    ## 3     c  0.0016920526\n\n``` r\n#using default naming\ndt[,mean(val), by=grp]\n```\n\n    ##    grp            V1\n    ## 1:   b -0.0086373767\n    ## 2:   a  0.0002478248\n    ## 3:   c  0.0016920526\n\n``` r\n#using custom naming\ndf %\u003e% \n  group_by(grp) %\u003e% \n  summarise(my_mean = mean(val))\n```\n\n    ## # A tibble: 3 × 2\n    ##     grp       my_mean\n    ##   \u003cchr\u003e         \u003cdbl\u003e\n    ## 1     a  0.0002478248\n    ## 2     b -0.0086373767\n    ## 3     c  0.0016920526\n\n``` r\n#using custom naming\ndt[,.(my_mean = mean(val)), by=grp]\n```\n\n    ##    grp       my_mean\n    ## 1:   b -0.0086373767\n    ## 2:   a  0.0002478248\n    ## 3:   c  0.0016920526\n\nJoins\n=====\n\nLeft join\n---------\n\n`dplyr` has very descriptive `?join` functions, while `data.table` can use short hand of `dt[dt2]` or `merge`. The `merge` syntax is similiar the base `merge`.\n\n``` r\ndf2 \u003c- df\ndf %\u003e% \n  left_join(df2, by=\"ind\")\n\ndt2 \u003c- dt\nsetkey(dt, ind)\nsetkey(dt2, ind)\n\ndt[dt2]\nmerge(dt, dt2, by = \"ind\", all.x = TRUE)\n```\n\nTransforming columns\n====================\n\n``` r\ndf %\u003e% \n  mutate_all(as.character)\n```\n\n    ## # A tibble: 100,000 × 3\n    ##      ind   grp                val\n    ##    \u003cchr\u003e \u003cchr\u003e              \u003cchr\u003e\n    ## 1      1     b -0.545280508170098\n    ## 2      2     a  -1.44659697798738\n    ## 3      3     c 0.0808276614999396\n    ## 4      4     b -0.614318501663225\n    ## 5      5     c  -1.51320799247204\n    ## 6      6     b -0.322949797444465\n    ## 7      7     c  0.409165129048615\n    ## 8      8     c   0.74801290811267\n    ## 9      9     b  0.623833723477989\n    ## 10    10     a  0.198130852264119\n    ## # ... with 99,990 more rows\n\n``` r\ndt[,lapply(.SD, as.character)]\n```\n\n    ##            ind grp                val\n    ##      1:      1   b -0.545280508170098\n    ##      2:      2   a  -1.44659697798738\n    ##      3:      3   c 0.0808276614999396\n    ##      4:      4   b -0.614318501663225\n    ##      5:      5   c  -1.51320799247204\n    ##     ---                              \n    ##  99996:  99996   b  0.383834442934876\n    ##  99997:  99997   c  0.637757323905886\n    ##  99998:  99998   a   1.58754272774486\n    ##  99999:  99999   c  -1.76046752347906\n    ## 100000: 100000   a -0.368335246887932\n\nAdditional stuff\n================\n\n``` r\n# count per group\ndf %\u003e% count(grp)\n```\n\n    ## # A tibble: 3 × 2\n    ##     grp     n\n    ##   \u003cchr\u003e \u003cint\u003e\n    ## 1     a 33570\n    ## 2     b 33386\n    ## 3     c 33044\n\n``` r\ndt[,.N, grp]\n```\n\n    ##    grp     N\n    ## 1:   b 33386\n    ## 2:   a 33570\n    ## 3:   c 33044\n\n``` r\n#eval operation to vector\ndt[,sum(val)]\n```\n\n    ## [1] -224.1358\n\n``` r\n#eval operation to vector where \ndt[grp==\"b\", sum(val)]\n```\n\n    ## [1] -288.3675\n\n``` r\n#referencing rows by key value\nsetkey(dt, grp)\ndt[.(\"b\"), sum(val)]\n```\n\n    ## [1] -288.3675\n\n[last minute example thrown in](http://stackoverflow.com/questions/43957195/linear-interpolation-by-group-in-r/43957539#43957539)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadamspannbauer%2Fdatatable_and_dplyr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadamspannbauer%2Fdatatable_and_dplyr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadamspannbauer%2Fdatatable_and_dplyr/lists"}