{"id":27162003,"url":"https://github.com/moj-analytical-services/intro_r_training_extension","last_synced_at":"2025-04-09T00:58:44.115Z","repository":{"id":84738809,"uuid":"271569981","full_name":"moj-analytical-services/intro_r_training_extension","owner":"moj-analytical-services","description":"An extension to the IntroRTraining course","archived":false,"fork":false,"pushed_at":"2024-12-12T15:13:56.000Z","size":2181,"stargazers_count":4,"open_issues_count":6,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-12-12T16:24:08.069Z","etag":null,"topics":["intro","introduction","r","training"],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/moj-analytical-services.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-11T14:37:42.000Z","updated_at":"2024-12-02T10:15:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"8faa3484-3994-41a4-817d-5c9b0d2f476b","html_url":"https://github.com/moj-analytical-services/intro_r_training_extension","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moj-analytical-services%2Fintro_r_training_extension","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moj-analytical-services%2Fintro_r_training_extension/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moj-analytical-services%2Fintro_r_training_extension/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moj-analytical-services%2Fintro_r_training_extension/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/moj-analytical-services","download_url":"https://codeload.github.com/moj-analytical-services/intro_r_training_extension/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247953051,"owners_count":21023947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["intro","introduction","r","training"],"created_at":"2025-04-09T00:58:43.320Z","updated_at":"2025-04-09T00:58:44.106Z","avatar_url":"https://github.com/moj-analytical-services.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"Introduction to R extension\n================\n\nThis repository is for the Introduction to R+ course offered by the Data\n\u0026 Analysis R Training Group.\n\nThe session is periodically run over Teams, and is intended to be\naccessible to anyone who is familiar with the content of the\n[Introduction to\nR](https://github.com/moj-analytical-services/IntroRTraining) training\ncourse. Alternatively, you can go through this material in your own\ntime - all the notes are available below and you can also find links to\nrecordings of previous sessions\n[here](https://moj-analytical-services.github.io/ap-tools-training/ITG.html#ITG).\nIf you work through the material by yourself please leave feedback about\nthe material [here](https://airtable.com/shr9u2OJB2pW8Y0Af).\n\n## Contents\n\n- [Pre-material](#pre-material)\n- [Learning outcomes](#learning-outcomes)\n- [Conditional statements](#conditional-statements)\n- [Iteration](#iteration)\n- [Handling missing data](#handling-missing-data)\n- [Reshaping data](#reshaping-data)\n- [String manipulation](#string-manipulation)\n- [Further reading](#further-reading)\n- [Bonus examples](#bonus-examples)\n- [Appendix](#appendix)\n\n## Pre-material\n\nBefore the session, please make sure that -\n\n1.  You have access to RStudio on the Analytical Platform\n2.  You have access to the [alpha-r-training s3\n    bucket](https://controlpanel.services.analytical-platform.service.justice.gov.uk/datasources/607/)\n3.  You have followed the steps in the [Configure Git and Github section\n    of the Platform User\n    Guidance](https://user-guidance.services.alpha.mojanalytics.xyz/github.html#setup-github-keys-to-access-it-from-r-studio-and-jupyter)\n    to configure Git and GitHub (this only needs doing once)\n4.  You have cloned this repository (instructions are in the Analytical\n    Platform User Guidance if you follow step 1\n    [here](https://user-guidance.services.alpha.mojanalytics.xyz/github.html#r-studio))\n5.  You set up {renv} and install the required packages using the following steps (in the Console window in RStudio):\n       * Run `install.packages(\"renv\")`.\n       * Run `renv::restore()`. You will be informed that the project is not activated and given several options. Select the option to `Activate the project and use the project library` (exact wording may differ). The set-up will take some time.\n       * Run `renv::restore()` again to install the required packages responding `y` when prompted.\n\nIf you have any problems with the above please get in touch with the\ncourse organisers or ask for help on either the\n\\#analytical-platform-support or \\#intro_r channel on [ASD\nslack](https://asdslack.slack.com).\n\nAll the examples in the presentation and README are available in the R\nscript example_code.R.\n\n# Introduction\n\n## Introduction\n\nThis course builds on the original [Introduction to R training\ncourse](https://github.com/moj-analytical-services/IntroRTraining), and\ncovers additional programming concepts. It provides examples that\ndemonstrate how the Tidyverse packages can assist with tasks typically\nencountered in MoJ Data \u0026 Analysis.\n\nDevelopment of the Tidyverse suite of packages was led by Hadley\nWickham, and more information about these packages can be found on the\n[Tidyverse website](https://www.tidyverse.org/) as well as in the book\n[R for Data Science](https://r4ds.had.co.nz/).\n\nThe first two chapters of this course cover two fundamentals of\nprogramming in R: conditional statements and loops. These two topics\ncome under the umbrella of ‘control flow’, which refers to how we can\nchange the order that pieces of code are run in. With conditional\nstatements we can introduce choices, where different pieces of code are\nrun depending on the input, and loops allow us to repeatedly run the\nsame piece of code.\n\n## Learning outcomes\n\n### By the end of this session you should know how to:\n\n- Change what the code does based on a condition\n- Classify values in a dataframe, based on a set of conditions\n- Read and combine data from multiple csv files\n- Easily apply a function to multiple columns in a dataframe\n- Deal with missing values in a dataframe\n- Reshape dataframes\n- Search for a string pattern in a dataframe\n\n## Before we start\n\nTo follow along with the code and participate in the exercises, open the\nscript “example_code.R” in RStudio. All the code that we’ll show in this\nsession is stored in “example_code.R”, and you can edit this script to\nwrite solutions to the exercises. You may also want to have the course\n[README](https://github.com/moj-analytical-services/intro_r_training_extension)\nopen as a reference.\n\nFirst, we need to load a few packages:\n\n``` r\n# Load packages\nlibrary(Rs3tools) # Used to help R interact with s3 cloud storage\nlibrary(dplyr) # Used for data manipulation\nlibrary(tidyr) # Used to help reshape and deal with missing data\nlibrary(stringr) # Used for string manipulation\nlibrary(readr) # Used to help read in data\n```\n\n# Conditional statements\n\n## if statements\n\nConditional statements can be used when you want a piece of code to be\nexecuted only if a particular condition is met. The most basic form of\nthese are ‘if’ statements. As a simple example, let’s say we wanted to\ncheck if a variable `x` is less than 10. We can write something like:\n\n``` r\nx \u003c- 9\n\n# A basic if statement\nif (x \u003c 10) {\n  print(\"x is less than 10\")\n}\n```\n\n    ## [1] \"x is less than 10\"\n\n``` r\nx \u003c- 11\n\nif (x \u003c 10) {\n  print(\"x is less than 10\")\n}\n```\n\n## if…else statements\n\nWe can also specify if we want something different to happen if the\ncondition is not met, using an ‘if…else’ statement:\n\n``` r\nx \u003c- 11\n\n# A basic if...else statement\nif (x \u003c 10) {\n  print(\"x is less than 10\")\n} else {\n  print(\"x is 10 or greater\")\n}\n```\n\n    ## [1] \"x is 10 or greater\"\n\n------------------------------------------------------------------------\n\nOr if there are multiple conditions where we want different things to\nhappen, we can add ‘else if’ commands:\n\n``` r\nx \u003c- 5\n\nif (x \u003c 10) {\n  print(\"x is less than 10\")\n} else if (x == 10) {\n  print(\"x is equal to 10\")\n} else {\n  print(\"x is greater than 10\")\n}\n```\n\n    ## [1] \"x is less than 10\"\n\n------------------------------------------------------------------------\n\nFor the conditions themselves, we can make use of R’s relational and\nlogical operators:\n\n| Operator | Definition                      |\n|:--------:|:--------------------------------|\n|    ==    | Equal to                        |\n|    !=    | Not equal to                    |\n|    \\\u003e    | Greater than                    |\n|    \\\u003c    | Less than                       |\n|   \\\u003e=    | Greater than or equal to        |\n|   \\\u003c=    | Less than or equal to           |\n|    ǀ     | Or                              |\n|    \u0026     | And                             |\n|    !     | Not                             |\n|   %in%   | The subject appears in a vector |\n| is.na()  | The subject is NA               |\n\n## Vectorising an if…else statement\n\nDplyr’s `if_else()` function is useful if we want to apply an ‘if…else’\nstatement to a vector, rather than a single value. When we use\n`if_else()` we need to provide it with three arguments, like this:\n`if_else(condition, true, false)`, where `condition` is the condition we\nwant to test, `true` is the value to use if the condition evaluates to\n`TRUE`, and `false` is the value to use if the condition evaluates to\n`FALSE`.\n\nFor example, if we had a vector containing a set of numbers, and we\nwanted to create an equivalent vector containing a ‘1’ if the number is\ngreater than zero, or a ‘0’ if the number is less than or equal to zero,\nthen we could do:\n\n``` r\nx \u003c- c(0, 74, 0, 8, 23, 15, 3, 0, -1, 9)\n\n# Vectorised if...else\ndplyr::if_else(x \u003e 0, 1, 0)\n```\n\n    ##  [1] 0 1 0 1 1 1 1 0 0 1\n\n------------------------------------------------------------------------\n\nWhen we’re manipulating dataframes, it can be useful to combine\n`if_else()` with the `mutate()` function from dplyr. Let’s take a look\nat the `offenders` dataframe, which is also used in the [Introduction to\nR](https://github.com/moj-analytical-services/IntroRTraining) course:\n\n``` r\n# First read and preview the data\noffenders \u003c- Rs3tools::s3_path_to_full_df(\n  \"alpha-r-training/intro-r-training/Offenders_Chicago_Police_Dept_Main.csv\"\n)\nstr(offenders)\n```\n\n    ## 'data.frame':    1413 obs. of  11 variables:\n    ##  $ LAST            : chr  \"RODRIGUEZ\" \"MARTINEZ\" \"GARCIA\" \"RODRIGUEZ\" ...\n    ##  $ FIRST           : chr  \"JUAN\" \"MOISES\" \"ELLIOTT\" \"JOSE\" ...\n    ##  $ BLOCK           : chr  \"009XX W CUYLER AVE\" \"011XX N KILBOURN AVE\" \"011XX W 18TH ST\" \"012XX W RACE AVE\" ...\n    ##  $ GENDER          : chr  \"MALE\" \"MALE\" \"MALE\" \"MALE\" ...\n    ##  $ REGION          : chr  \"West\" \"East\" \"South\" \"North\" ...\n    ##  $ BIRTH_DATE      : chr  \"06/22/1955\" \"02/07/1954\" \"08/11/1970\" \"02/10/1959\" ...\n    ##  $ HEIGHT          : int  198 198 201 237 201 199 201 236 198 199 ...\n    ##  $ WEIGHT          : int  190 180 200 195 220 130 200 235 140 130 ...\n    ##  $ PREV_CONVICTIONS: num  0 0 0 0 0 0 0 0 0 0 ...\n    ##  $ SENTENCE        : chr  \"Court_order\" \"Prison_\u003c12m\" \"Court_order\" \"Court_order\" ...\n    ##  $ AGE             : int  58 59 43 54 34 50 32 43 34 18 ...\n\n------------------------------------------------------------------------\n\nLet’s say we wanted a simple way to be able to separate youths from\nadult offenders. We can add a column that contains ‘Youth’ if the\noffender is under the age of 18, and ‘Adult’ otherwise:\n\n``` r\n# Now use mutate to add the new column\noffenders \u003c- offenders %\u003e%\n  dplyr::mutate(YOUTH_OR_ADULT = dplyr::if_else(AGE \u003c 18, \"Youth\", \"Adult\"))\n\nstr(offenders)\n```\n\n    ## 'data.frame':    1413 obs. of  12 variables:\n    ##  $ LAST            : chr  \"RODRIGUEZ\" \"MARTINEZ\" \"GARCIA\" \"RODRIGUEZ\" ...\n    ##  $ FIRST           : chr  \"JUAN\" \"MOISES\" \"ELLIOTT\" \"JOSE\" ...\n    ##  $ BLOCK           : chr  \"009XX W CUYLER AVE\" \"011XX N KILBOURN AVE\" \"011XX W 18TH ST\" \"012XX W RACE AVE\" ...\n    ##  $ GENDER          : chr  \"MALE\" \"MALE\" \"MALE\" \"MALE\" ...\n    ##  $ REGION          : chr  \"West\" \"East\" \"South\" \"North\" ...\n    ##  $ BIRTH_DATE      : chr  \"06/22/1955\" \"02/07/1954\" \"08/11/1970\" \"02/10/1959\" ...\n    ##  $ HEIGHT          : int  198 198 201 237 201 199 201 236 198 199 ...\n    ##  $ WEIGHT          : int  190 180 200 195 220 130 200 235 140 130 ...\n    ##  $ PREV_CONVICTIONS: num  0 0 0 0 0 0 0 0 0 0 ...\n    ##  $ SENTENCE        : chr  \"Court_order\" \"Prison_\u003c12m\" \"Court_order\" \"Court_order\" ...\n    ##  $ AGE             : int  58 59 43 54 34 50 32 43 34 18 ...\n    ##  $ YOUTH_OR_ADULT  : chr  \"Adult\" \"Adult\" \"Adult\" \"Adult\" ...\n\n## Vectorising multiple if…else statements\n\nIn the previous section we saw how we can apply a single condition to a\nvector, but what if we want to apply several conditions, each with a\ndifferent outcome, at the same time? We can use the `case_when()`\nfunction from dplyr to do this. Let’s say that we wanted to add a column\nto the `offenders` dataframe with an age band for each offender. We can\ndo something like this:\n\n``` r\n# Add an age band column\noffenders \u003c- offenders %\u003e%\n  dplyr::mutate(AGE_BAND = dplyr::case_when(\n    AGE \u003c 18 ~ \"\u003c18\",\n    AGE \u003c 30 ~ \"18-29\",\n    AGE \u003c 40 ~ \"30-39\",\n    AGE \u003c 50 ~ \"40-49\",\n    AGE \u003c 60 ~ \"50-59\",\n    AGE \u003e= 60 ~ \"60+\",\n    TRUE ~ \"Unknown\"\n  ))\noffenders %\u003e% select(BIRTH_DATE, AGE, AGE_BAND) %\u003e% str(vec.len=6)\n```\n\n    ## 'data.frame':    1413 obs. of  3 variables:\n    ##  $ BIRTH_DATE: chr  \"06/22/1955\" \"02/07/1954\" \"08/11/1970\" \"02/10/1959\" \"04/16/1979\" \"11/19/1963\" ...\n    ##  $ AGE       : int  58 59 43 54 34 50 32 43 34 18 42 38 41 22 45 ...\n    ##  $ AGE_BAND  : chr  \"50-59\" \"50-59\" \"40-49\" \"50-59\" \"30-39\" \"50-59\" ...\n\n------------------------------------------------------------------------\n\nIn the `case_when()` function, each argument should be a two-sided\nformula. For each formula, the condition appears to the left of a ‘`~`’\nsymbol, and on the right is the value to assign if the condition\nevaluates to `TRUE`.\n\nNote that the order of conditional statements in the `case_when()`\nfunction is important if there are overlapping conditions. The\nconditions will be evaluated in the order that they appear in, so in the\nabove example, the `case_when()` will first check if the person is under\n18, then if they are under 30 (but over 18), and so on.\n\nA default value can be assigned in the event that none of the conditions\nare met. This is done by putting `TRUE` in the place of a condition. In\nthe example above, if none of the conditions are met then a value of\n`\"Unknown\"` is assigned.\n\n------------------------------------------------------------------------\n\n### Exercise 1\n\nAdd a column called ‘COURT_ORDER’ to the `offenders` dataframe. The\ncolumn should contain a ‘1’ if the offender received a court order, or a\n‘0’ otherwise, based on the categories in the ‘SENTENCE’ column.\n\n**Hint:** you’ll need to apply the `if_else()` function with `mutate()`.\n\n### Exercise 2\n\nAdd a column called ‘PREV_CONVICTIONS_BAND’ to the `offenders`\ndataframe. The column should contain the following categories: ‘Low’,\n‘Medium’, ‘High’, based on the number of convictions given in the\n‘PREV_CONVICTIONS’ column. For example, you can consider less than 5\nPREV_CONVICTIONS to be ‘Low’, 5 to 10 to be ‘Medium’, and over 10 to be\n‘High’.\n\n**Hint:** you’ll need to use the `case_when()` function with `mutate()`.\n\n------------------------------------------------------------------------\n\n------------------------------------------------------------------------\n\n# Iteration\n\n## Introduction\n\n‘For’ and ‘while’ loops are used to repeatedly execute a piece of code,\nand are a fundamental part of most programming languages. This chapter\nintroduces how to use them in R, as well as showing how we can iterate\nover the columns of a dataframe without needing to write a loop.\n\nA general rule of thumb in programming is to avoid copying and pasting a\npiece of code more than once; if you find that you are repeating similar\npieces of code over and over again, this is a sign that either a loop or\na\n[function](https://github.com/moj-analytical-services/writing_functions_in_r)\n(or both) are required. Keeping your code concise will help make it more\nreadable and easier to understand.\n\n## For loop basics\n\nLet’s start with a very basic example to illustrate what a for loop\ndoes. Say we wanted to print the numbers 1 to 5; without a for loop we’d\nneed to write something like this:\n\n``` r\n# Example of repeating the same piece of code for a set of values\nprint(1)\n## [1] 1\nprint(2)\n## [1] 2\nprint(3)\n## [1] 3\nprint(4)\n## [1] 4\nprint(5)\n## [1] 5\n```\n\n------------------------------------------------------------------------\n\nClearly there is some code repetition here, so we can achieve the same\nresult using a for loop:\n\n``` r\n# A basic for loop\nfor (i in 1:5) {\n  print(i)\n}\n```\n\n    ## [1] 1\n    ## [1] 2\n    ## [1] 3\n    ## [1] 4\n    ## [1] 5\n\nInside the brackets of the for loop you define a variable - in this case\ncalled `i` - along with what you want to iterate over, referred to as\nthe iterable. In this case the iterable is a sequence of the numbers 1\nto 5, denoted `1:5` in R. For each iteration, the variable `i` will take\non a value equal to the next element of the iterable. The loop body goes\ninside the curly brackets, which is where you define what you want to\nhappen for each iteration (in this case printing the value of `i`).\n\n------------------------------------------------------------------------\n\nIn the previous example we iterated over a sequence of numbers, but in R\nyou can iterate over anything you like. Here’s a similar example, but\niterating over a vector of strings instead of a sequence of numbers:\n\n``` r\nfruits \u003c- c(\"strawberry\", \"apple\", \"pear\", \"orange\")\n\n# Iterating over a vector\nfor (fruit in fruits) {\n  print(fruit)\n}\n```\n\n    ## [1] \"strawberry\"\n    ## [1] \"apple\"\n    ## [1] \"pear\"\n    ## [1] \"orange\"\n\n------------------------------------------------------------------------\n\nYou can also use for loops to populate or modify a vector or dataframe.\nThe following example shows how we can add the first ten numbers of the\nFibonacci sequence to a vector:\n\n``` r\n# Fibonacci for loop example\n\nn \u003c- 10 # Specify what length we want our output vector to be\nfibonacci \u003c- vector(\"numeric\", n) # Define an empty numeric vector of length n to populate using the loop\n\n# Set up the first couple of numbers to get the sequence started\nfibonacci[1] \u003c- 0 \nfibonacci[2] \u003c- 1\n\n# Add the rest of the sequence\nfor (i in 3:n) {\n  fibonacci[i] \u003c- fibonacci[i-1] + fibonacci[i-2]\n}\n\nprint(fibonacci)\n```\n\n    ##  [1]  0  1  1  2  3  5  8 13 21 34\n\nWhen writing a for loop you must define something to iterate over a\nfixed number of times in advance. It is also possible to iterate\nindefinitely using a different kind of loop - this is covered later on\nin the section on while loops.\n\n## More options with for loops\n\n### Iterating over the index of a vector\n\nIf you wanted to get an index number for each element of the iterable,\nyou can use the `seq_along()` function. For example:\n\n``` r\n# Iterating over the indices of a vector\n\nfruits \u003c- c(\"strawberry\", \"apple\", \"pear\", \"orange\")\n\nfor (i in seq_along(fruits)) {\n  # Use paste() to combine two strings together\n  print(paste(i, fruits[i]))\n}\n```\n\n    ## [1] \"1 strawberry\"\n    ## [1] \"2 apple\"\n    ## [1] \"3 pear\"\n    ## [1] \"4 orange\"\n\n------------------------------------------------------------------------\n\n### Conditionally exiting a loop\n\nYou might want to stop a loop iterating under a certain condition. In\nthis case you can use a `break` statement in combination with an ‘if’ or\n‘if…else’ statement, like so:\n\n``` r\nfor (i in 1:10) {\n  \n  # Exit the for loop if i is greater than 5\n  if (i \u003e 5) {\n    break\n  }\n  \n  print(i)\n}\n```\n\n    ## [1] 1\n    ## [1] 2\n    ## [1] 3\n    ## [1] 4\n    ## [1] 5\n\n------------------------------------------------------------------------\n\n### Conditionally skip to the next iteration\n\nThe `next` statement can be use to skip to the next iteration of the\nloop under a certain condition. For example, we can skip to the next\niteration if the iterable is NA (not available):\n\n``` r\ndata \u003c- c(56, 92, NA, 40, 11)\n\nfor (i in data) {\n  \n  # Skip this iteration if i is NA\n  if (is.na(i)) {\n    next\n  }\n  \n  print(i)\n}\n```\n\n    ## [1] 56\n    ## [1] 92\n    ## [1] 40\n    ## [1] 11\n\n------------------------------------------------------------------------\n\n### Handling outputs of unknown length\n\nThere are cases where the size of an output from a loop is not known\nbeforehand. For example, this might be because different iterations\nresult in outputs of different lengths.\n\nLet’s say we want to combine segments of a dataset, and we don’t know in\nadvance how many segments there are or how many rows they have. There is\na shared folder prepared in the alpha-r-training s3 bucket, which\ncontains some data for us to read in and combine together. First we need\nto get a list of files to read in, which we can do using the\n`list_files_in_buckets()` function from Rs3tools:\n\n``` r\n# Get dataframe with all available files/folders from an s3 path\nfiles \u003c- Rs3tools::list_files_in_buckets(\"alpha-r-training\", prefix=\"intro-r-extension/fruit\")\n\n# Get a list of csv file names\nfiles \u003c- files %\u003e%\n  dplyr::filter(grepl(\".csv\", path)) %\u003e%\n  dplyr::pull(path)\n\nfiles\n```\n\n    ## alpha-r-training/intro-r-extension/fruit/fruit1.csv\n    ## alpha-r-training/intro-r-extension/fruit/fruit2.csv\n    ## alpha-r-training/intro-r-extension/fruit/fruit3.csv\n\n------------------------------------------------------------------------\n\nNow we can use a for loop to read in each file as a dataframe, and add\neach dataframe to a list. After the for loop, the `bind_rows()` function\nfrom dplyr can be used to combine the data into a single dataframe.\n\n``` r\n# First define an empty list to be filled by the loop\nfruit_list \u003c- vector(\"list\", length(files))\n\n# Loop over each file, and add the data to a list\nfor (i in seq_along(files)) {\n  fruit_list[[i]] \u003c- Rs3tools::s3_path_to_full_df(files[i])\n}\n\n# Combine the list of dataframes into a single dataframe\nfruit \u003c- dplyr::bind_rows(fruit_list)\nfruit\n```\n\n    ##        Item Cost.Jan Cost.Feb Cost.Mar\n    ## 1    Orange     0.56     0.50     0.57\n    ## 2     Apple     0.42     0.51     0.49\n    ## 3    Banana     0.15     0.17     0.21\n    ## 4     Lemon     0.30     0.32     0.35\n    ## 5      Pear     0.41     0.39     0.44\n    ## 6     Melon     1.10     1.15     1.11\n    ## 7 Pineapple     1.18     1.19     1.24\n    ## 8     Peach     0.55     0.53     0.58\n    ## 9      Plum     0.38     0.41     0.41\n\n------------------------------------------------------------------------\n\nBy doing this we’ve combined together various segments of a dataset,\nwithout needing to know how many segments there are or how many rows are\nin each segment beforehand.\n\nNote: We’ve introduced a type of R object called a list in this example.\nLists are a type of vector that allow us to put a whole dataframe as an\nelement in the list. Compare this to the vectors we’ve met before, known\nas ‘atomic vectors’, where the elements can only contain a single value\nand they all need to be the same type (numeric, character, etc). The\nreason for doing it this way is because it’s more memory efficient to\nadd the dataframes to a list and use `bind_rows()` afterwards compared\nto appending the dataframes in each loop iteration.\n\n## While loops\n\nThere may be cases where we want to keep looping over a piece of code\nuntil a certain condition is met, rather than having to specify in\nadvance how many times a loop should run. In these cases a while loop\ncan be used, which can be thought of as a repeating if statement.\n\nFor example, we can use a while loop to achieve a similar result to the\nfirst for loop example above:\n\n``` r\n# First specify an initial value for the variable used in the while loop\ni \u003c- 1\n\n# Now define a while loop\nwhile (i \u003c= 5) { # The loop will continue until the condition i\u003c=5 is met\n  print(i)\n  i = i + 1 # Set the value of the variable for the next loop iteration\n}\n```\n\n    ## [1] 1\n    ## [1] 2\n    ## [1] 3\n    ## [1] 4\n    ## [1] 5\n\n------------------------------------------------------------------------\n\nThe syntax for a while loop is similar to that of a for loop, but in the\nbrackets a condition is specified instead of an iterable. Prior to\nwriting the while loop you’ll also need to specify an initial value for\nthe variable used in the loop, and there should be something in the body\nof the loop to change the variable during each iteration - otherwise the\ncondition can never be met!\n\nGenerally a while loop should only be used in circumstances where it\nisn’t possible to achieve the desired result using a for loop. The\nreason for this is that it can be easy to accidentally set up an\ninfinite loop, where a bug in the code means that the condition is never\nmet for a while loop to end.\n\n## Iterating over columns of a dataframe\n\nAlthough loops are an essential programming tool, there are cases where\nthe same outcome can be achieved in a more efficient way by using a\nbuilt-in function. For example, the tidyverse packages include functions\nthat allow us to apply operations across all (or a subset) of the\ncolumns in a dataframe at the same time. The advantages of using these\nbuilt-in functions are that they can make the code more concise and\neasier to read, plus they’re often faster to run than the loop\nequivalent.\n\n------------------------------------------------------------------------\n\nWe’ve already met the `mutate()` function from dplyr in the\n[Introduction to\nR](https://github.com/moj-analytical-services/IntroRTraining) course,\nwhich is a convenient way to apply an operation to all values in a\ncolumn of a dataframe. For example, going back to the `fruit` dataset\nthat we combined together earlier, here’s how we can make all characters\nin the `Item` column uppercase, using the `toupper()` function:\n\n``` r\n# Convert Item column to uppercase\nfruit \u003c- fruit %\u003e% dplyr::mutate(Item = toupper(Item))\nfruit\n```\n\n    ##        Item Cost.Jan Cost.Feb Cost.Mar\n    ## 1    ORANGE     0.56     0.50     0.57\n    ## 2     APPLE     0.42     0.51     0.49\n    ## 3    BANANA     0.15     0.17     0.21\n    ## 4     LEMON     0.30     0.32     0.35\n    ## 5      PEAR     0.41     0.39     0.44\n    ## 6     MELON     1.10     1.15     1.11\n    ## 7 PINEAPPLE     1.18     1.19     1.24\n    ## 8     PEACH     0.55     0.53     0.58\n    ## 9      PLUM     0.38     0.41     0.41\n\n------------------------------------------------------------------------\n\nWe can also use `mutate()` to apply a function to multiple columns in\none go by combining it with the `across()` function from dplyr. This\nexample demonstrates how to multiply the values in all numeric columns\nby 100:\n\n``` r\nfruit_pence \u003c- fruit %\u003e% dplyr::mutate(dplyr::across(where(is.numeric), ~ .x * 100))\nfruit_pence\n```\n\n    ##        Item Cost.Jan Cost.Feb Cost.Mar\n    ## 1    ORANGE       56       50       57\n    ## 2     APPLE       42       51       49\n    ## 3    BANANA       15       17       21\n    ## 4     LEMON       30       32       35\n    ## 5      PEAR       41       39       44\n    ## 6     MELON      110      115      111\n    ## 7 PINEAPPLE      118      119      124\n    ## 8     PEACH       55       53       58\n    ## 9      PLUM       38       41       41\n\nHere we’re using `mutate()` with `across()` to apply a function to all\nnumeric columns. The `~ .x * 100` part is what’s called a lambda or\nanonymous function, and this is what tells `mutate()` and `across()` to\nmultiply by 100. The lambda function is a function with no name -\nthey’re generally used in combination with another function (in this\ncase `across()`) to apply a simple operation without needing to define a\ndedicated function elsewhere in the code.\n\n------------------------------------------------------------------------\n\nOf course `across()` can also be used to apply a named function to\nmultiple columns of a dataframe. Here’s how we can apply the `signif()`\nfunction to round values in all numeric columns to 1 significant figure:\n\n``` r\nrounded_fruit \u003c- fruit %\u003e% dplyr::mutate(dplyr::across(where(is.numeric), signif, 1))\n\nrounded_fruit\n```\n\n    ##        Item Cost.Jan Cost.Feb Cost.Mar\n    ## 1    ORANGE      0.6      0.5      0.6\n    ## 2     APPLE      0.4      0.5      0.5\n    ## 3    BANANA      0.2      0.2      0.2\n    ## 4     LEMON      0.3      0.3      0.4\n    ## 5      PEAR      0.4      0.4      0.4\n    ## 6     MELON      1.0      1.0      1.0\n    ## 7 PINEAPPLE      1.0      1.0      1.0\n    ## 8     PEACH      0.6      0.5      0.6\n    ## 9      PLUM      0.4      0.4      0.4\n\nNote: When the `signif()` function is passed as an argument to\n`across()`, the brackets aren’t included (i.e. `signif` is passed rather\nthan `signif()`). This means that any arguments for `signif` need to be\nincluded as extra arguments for `across()` instead (i.e. putting\n`signif, 1` rather than `signif(1)` when using with `across()`).\n\n------------------------------------------------------------------------\n\n### Exercise 1\n\nWrite a for loop to print “The current date is …” for each date in the\nfollowing string vector:\n\n``` r\n# Set up a vector for Iteration - exercise 1\ndates \u003c- c(\"2020-03-01\", \"2020-06-01\", \"2020-09-01\", \"2020-12-01\")\n```\n\n**Hint:** You can use the `paste()` function to join strings together,\nand the `print()` function to print the result in the Console.\n\n### Exercise 2\n\nModify your solution to the previous exercise to skip to the next loop\niteration if `date` is equal to ‘2020-06-01’.\n\n------------------------------------------------------------------------\n\n# Handling missing data\n\n## Introduction\n\nIt’s often the case that datasets will contain missing values, which are\nusually denoted by `NA` in R. ‘NA’ stands for ‘not available’, while\nother programming languages might use ‘NaN’ (not a number) or ‘null’\ninstead. Care needs to be taken to make sure these missing values are\nhandled in the most appropriate way for a particular situation. This\nsection introduces a few methods for handling missing values in\ndifferent situations.\n\n## Identifying missing values\n\nThe function `is.na()` can be used to identify missing values, and it\ncan be applied to a single value or a vector. It returns `TRUE` if a\nvalue is `NA`, and `FALSE` otherwise:\n\n``` r\n# Check whether or not a single value is missing\nx \u003c- 7\nis.na(x)\n```\n\n    ## [1] FALSE\n\n``` r\n# Check whether or not each element of a vector is missing\nx \u003c- c(7, 23, 5, 14, NA, 1, 11, NA)\nis.na(x)\n```\n\n    ## [1] FALSE FALSE FALSE FALSE  TRUE FALSE FALSE  TRUE\n\n------------------------------------------------------------------------\n\nIf instead you wanted to identify values that are not missing, then you\ncan combine `is.na()` with the ‘not’ operator, `!`, like so:\n\n``` r\n# Check if values are NOT missing\n!is.na(x)\n```\n\n    ## [1]  TRUE  TRUE  TRUE  TRUE FALSE  TRUE  TRUE FALSE\n\nWhen working with dataframes, the `complete.cases()` function is useful\nto check which rows are complete (i.e. the row doesn’t contain any\nmissing values):\n\n``` r\n# Check which rows of a dataframe do not contain any missing values\ndf \u003c- tibble::tibble(\n  \"x\" = c(0, 1, 2, NA, 4),\n  \"y\" = c(18, NA, 45, 15, 2),\n)\n\ncomplete.cases(df)\n```\n\n    ## [1]  TRUE FALSE  TRUE FALSE  TRUE\n\n## Handling missing values using a function argument\n\nSome functions have built-in arguments where you can specify what you\nwant to happen to missing values. For example, the `sum()` function has\nan argument called `na.rm` that you can use to specify if you want the\n`NA` values to be removed before the sum is calculated. By default\n`na.rm` is set to `FALSE`:\n\n``` r\n# What happens if you sum a vector containing missing values\nx \u003c- c(7, 23, 5, 14, NA, 1, 11, NA)\nsum(x)\n```\n\n    ## [1] NA\n\nSo if we try to sum over numeric vector that contains `NA` values, then\nthe result is `NA`. By setting `na.rm` to `TRUE`, however, we can remove\nthe `NA` values before continuing with the sum:\n\n``` r\n# We can use a function argument to ignore the missing values\nx \u003c- c(7, 23, 5, 14, NA, 1, 11, NA)\nsum(x, na.rm=TRUE)\n```\n\n    ## [1] 61\n\n## Converting values to NA\n\nThere might be occasions where we want to set some values to `NA`, for\nexample if they are invalid. The `replace()` function can be used to\nreplace values based on a particular condition. This example shows how\nnegative values in a vector can be replaced with `NA`:\n\n``` r\n# Setting values to NA under a certain condition\nx \u003c- c(7, 23, 5, -14, 0, -1, 11, 0)\nreplace(x, x \u003c 0, NA)\n```\n\n    ## [1]  7 23  5 NA  0 NA 11  0\n\n## Replacing missing values\n\nSometimes it’s necessary to replace missing values; for example, when\ndisplaying data in a table or chart, or when the missing values would\ncause problems for a particular calculation. There are a few different\noptions that we’ll visit in this section.\n\n### Replacing with a specific value\n\nWe can also use the `replace()` function to replace missing values with\na specific value. In this example we’re replacing missing values with\nzero:\n\n``` r\n# Replacing NA values with 0 in a vector\nx \u003c- c(7, 23, 5, -14, NA, -1, 11,NA)\nreplace(x, is.na(x), 0)\n```\n\n    ## [1]   7  23   5 -14   0  -1  11   0\n\n------------------------------------------------------------------------\n\nThe `replace()` function can also be applied to a whole dataframe, like\nso:\n\n``` r\n# Replacing NA values with 0 over a whole dataframe\ndf \u003c- tibble::tibble(\n  \"x\" = c(0, 1, 2, NA, 4),\n  \"y\" = c(18, NA, 45, 15, 2),\n)\n\ndf %\u003e% replace(is.na(.), 0)\n```\n\n    ## # A tibble: 5 x 2\n    ##       x     y\n    ##   \u003cdbl\u003e \u003cdbl\u003e\n    ## 1     0    18\n    ## 2     1     0\n    ## 3     2    45\n    ## 4     0    15\n    ## 5     4     2\n\n------------------------------------------------------------------------\n\nThe `replace_na()` function from tidyr also provides a convenient way to\nreplace missing values in a dataframe, and is especially useful if you\nwant to use different replacement values for different columns.\n\nHere’s an example of how to use `replace_na()` with the `offenders`\ndataframe, where we’re replacing missing values in the `HEIGHT` column\nwith ‘Unknown’:\n\n``` r\n# Replace NAs in a specific column of a dataframe\noffenders_replacena \u003c- offenders %\u003e%\n  dplyr::mutate(HEIGHT = as.character(HEIGHT)) %\u003e%\n  tidyr::replace_na(list(HEIGHT = \"Unknown\"))\n\n# Display the dataframe in descending height order, so we can see the 'Unknown' values\noffenders_replacena %\u003e% dplyr::arrange(desc(HEIGHT)) %\u003e% str()\n```\n\n    ## 'data.frame':    1413 obs. of  15 variables:\n    ##  $ LAST                 : chr  \"FERNANDEZ\" \"GARCIA\" \"FIGUEROA\" \"BURKHART\" ...\n    ##  $ FIRST                : chr  \"FRANCISCO\" \"KEMICH\" \"JOSE\" \"RONALD\" ...\n    ##  $ BLOCK                : chr  \"028XX S CHRISTIANA AVE\" \"033XX W 38TH ST\" \"054XX N ASHLAND AVE\" \"007XX N TRUMBULL AVE\" ...\n    ##  $ GENDER               : chr  \"MALE\" \"MALE\" \"MALE\" \"MALE\" ...\n    ##  $ REGION               : chr  \"West\" \"East\" \"South\" \"East\" ...\n    ##  $ BIRTH_DATE           : chr  \"08/06/1981\" \"06/03/1971\" \"07/13/1985\" \"06/15/1963\" ...\n    ##  $ HEIGHT               : chr  \"Unknown\" \"Unknown\" \"Unknown\" \"Unknown\" ...\n    ##  $ WEIGHT               : int  180 170 220 225 180 125 200 185 240 170 ...\n    ##  $ PREV_CONVICTIONS     : num  0 0 0 0 0 0 0 0 1.4 1.4 ...\n    ##  $ SENTENCE             : chr  \"Court_order\" \"Prison_\u003c12m\" \"Prison_\u003c12m\" \"Court_order\" ...\n    ##  $ AGE                  : int  32 42 28 50 48 61 45 32 43 41 ...\n    ##  $ YOUTH_OR_ADULT       : chr  \"Adult\" \"Adult\" \"Adult\" \"Adult\" ...\n    ##  $ AGE_BAND             : chr  \"30-39\" \"40-49\" \"18-29\" \"50-59\" ...\n    ##  $ COURT_ORDER          : num  1 0 0 1 1 0 1 1 1 0 ...\n    ##  $ PREV_CONVICTIONS_BAND: chr  \"Low\" \"Low\" \"Low\" \"Low\" ...\n\n------------------------------------------------------------------------\n\n### Replacing with values from another column\n\nThe `coalesce()` function from dplyr can be used to fill in missing\nvalues with values from another column. Before we jump into an example,\nlet’s first prepare a dataframe:\n\n``` r\n# Set up a dataframe to use in the next example\nevent_dates \u003c- tibble::tibble(\n  \"event_id\" = c(0, 1, 2, 3, 4, 5),\n  \"date\" = c(\"2016-04-13\", \"2015-12-29\", \"2016-06-02\", \"2017-01-27\", \"2015-10-21\", \"2018-03-15\"),\n  \"new_date\" = c(\"2016-08-16\", NA, NA, \"2017-03-02\", NA, \"2018-11-20\")\n)\n\nevent_dates\n```\n\n    ## # A tibble: 6 x 3\n    ##   event_id date       new_date  \n    ##      \u003cdbl\u003e \u003cchr\u003e      \u003cchr\u003e     \n    ## 1        0 2016-04-13 2016-08-16\n    ## 2        1 2015-12-29 \u003cNA\u003e      \n    ## 3        2 2016-06-02 \u003cNA\u003e      \n    ## 4        3 2017-01-27 2017-03-02\n    ## 5        4 2015-10-21 \u003cNA\u003e      \n    ## 6        5 2018-03-15 2018-11-20\n\n------------------------------------------------------------------------\n\nWe can use `coalesce()` to fill in the missing dates in the `new_date`\ncolumn with the equivalent date in the `date` column:\n\n``` r\n# Fill missing values in one column using corresponding values in another column\nevent_dates %\u003e%\n  dplyr::mutate(new_date = dplyr::coalesce(new_date, date))\n```\n\n    ## # A tibble: 6 x 3\n    ##   event_id date       new_date  \n    ##      \u003cdbl\u003e \u003cchr\u003e      \u003cchr\u003e     \n    ## 1        0 2016-04-13 2016-08-16\n    ## 2        1 2015-12-29 2015-12-29\n    ## 3        2 2016-06-02 2016-06-02\n    ## 4        3 2017-01-27 2017-03-02\n    ## 5        4 2015-10-21 2015-10-21\n    ## 6        5 2018-03-15 2018-11-20\n\n------------------------------------------------------------------------\n\n### Replacing with the previous value in a column\n\nIf we were to encounter a dataframe like the following, where each group\nname in the `year` column appears only once, then we might want to fill\nin the missing labels before beginning any analysis.\n\n``` r\n# Construct the example dataframe\ndf \u003c- tidyr::crossing(year = c(\"2017\", \"2018\", \"2019\"),\n                      quarter = c(\"Q1\", \"Q2\", \"Q3\", \"Q4\")) %\u003e%\n      dplyr::mutate(count = sample(length(year)))\n\ndf$year[duplicated(df$year)] \u003c- NA # This removes repeated row labels\ndf\n```\n\n    ## # A tibble: 12 x 3\n    ##    year  quarter count\n    ##    \u003cchr\u003e \u003cchr\u003e   \u003cint\u003e\n    ##  1 2017  Q1          7\n    ##  2 \u003cNA\u003e  Q2          4\n    ##  3 \u003cNA\u003e  Q3          5\n    ##  4 \u003cNA\u003e  Q4          2\n    ##  5 2018  Q1         11\n    ##  6 \u003cNA\u003e  Q2          9\n    ##  7 \u003cNA\u003e  Q3          3\n    ##  8 \u003cNA\u003e  Q4         12\n    ##  9 2019  Q1          1\n    ## 10 \u003cNA\u003e  Q2         10\n    ## 11 \u003cNA\u003e  Q3          6\n    ## 12 \u003cNA\u003e  Q4          8\n\n------------------------------------------------------------------------\n\nThe `fill()` function from tidyr is a convenient way to do this, and can\nbe used like this:\n\n``` r\n# Fill missing values in a column using the nearest previous non-NA value from the same column\ndf %\u003e% tidyr::fill(year)\n```\n\n    ## # A tibble: 12 x 3\n    ##    year  quarter count\n    ##    \u003cchr\u003e \u003cchr\u003e   \u003cint\u003e\n    ##  1 2017  Q1          7\n    ##  2 2017  Q2          4\n    ##  3 2017  Q3          5\n    ##  4 2017  Q4          2\n    ##  5 2018  Q1         11\n    ##  6 2018  Q2          9\n    ##  7 2018  Q3          3\n    ##  8 2018  Q4         12\n    ##  9 2019  Q1          1\n    ## 10 2019  Q2         10\n    ## 11 2019  Q3          6\n    ## 12 2019  Q4          8\n\n## Removing rows with missing values from a dataframe\n\nThe `drop_na()` function from tidyr allows us to easily remove rows\ncontaining `NA` values from a dataframe. Let’s say we wanted to remove\nall incomplete rows from the `offenders` dataset. We can either do this:\n\n``` r\n# Remove entire row if it contains a missing value\noffenders_nona \u003c- offenders %\u003e% tidyr::drop_na()\nstr(offenders_nona)\n```\n\n    ## 'data.frame':    1389 obs. of  15 variables:\n    ##  $ LAST                 : chr  \"RODRIGUEZ\" \"MARTINEZ\" \"GARCIA\" \"RODRIGUEZ\" ...\n    ##  $ FIRST                : chr  \"JUAN\" \"MOISES\" \"ELLIOTT\" \"JOSE\" ...\n    ##  $ BLOCK                : chr  \"009XX W CUYLER AVE\" \"011XX N KILBOURN AVE\" \"011XX W 18TH ST\" \"012XX W RACE AVE\" ...\n    ##  $ GENDER               : chr  \"MALE\" \"MALE\" \"MALE\" \"MALE\" ...\n    ##  $ REGION               : chr  \"West\" \"East\" \"South\" \"North\" ...\n    ##  $ BIRTH_DATE           : chr  \"06/22/1955\" \"02/07/1954\" \"08/11/1970\" \"02/10/1959\" ...\n    ##  $ HEIGHT               : int  198 198 201 237 201 199 201 236 198 199 ...\n    ##  $ WEIGHT               : int  190 180 200 195 220 130 200 235 140 130 ...\n    ##  $ PREV_CONVICTIONS     : num  0 0 0 0 0 0 0 0 0 0 ...\n    ##  $ SENTENCE             : chr  \"Court_order\" \"Prison_\u003c12m\" \"Court_order\" \"Court_order\" ...\n    ##  $ AGE                  : int  58 59 43 54 34 50 32 43 34 18 ...\n    ##  $ YOUTH_OR_ADULT       : chr  \"Adult\" \"Adult\" \"Adult\" \"Adult\" ...\n    ##  $ AGE_BAND             : chr  \"50-59\" \"50-59\" \"40-49\" \"50-59\" ...\n    ##  $ COURT_ORDER          : num  1 0 1 1 0 0 0 1 1 0 ...\n    ##  $ PREV_CONVICTIONS_BAND: chr  \"Low\" \"Low\" \"Low\" \"Low\" ...\n\n------------------------------------------------------------------------\n\nOr alternatively you can remove rows that contain `NA` values in\nspecific columns:\n\n``` r\n# Remove entire row if it contains missing values in specific columns\noffenders_nona \u003c- offenders %\u003e% tidyr::drop_na(HEIGHT, WEIGHT)\nstr(offenders_nona)\n```\n\n    ## 'data.frame':    1389 obs. of  15 variables:\n    ##  $ LAST                 : chr  \"RODRIGUEZ\" \"MARTINEZ\" \"GARCIA\" \"RODRIGUEZ\" ...\n    ##  $ FIRST                : chr  \"JUAN\" \"MOISES\" \"ELLIOTT\" \"JOSE\" ...\n    ##  $ BLOCK                : chr  \"009XX W CUYLER AVE\" \"011XX N KILBOURN AVE\" \"011XX W 18TH ST\" \"012XX W RACE AVE\" ...\n    ##  $ GENDER               : chr  \"MALE\" \"MALE\" \"MALE\" \"MALE\" ...\n    ##  $ REGION               : chr  \"West\" \"East\" \"South\" \"North\" ...\n    ##  $ BIRTH_DATE           : chr  \"06/22/1955\" \"02/07/1954\" \"08/11/1970\" \"02/10/1959\" ...\n    ##  $ HEIGHT               : int  198 198 201 237 201 199 201 236 198 199 ...\n    ##  $ WEIGHT               : int  190 180 200 195 220 130 200 235 140 130 ...\n    ##  $ PREV_CONVICTIONS     : num  0 0 0 0 0 0 0 0 0 0 ...\n    ##  $ SENTENCE             : chr  \"Court_order\" \"Prison_\u003c12m\" \"Court_order\" \"Court_order\" ...\n    ##  $ AGE                  : int  58 59 43 54 34 50 32 43 34 18 ...\n    ##  $ YOUTH_OR_ADULT       : chr  \"Adult\" \"Adult\" \"Adult\" \"Adult\" ...\n    ##  $ AGE_BAND             : chr  \"50-59\" \"50-59\" \"40-49\" \"50-59\" ...\n    ##  $ COURT_ORDER          : num  1 0 1 1 0 0 0 1 1 0 ...\n    ##  $ PREV_CONVICTIONS_BAND: chr  \"Low\" \"Low\" \"Low\" \"Low\" ...\n\n------------------------------------------------------------------------\n\n### Exercise 1\n\nFor the following dataframe, use the `filter()` function from dplyr with\n`complete.cases()` to extract the rows **with** missing values:\n\n``` r\n# Set up example dataframe for Missing data - exercise 1\nfruit \u003c- tibble::tibble(\n  \"Item\" = c(\"Orange\", \"Apple\", \"Banana\", \"Lemon\", \"Pear\"),\n  \"Cost\" = c(0.5, 0.4, 0.1, 0.3, NA),\n  \"Quantity\" = c(23, NA, 15, 9, 11)\n)\n```\n\n**Hint:** you can use a `.` inside the `complete.cases()` function to\napply it to all columns of the dataframe.\n\n------------------------------------------------------------------------\n\n### Exercise 2\n\nFor the following dataframe, use the `replace_na()` function from tidyr\nto replace missing values in the `Cost` column with “Unknown” and the\n`Quantity` column with 0.\n\n``` r\n# Set up example dataframe for Missing data - exercise 2\nfruit \u003c- tibble::tibble(\n  \"Item\" = c(\"Orange\", \"Apple\", \"Banana\", \"Lemon\", \"Pear\"),\n  \"Cost\" = c(\"£0.50\", \"£0.40\", \"£0.10\", \"£0.30\", NA),\n  \"Quantity\" = c(23, NA, 15, 9, 11)\n)\n```\n\n**Hint:** you can add multiple arguments to `replace_na(list(...))`,\nwith one argument for each column where NA values need replacing.\n\n------------------------------------------------------------------------\n\n# Reshaping data\n\n## Introduction\n\nThe exact same data can be represented in different orientations,\ndepending on the purpose.\n\nA dataframe that is in long format has a single column for each\nvariable. The number of columns is minimised, at the expense of having\nmany rows.\n\nA dataframe that is in wide format spreads a variable across several\ncolumns. The number of rows is minimised, at the expense of many\ncolumns.\n\nThere are advantages and disadvantages of each depending on context, and\nit is useful to know how to switch between these. It is very easy with\nthe `tidyverse` functions (package `tidyr`) `pivot_wider()` and\n`pivot_longer()`.\n\n## Widening data\n\nWe read in a data table.\n\n``` r\n# read in the fake annual offences data\nannual_offences \u003c- \n  Rs3tools::s3_path_to_full_df(\n    s3_path = \"s3://alpha-r-training/intro-r-extension/annual_offences_fake.csv\", \n    colClasses = c(\"integer\", \"character\", \"integer\")) %\u003e%\n  tibble::tibble()\n\nhead(annual_offences)\n```\n\n    ## # A tibble: 6 x 3\n    ##    year offence_code count\n    ##   \u003cint\u003e \u003cchr\u003e        \u003cint\u003e\n    ## 1  2016 00101          219\n    ## 2  2016 00304         4730\n    ## 3  2016 00305           28\n    ## 4  2016 00399         6405\n    ## 5  2016 00405            9\n    ## 6  2016 00406            3\n\n------------------------------------------------------------------------\n\n``` r\nn_rows \u003c- dim(annual_offences)[1]\nn_cols \u003c- dim(annual_offences)[2]\nprint(paste(\"The table is\", n_rows, \"rows by\", n_cols, \"cols, making\", n_rows * n_cols, \"cells\", sep = \" \"))\n```\n\n    ## [1] \"The table is 3563 rows by 3 cols, making 10689 cells\"\n\nThe data represent fake frequencies of offences from 2016 to 2020,\nrepresented by real Home Office offence codes. If an offence was\nprosecuted in a year, there is a corresponding line in this data table,\nwith the offence code indicated by the `offence_code` column, the year\nindicated by the `year` column, and the `count` column representing the\nnumber of times the offence was prosecuted. If an offence was not\nprosecuted in a year, the corresponding combination of `year` and\n`offence` does not exist. The table has been sorted by year and offence\ncode.\n\n------------------------------------------------------------------------\n\nThe long format may be a good way to store data like these for some\npurposes, but what if we want to put it into wide format, e.g. to make\nit easier for a human to read? We use the `tidyr` function\n`pivot_wider()`:\n\n``` r\n# basic implementation of pivot_wider()\nwide_annual_offences \u003c- annual_offences %\u003e%\n  tidyr::pivot_wider(\n    names_from = 'year',\n    values_from = 'count'\n  )\n\nhead(wide_annual_offences)\n```\n\n    ## # A tibble: 6 x 6\n    ##   offence_code `2016` `2017` `2018` `2019` `2020`\n    ##   \u003cchr\u003e         \u003cint\u003e  \u003cint\u003e  \u003cint\u003e  \u003cint\u003e  \u003cint\u003e\n    ## 1 00101           219    188    177    154    122\n    ## 2 00304          4730   4953   4954   5613   4485\n    ## 3 00305            28     20     17     10      6\n    ## 4 00399          6405   5879   5149   4538   3415\n    ## 5 00405             9      3      4      4     NA\n    ## 6 00406             3     NA      1     NA     NA\n\n------------------------------------------------------------------------\n\n``` r\nn_rows \u003c- dim(wide_annual_offences)[1]\nn_cols \u003c- dim(wide_annual_offences)[2]\nprint(paste(\"The table is\", n_rows, \"rows by\", n_cols, \"cols, making\", n_rows * n_cols, \"cells\", sep = \" \"))\n```\n\n    ## [1] \"The table is 978 rows by 6 cols, making 5868 cells\"\n\nWhat’s happened? We passed `count` to the argument `values_from` and\n`year` to the argument `names_from`. This tells the function that we\nwant to make new columns based on `year`, and populate it with the\nvalues from `count`.\n\nRemember that the data are sorted first by year, and then by offence? If\nwe imagine each year as a stack of data, and the table containing one\nstack for each year, then what we’re effectively doing here is taking\nthe count data for each stack and putting them in their own column. We\nend up with a table that has one row per offence code, and one column\nfor each year. There are fewer cells in total, although the same data\nare represented in both tables.\n\n------------------------------------------------------------------------\n\nThere are a couple of ways we could get more useful results from this\nfunction, though.\n\nFirst, it’s generally not a good idea to have column names that begin\nwith numbers. Fortunately, `pivot_wider()` has the useful argument\n`names_prefix` to remedy this:\n\n``` r\n# adding a prefix to new columns\nwide_annual_offences \u003c- annual_offences %\u003e%\n  tidyr::pivot_wider(\n    names_from = 'year',\n    values_from = 'count',\n    names_prefix = 'count_'\n  )\nhead(wide_annual_offences)\n```\n\n    ## # A tibble: 6 x 6\n    ##   offence_code count_2016 count_2017 count_2018 count_2019 count_2020\n    ##   \u003cchr\u003e             \u003cint\u003e      \u003cint\u003e      \u003cint\u003e      \u003cint\u003e      \u003cint\u003e\n    ## 1 00101               219        188        177        154        122\n    ## 2 00304              4730       4953       4954       5613       4485\n    ## 3 00305                28         20         17         10          6\n    ## 4 00399              6405       5879       5149       4538       3415\n    ## 5 00405                 9          3          4          4         NA\n    ## 6 00406                 3         NA          1         NA         NA\n\n------------------------------------------------------------------------\n\nWhen transforming count data like this we may have legitimate good\nreason to replace our NAs with 0s, which we can do with `values_fill()`:\n\n``` r\n# replacing NAs with 0s\nwide_annual_offences \u003c- annual_offences %\u003e%\n  tidyr::pivot_wider(\n    names_from = 'year',\n    values_from = 'count',\n    names_prefix = 'count_',\n    values_fill = 0\n  )\nhead(wide_annual_offences)\n```\n\n    ## # A tibble: 6 x 6\n    ##   offence_code count_2016 count_2017 count_2018 count_2019 count_2020\n    ##   \u003cchr\u003e             \u003cint\u003e      \u003cint\u003e      \u003cint\u003e      \u003cint\u003e      \u003cint\u003e\n    ## 1 00101               219        188        177        154        122\n    ## 2 00304              4730       4953       4954       5613       4485\n    ## 3 00305                28         20         17         10          6\n    ## 4 00399              6405       5879       5149       4538       3415\n    ## 5 00405                 9          3          4          4          0\n    ## 6 00406                 3          0          1          0          0\n\n------------------------------------------------------------------------\n\nOnce our table is in wide format, and clean, we can easily do\ntransformations like this. Here we use `dplyr` functions to create a new\ncolumn that adds up yearly totals across each column that has count\ndata:\n\n``` r\n# Creating a new column from the ones we've created\nwide_annual_offences_with_totals \u003c- wide_annual_offences %\u003e%\n  dplyr::mutate(\n    count_2016_2020 =\n      rowSums(dplyr::across(c('count_2016', 'count_2017','count_2018','count_2019','count_2020')))\n    )\nhead(wide_annual_offences_with_totals)\n```\n\n    ## # A tibble: 6 x 7\n    ##   offence_code count_2016 count_2017 count_2018 count_2019 count_2020 count_2016_2020\n    ##   \u003cchr\u003e             \u003cint\u003e      \u003cint\u003e      \u003cint\u003e      \u003cint\u003e      \u003cint\u003e           \u003cdbl\u003e\n    ## 1 00101               219        188        177        154        122             860\n    ## 2 00304              4730       4953       4954       5613       4485           24735\n    ## 3 00305                28         20         17         10          6              81\n    ## 4 00399              6405       5879       5149       4538       3415           25386\n    ## 5 00405                 9          3          4          4          0              20\n    ## 6 00406                 3          0          1          0          0               4\n\n------------------------------------------------------------------------\n\nThe final and most advanced thing we will do with `pivot_wider()` is to\npass it an auxiliary function to transform the values that it places in\nits new columns.\n\nHere we are passing an anonymous function which itself calls the\n`round()` function to round our counts. Setting the `digits` argument of\n`round()` to -1 means that the values get rounded to the nearest 10,\nrather than the default behaviour of rounding to the nearest whole\nnumber.\n\n``` r\n# passing an auxiliary function to `pivot_wider()`\nwide_annual_offences_rounded \u003c- annual_offences %\u003e%\n  tidyr::pivot_wider(\n    names_from = 'year',\n    values_from = 'count',\n    names_prefix = 'count_',\n    values_fill = 0,\n    values_fn = list(count = ~round(.x, digits = -1))\n  )\nhead(wide_annual_offences_rounded)\n```\n\n    ## # A tibble: 6 x 6\n    ##   offence_code count_2016 count_2017 count_2018 count_2019 count_2020\n    ##   \u003cchr\u003e             \u003cdbl\u003e      \u003cdbl\u003e      \u003cdbl\u003e      \u003cdbl\u003e      \u003cdbl\u003e\n    ## 1 00101               220        190        180        150        120\n    ## 2 00304              4730       4950       4950       5610       4480\n    ## 3 00305                30         20         20         10         10\n    ## 4 00399              6400       5880       5150       4540       3420\n    ## 5 00405                10          0          0          0          0\n    ## 6 00406                 0          0          0          0          0\n\n## Lengthening data\n\nLet’s consider our earlier widened table, with original counts, columns\nwith prefixes and NAs replaced with 0s.\n\nWhat if we want to go from our widened table back to our original one,\nhere?\n\n``` r\nhead(wide_annual_offences, 3)\n```\n\n    ## # A tibble: 3 x 6\n    ##   offence_code count_2016 count_2017 count_2018 count_2019 count_2020\n    ##   \u003cchr\u003e             \u003cint\u003e      \u003cint\u003e      \u003cint\u003e      \u003cint\u003e      \u003cint\u003e\n    ## 1 00101               219        188        177        154        122\n    ## 2 00304              4730       4953       4954       5613       4485\n    ## 3 00305                28         20         17         10          6\n\n``` r\nhead(annual_offences, 3)\n```\n\n    ## # A tibble: 3 x 3\n    ##    year offence_code count\n    ##   \u003cint\u003e \u003cchr\u003e        \u003cint\u003e\n    ## 1  2016 00101          219\n    ## 2  2016 00304         4730\n    ## 3  2016 00305           28\n\n------------------------------------------------------------------------\n\nWe use the function `pivot_longer()` for this. You can pass column names\nto it like this:\n\n``` r\n# basic transformation of a table into long format\nlong_annual_offences \u003c- wide_annual_offences %\u003e%\n  tidyr::pivot_longer(\n    cols = c('count_2016', 'count_2017', 'count_2018', 'count_2019', 'count_2020')\n  )\nhead(long_annual_offences)\n```\n\n    ## # A tibble: 6 x 3\n    ##   offence_code name       value\n    ##   \u003cchr\u003e        \u003cchr\u003e      \u003cint\u003e\n    ## 1 00101        count_2016   219\n    ## 2 00101        count_2017   188\n    ## 3 00101        count_2018   177\n    ## 4 00101        count_2019   154\n    ## 5 00101        count_2020   122\n    ## 6 00304        count_2016  4730\n\n------------------------------------------------------------------------\n\nOr, as our column names are conveniently named with a prefix, we can use\n`starts_with()` from `dplyr`:\n\n``` r\n# identifying columns using `starts_with()`\nlong_annual_offences \u003c- wide_annual_offences %\u003e%\n  tidyr::pivot_longer(\n    cols = dplyr::starts_with('count')\n  )\nhead(long_annual_offences)\n```\n\n    ## # A tibble: 6 x 3\n    ##   offence_code name       value\n    ##   \u003cchr\u003e        \u003cchr\u003e      \u003cint\u003e\n    ## 1 00101        count_2016   219\n    ## 2 00101        count_2017   188\n    ## 3 00101        count_2018   177\n    ## 4 00101        count_2019   154\n    ## 5 00101        count_2020   122\n    ## 6 00304        count_2016  4730\n\n------------------------------------------------------------------------\n\nEssentially, these data are the same as what we started with, but there\nare some differences.\n\n``` r\n# checking if the original table and working table are identical\nidentical(long_annual_offences, annual_offences)\n```\n\n    ## [1] FALSE\n\n``` r\nhead(annual_offences, 3)\n```\n\n    ## # A tibble: 3 x 3\n    ##    year offence_code count\n    ##   \u003cint\u003e \u003cchr\u003e        \u003cint\u003e\n    ## 1  2016 00101          219\n    ## 2  2016 00304         4730\n    ## 3  2016 00305           28\n\n``` r\nhead(long_annual_offences, 3)\n```\n\n    ## # A tibble: 3 x 3\n    ##   offence_code name       value\n    ##   \u003cchr\u003e        \u003cchr\u003e      \u003cint\u003e\n    ## 1 00101        count_2016   219\n    ## 2 00101        count_2017   188\n    ## 3 00101        count_2018   177\n\n------------------------------------------------------------------------\n\nIn fact, there are *six* differences between these tables. Have a look\nyourself, and put suggestions in the chat as to what these might be.\nThen we’ll cover how to correct these differences and make our working\ntable identical to the original table.\n\nThankfully we can iron out these differences through a combination of\namending our call to `pivot_wider()` and passing the result to some\n`dplyr` functions.\n\n------------------------------------------------------------------------\n\nFirst, the default column name `value` has been assigned to our count,\nwhich we correct with the argument `values_to`, giving it the label we\nsee in the original table:\n\n``` r\n# specifying a name for the `values` column\nlong_annual_offences \u003c- wide_annual_offences %\u003e%\n  tidyr::pivot_longer(\n    cols = dplyr::starts_with('count'),\n    values_to = 'count'\n  )\nhead(long_annual_offences)\n```\n\n    ## # A tibble: 6 x 3\n    ##   offence_code name       count\n    ##   \u003cchr\u003e        \u003cchr\u003e      \u003cint\u003e\n    ## 1 00101        count_2016   219\n    ## 2 00101        count_2017   188\n    ## 3 00101        count_2018   177\n    ## 4 00101        count_2019   154\n    ## 5 00101        count_2020   122\n    ## 6 00304        count_2016  4730\n\n``` r\nidentical(long_annual_offences, annual_offences)\n```\n\n    ## [1] FALSE\n\n------------------------------------------------------------------------\n\nThere’s another default name that it’s assigned too — it’s used `name`\nwhen we want `year` to indicate the years. We correct this with an\nequivalent argument:\n\n``` r\n# specifying a name for the `names` column\nlong_annual_offences \u003c- wide_annual_offences %\u003e%\n  tidyr::pivot_longer(\n    cols = dplyr::starts_with('count'),\n    values_to = 'count',\n    names_to = 'year'\n  )\nhead(long_annual_offences)\n```\n\n    ## # A tibble: 6 x 3\n    ##   offence_code year       count\n    ##   \u003cchr\u003e        \u003cchr\u003e      \u003cint\u003e\n    ## 1 00101        count_2016   219\n    ## 2 00101        count_2017   188\n    ## 3 00101        count_2018   177\n    ## 4 00101        count_2019   154\n    ## 5 00101        count_2020   122\n    ## 6 00304        count_2016  4730\n\n``` r\nidentical(long_annual_offences, annual_offences)\n```\n\n    ## [1] FALSE\n\n------------------------------------------------------------------------\n\nWe also want to remove those prefixes:\n\n``` r\n# providing substring prefix to remove from column names before using them in our combined `names` column\nlong_annual_offences \u003c- wide_annual_offences %\u003e%\n  tidyr::pivot_longer(\n    cols = dplyr::starts_with('count'),\n    values_to = 'count',\n    names_to = 'year',\n    names_prefix = 'count_'\n  )\nhead(long_annual_offences)\n```\n\n    ## # A tibble: 6 x 3\n    ##   offence_code year  count\n    ##   \u003cchr\u003e        \u003cchr\u003e \u003cint\u003e\n    ## 1 00101        2016    219\n    ## 2 00101        2017    188\n    ## 3 00101        2018    177\n    ## 4 00101        2019    154\n    ## 5 00101        2020    122\n    ## 6 00304        2016   4730\n\n``` r\nidentical(long_annual_offences, annual_offences)\n```\n\n    ## [1] FALSE\n\n------------------------------------------------------------------------\n\nStill more to do! We have the right number of columns in our new table,\nbut we have more rows than we should. That’s because of those\nyear/offence combinations where there are no incidences.\n\n``` r\nn_rows \u003c- dim(annual_offences)[1]\nn_cols \u003c- dim(annual_offences)[2]\nprint(paste(\"The original table is\", n_rows, \"rows by\", n_cols, \"cols, making\", n_rows * n_cols, \"cells\", sep = \" \"))\n```\n\n    ## [1] \"The original table is 3563 rows by 3 cols, making 10689 cells\"\n\n``` r\nn_rows \u003c- dim(long_annual_offences)[1]\nn_cols \u003c- dim(long_annual_offences)[2]\nprint(paste(\"Our working table is\", n_rows, \"rows by\", n_cols, \"cols, making\", n_rows * n_cols, \"cells\", sep = \" \"))\n```\n\n    ## [1] \"Our working table is 4890 rows by 3 cols, making 14670 cells\"\n\n------------------------------------------------------------------------\n\nLet’s get `dplyr` involved, and filter these out:\n\n``` r\n# Filtering out rows with no offences\nlong_annual_offences \u003c- wide_annual_offences %\u003e%\n  tidyr::pivot_longer(\n    cols = dplyr::starts_with('count'),\n    values_to = 'count',\n    names_to = 'year',\n    names_prefix = 'count_'\n  ) %\u003e%\n  dplyr::filter(count \u003e 0)\n```\n\nWe now have the same number of rows in original table and the one we’re\nworking on:\n\n``` r\nnrow(long_annual_offences) == nrow(annual_offences)\n```\n\n    ## [1] TRUE\n\n------------------------------------------------------------------------\n\nBut we’re still not quite there…\n\n``` r\nidentical(long_annual_offences, annual_offences)\n```\n\n    ## [1] FALSE\n\nFinally, we use `dplyr` to: 1) fix data types and reorder columns with\n`transmute()`, 2) order rows with `arrange()`:\n\n``` r\n# Use `dplyr` functions to do some final tidying\nlong_annual_offences \u003c- wide_annual_offences %\u003e%\n  tidyr::pivot_longer(\n    cols = dplyr::starts_with('count'),\n    values_to = 'count',\n    names_to = 'year',\n    names_prefix = 'count_'\n  ) %\u003e%\n  dplyr::filter(count \u003e 0) %\u003e%\n  dplyr::transmute(\n    year = as.integer(year),\n    offence_code,\n    count\n  ) %\u003e%\n  dplyr::arrange(year, offence_code)\n```\n\n------------------------------------------------------------------------\n\nWhat do they both look like now?\n\n``` r\nhead(annual_offences, 3)\n```\n\n    ## # A tibble: 3 x 3\n    ##    year offence_code count\n    ##   \u003cint\u003e \u003cchr\u003e        \u003cint\u003e\n    ## 1  2016 00101          219\n    ## 2  2016 00304         4730\n    ## 3  2016 00305           28\n\n``` r\nhead(long_annual_offences, 3)\n```\n\n    ## # A tibble: 3 x 3\n    ##    year offence_code count\n    ##   \u003cint\u003e \u003cchr\u003e        \u003cint\u003e\n    ## 1  2016 00101          219\n    ## 2  2016 00304         4730\n    ## 3  2016 00305           28\n\n------------------------------------------------------------------------\n\nSuccess!\n\n``` r\nidentical(long_annual_offences, annual_offences)\n```\n\n    ## [1] TRUE\n\nThere are many additional arguments that can be passed to\n`pivot_wider()` and `pivot_longer()`, which are explained in the\nfunction help files, e.g. `?pivot_wider`. We’ve just covered some of the\nmore basic ones to show how we can easily go between between wide and\nlong format data. Now you can have a go yourself in the exercises below!\n\n------------------------------------------------------------------------\n\n### Exercise 1\n\nYou have received a summary table showing quarterly totals of adult\nreoffenders in England and Wales, beginning in the second quarter of\n2010. The data are split by number of previous offences of the offender\nprior to their current offence.\n\nRead in the data:\n\n``` r\n# Example data for Reshaping exercises\nreoffending_real \u003c- Rs3tools::s3_path_to_full_df(\n    s3_path = \"s3://alpha-r-training/intro-r-extension/adult_reoff_by_prev_off_number_2.csv\")\n```\n\n1)  Examine this data table. Would you describe it as being in wide or\n    long format?\n2)  Is it more ‘machine readable’ or ‘human readable’?\n3)  What, if anything, would you need to do to the data before passing\n    it to be read for plotting by functions from a package like\n    `ggplot2`?\n\nNote, these are real data on reoffending, publicly available, derived\nfrom the table\n[here](https://www.gov.uk/government/statistics/proven-reoffending-statistics-april-to-june-2021).\n\n------------------------------------------------------------------------\n\nHere’s a preview of the data table:\n\n``` r\nhead(reoffending_real)\n```\n\n    ##                    prev_conv_n total_2010_Q2 total_2010_Q3 total_2010_Q4 total_2011_Q1 total_2011_Q2 total_2011_Q3 total_2011_Q4 total_2012_Q1\n    ## 1         No previous offences         42165         42427         41106         40870         39092         39411         37792         36869\n    ## 2     1 to 2 previous offences         26905         27522         26239         26455         25318         25696         24729         24527\n    ## 3     3 to 6 previous offences         24549         25467         24309         24864         24264         24586         23181         23817\n    ## 4    7 to 10 previous offences         13217         13985         13230         13443         13198         13472         12770         13387\n    ## 5 11 or more previous offences         51428         53846         52304         52659         52213         54824         51638         53484\n    ##   total_2012_Q2 total_2012_Q3 total_2012_Q4 total_2013_Q1 total_2013_Q2 total_2013_Q3 total_2013_Q4 total_2014_Q1 total_2014_Q2 total_2014_Q3\n    ## 1         34897         35939         34615         33332         32526         32955         33405         33268         31098         31313\n    ## 2         22546         23663         22255         21621         21157         21776         21151         21265         19631         20082\n    ## 3         21886         22625         21319         21230         20683         21248         20598         20778         19459         19785\n    ## 4         12269         12563         11859         11813         11753         12094         11633         11744         11062         11204\n    ## 5         50015         51921         49219         48893         49361         50603         49040         49920         47523         48618\n    ##   total_2014_Q4 total_2015_Q1 total_2015_Q2 total_2015_Q3 total_2015_Q4 total_2016_Q1 total_2016_Q2 total_2016_Q3 total_2016_Q4 total_2017_Q1\n    ## 1         30775         30587         29624         29254         28663         27813         26888         25753         24828         25662\n    ## 2         19357         19732         18582         18661         18108         17741         16802         16022         15170         15678\n    ## 3         18704         18981         18425         18394         17966         17514         17006         16161         15470         16004\n    ## 4         10731         11105         10838         10580         10357         10262          9962          9546          9069          9453\n    ## 5         46296         46641         45963         45455         45593         45249         44399         42993         41271         43572\n    ##   total_2017_Q2 total_2017_Q3 total_2017_Q4 total_2018_Q1 total_2018_Q2 total_2018_Q3 total_2018_Q4 total_2019_Q1 total_2019_Q2 total_2019_Q3\n    ## 1         23376         22952         23332         23436         21982         21524         21433         22358         21407         21423\n    ## 2         14319         14005         13689         13796         13519         12868         12680         13370         12685         12644\n    ## 3         15123         14499         13986         14359         13846         13474         13145         13635         12932         12937\n    ## 4          8903          8677          8222          8392          8300          7925          7901          8107          7764          7733\n    ## 5         41272         41006         39503         40151         38786         38290         37297         37433         36497         36135\n    ##   total_2019_Q4 total_2020_Q1 total_2020_Q2 total_2020_Q3 total_2020_Q4 total_2021_Q1 total_2021_Q2\n    ## 1         21049         20309          8067         18382         19566         17481         17700\n    ## 2         11886         12057          4992         11783         12637         11599         11323\n    ## 3         12366         12229          5363         12258         13129         12279         12257\n    ## 4          7542          7170          3209          7301          7865          7217          7245\n    ## 5         34274         33492         18066         32028         33953         31702         31657\n\n------------------------------------------------------------------------\n\n### Exercise 2\n\n1)  Put the data into long format using the appropriate function.\n2)  Remove relevant prefixes.\n3)  Pass the labels ‘quarter’ and ‘count’ to the appropriate arguments\n    to name the columns in your long format table.\n\n### Exercise 3\n\nYour project manager likes the resulting plot, but wants to be able to\nsee trends in counts over time more easily. Going from the long format\ntable:\n\n1)  Put the data back into wide format.\n2)  Add a prefix of your choice to the new columns you create.\n3)  Round the values to the nearest thousand.\n\n------------------------------------------------------------------------\n\n------------------------------------------------------------------------\n\n------------------------------------------------------------------------\n\n# String manipulation\n\n## Introduction\n\nIn this chapter we’ll look at strings and some techniques to help work\nwith them, mainly making use of the `stringr` package from Tidyverse.\n\nThere are two ways to create a string in R, by using either single or\ndouble quotes. There is no practical difference in behaviour for the two\noptions, but the convention is to use double quotes (`\"`).\n\n``` r\n# Two options to define a string\nstring1 \u003c- \"a string using double quotes\"\nstring2 \u003c- 'another string using single quotes'\n\nstring1\n```\n\n    ## [1] \"a string using double quotes\"\n\n``` r\nstring2\n```\n\n    ## [1] \"another string using single quotes\"\n\n------------------------------------------------------------------------\n\nThere is an advantage to having two ways to define a string, which is\nthat the two types of quotation marks can be combined for cases when the\nstring itself needs to contain a quotation mark. Here are some examples\nof how to define a string in R:\n\n``` r\n# Some strings containing quotation marks\nstring3 \u003c- \"here is a 'quote' within a string\"\nstring4 \u003c- 'here is a \"quote\" within a string'\n\nstring3\n```\n\n    ## [1] \"here is a 'quote' within a string\"\n\n``` r\nstring4\n```\n\n    ## [1] \"here is a \\\"quote\\\" within a string\"\n\nNotice the difference in how `string4` is displayed - R has added escape\ncharacters (`\\`) before the double quote marks. These escape characters\nchange the behaviour of the following character. In this case it stops\nthe following double quote mark from defining the end of the string, and\ninstead allows it to be a part of the string.\n\n------------------------------------------------------------------------\n\nOften it’s necessary to work with a set of strings in a character\nvector, and in the following sections we’ll look at how various\n`stringr` functions can help us work with character vectors. A new\ncharacter vector can be constructed using the `c()` function that we’ve\nmet before:\n\n``` r\n# Example of a character vector\nstring_vector \u003c- c(\"a\", \"vector\", \"of\", \"strings\")\nstring_vector\n```\n\n    ## [1] \"a\"       \"vector\"  \"of\"      \"strings\"\n\n## String Length\n\nThe first `stringr` function we’ll look at is `str_length()`, which\nsimply returns the length of each string in a character vector:\n\n``` r\n# Find out how many characters are in each string\nstringr::str_length(string_vector)\n```\n\n    ## [1] 1 6 2 7\n\n## Combining Strings\n\nThe `str_c()` function is used to combine multiple strings together,\nwhere each string is included as a separate argument, like so:\n\n``` r\n# Combining several strings into one\nstringr::str_c(\"some\", \"strings\", \"to\", \"combine\")\n```\n\n    ## [1] \"somestringstocombine\"\n\nThere are two optional arguments, `sep` and `collapse` that can be used\nto modify the behaviour of `str_c()`. The `sep` argument allows us to\ndefine a separator to put between the strings when they’re combined:\n\n``` r\n# Using custom separator\nstringr::str_c(\"some\", \"space\", \"separated\", \"strings\", sep=\" \")\n```\n\n    ## [1] \"some space separated strings\"\n\n------------------------------------------------------------------------\n\nThe `str_c()` is especially useful because it is vectorised, and when\napplying it to character vectors the `collapse` argument can be used to\ncombine a vector of strings into a single string:\n\n``` r\n# Collapsing a character vector into a single string\nvector_to_collapse \u003c- c(\"some\", \"strings\", \"to\", \"combine\")\nstringr::str_c(vector_to_collapse, collapse=\"\")\n```\n\n    ## [1] \"somestringstocombine\"\n\nThe value of `collapse` will determine how the collapsed strings are\nseparated.\n\n------------------------------------------------------------------------\n\nYou can input multiple character vectors to `str_c()` and it will\ncombine them together. You can either output another character vector:\n\n``` r\n# Combining two character vectors\nstring_vector1 \u003c- c(\"A\", \"B\", \"C\", \"D\")\nstring_vector2 \u003c- c(\"1\", \"2\", \"3\", \"4\")\nstringr::str_c(string_vector1, string_vector2, sep=\" - \")\n```\n\n    ## [1] \"A - 1\" \"B - 2\" \"C - 3\" \"D - 4\"\n\nOr collapse the vectors into a single string:\n\n``` r\n# Combining and collapsing two character vectors\nstring_vector1 \u003c- c(\"A\", \"B\", \"C\", \"D\")\nstring_vector2 \u003c- c(\"1\", \"2\", \"3\", \"4\")\nstringr::str_c(string_vector1, string_vector2, sep=\" - \", collapse=\" \")\n```\n\n    ## [1] \"A - 1 B - 2 C - 3 D - 4\"\n\n------------------------------------------------------------------------\n\nIt can also combine a single string with a vector of strings, like so:\n\n``` r\n# The single string will be 'recycled' to match the length of the vector\nstringr::str_c(\"a\", c(\"b\", \"c\", \"d\"), sep=\" \")\n```\n\n    ## [1] \"a b\" \"a c\" \"a d\"\n\nCompare this with what happens when we combine these strings with `c()`\n:\n\n``` r\n# Combining strings into a single vector with c() \nc(\"a\", c(\"b\", \"c\", \"d\"))\n```\n\n    ## [1] \"a\" \"b\" \"c\" \"d\"\n\n------------------------------------------------------------------------\n\nIt’s worth noting what happens if you pass vectors of different lengths\nto `str_c()`:\n\n``` r\n# Combining vectors of different lengths\nstring_vector1 \u003c- c(\"A\", \"B\", \"C\")\nstring_vector2 \u003c- c(\"1\", \"2\", \"3\", \"4\", \"5\")\nstringr::str_c(string_vector1, string_vector2, sep=\" - \")\n```\n\n    ## Warning in stri_c(..., sep = sep, collapse = collapse, ignore_null = TRUE): longer object length is not a multiple of shorter object length\n\n    ## [1] \"A - 1\" \"B - 2\" \"C - 3\" \"A - 4\" \"B - 5\"\n\nThe code produces a warning but otherwise runs. In the output, you can\nsee that the elements of the shorter vector have been repeated when\ncombined with the additional elements of the longer vector.\n\n## Extracting and replacing substrings\n\nSelecting part of a string can be done using the `str_sub()` function.\nThe `start` and `end` arguments are used to define the position of the\nsubstring you want to extract.\n\n``` r\n# Extracting substrings based on the position within the string\nx \u003c- c(\"First value\", \"Second value\", \"Third value\")\nstringr::str_sub(x, start=1, end=3)\n```\n\n    ## [1] \"Fir\" \"Sec\" \"Thi\"\n\n``` r\n# Negative values for the start and end count backwards from the end of the string\nstringr::str_sub(x, start=-5, end=-1)\n```\n\n    ## [1] \"value\" \"value\" \"value\"\n\n------------------------------------------------------------------------\n\nYou can also use `str_sub()` to help replace substrings:\n\n``` r\n# Replacing a substring based on the position within the string\nstringr::str_sub(x, start=-5, end=-1) \u003c- \"item\"\nx\n```\n\n    ## [1] \"First item\"  \"Second item\" \"Third item\"\n\n## Detecting a matched pattern\n\nThe `str_detect()` function can be used to check if part of a string\nmatches a particular pattern. For example, let’s say we wanted to check\nif any strings in a character vector contain “blue”:\n\n``` r\n# Detecting the presence of the word 'blue' in a character vector\ncolours \u003c- c(\"scarlet red\", \"ultramarine blue\", \"cadmium red\", \"cobalt blue\", \"cerulean blue\")\nstringr::str_detect(colours, \"blue\")\n```\n\n    ## [1] FALSE  TRUE FALSE  TRUE  TRUE\n\nBecause booleans (`TRUE` or `FALSE`) can be represented as numbers (1 or\n0), you can apply some functions typically used for numbers to boolean\nvectors:\n\n``` r\n# Count how many strings contain 'blue'\nsum(stringr::str_detect(colours, \"blue\"))\n```\n\n    ## [1] 3\n\n------------------------------------------------------------------------\n\nThere can be unintended consequences for pattern matching, let’s say we\nwanted to find strings containing the colour “red” in another character\nvector:\n\n``` r\n# Detecting the presence of the word 'red' in a character vector, with an unintended consequence\ncolours \u003c- c(\"scarlet red\", \"ultramarine blue\", \"cadmium red\", \"cobalt blue\", \"weathered\")\nstringr::str_detect(colours, \"red\")\n```\n\n    ## [1]  TRUE FALSE  TRUE FALSE  TRUE\n\nThere are options to help deal with cases like this that we’ll visit\nlater on.\n\n------------------------------------------------------------------------\n\n### Regular expressions\n\nRegular expressions (regex) are extremely helpful for pattern matching.\nSince regex could be an entire course by itself, here we only introduce\na few basics to get started. See the [further reading](#further-reading)\nsection if you’re interested in learning more about regex.\n\nThere’s a common syntax for defining the patterns to match that can be\nused across multiple programming languages. Here are a few patterns to\nget started with:\n\n- `[A-Za-z]` — All uppercase and lowercase letters\n- `[0-9]` — All numbers\n- `[A-Za-z0-9]` — All letters and all numbers\n- `\\\\s` — A single space\n- `^a` — Begins with ‘a’\n- `a$` — Ends with ‘a’\n- `[^a]` — Anything other than ‘a’\n- `\\\\b` — A word boundary (e.g. a space, punctuation mark or the\n  start/end of a string)\n\nR also contains some pre-built regex classes that you might also\nencounter, for example `[:alpha:]` to match any letters and `[:digit:]`\nto match any numbers.\n\n------------------------------------------------------------------------\n\nWe can use regex to help extract a more general pattern, such as only\nstrings that contain letters:\n\n``` r\n# Detect strings containing any letters using regex\ncolours \u003c- c(\"1.\", \"ultramarine blue\", \"2. cadmium red\", \"cobalt blue\", \"-\")\nstringr::str_detect(colours, \"[A-Za-z]\")\n```\n\n    ## [1] FALSE  TRUE  TRUE  TRUE FALSE\n\nOr only strings that contain letters or numbers:\n\n``` r\nstringr::str_detect(colours, \"[A-Za-z0-9]\")\n```\n\n    ## [1]  TRUE  TRUE  TRUE  TRUE FALSE\n\nOr only strings that contain something other than letters, numbers, and\nspaces:\n\n``` r\nstringr::str_detect(colours, \"[^[A-Za-z0-9\\\\s]]\")\n```\n\n    ## [1]  TRUE FALSE  TRUE FALSE  TRUE\n\n------------------------------------------------------------------------\n\nWe can revisit the example from earlier, where we wanted to identify\nstrings containing the colour “red”:\n\n``` r\n# Detecting the presence of the word 'red' in a character vector, with help from regex\ncolours \u003c- c(\"scarlet red\", \"ultramarine blue\", \"cadmium red\", \"cobalt blue\", \"weathered\")\nstringr::str_detect(colours, \"\\\\bred\\\\b\")\n```\n\n    ## [1]  TRUE FALSE  TRUE FALSE FALSE\n\nThe word boundary regex allows us to exclude words like “weathered” when\nlooking for the word “red”.\n\n## Extracting a matched pattern\n\nWe can use the `str_extract()` function to extract strings that match a\nparticular pattern:\n\n``` r\n# Extracting substrings based on a matched pattern\ncolours \u003c- c(\"scarlet red\", \"ultramarine blue\", \"cadmium red\", \"cobalt blue\", \"cerulean blue\")\nstringr::str_extract(colours, \"blue\")\n```\n\n    ## [1] NA     \"blue\" NA     \"blue\" \"blue\"\n\n## Replacing a matched pattern\n\nYou can use the `str_replace()` and `str_replace_all()` functions to\nfind and replace parts of a string. `str_replace()` replaces the first\ninstance of the pattern, whereas `str_replace_all()` replaces all\ninstances of the pattern. Here’s an alternative version of an example we\nsaw earlier, using a different approach to replace “value” with “item”:\n\n``` r\nx \u003c- c(\"First value\", \"Second value\", \"Third value\")\n# Replace 'value' with 'item'\nstringr::str_replace(x, \"value\", \"item\")\n```\n\n    ## [1] \"First item\"  \"Second item\" \"Third item\"\n\n------------------------------------------------------------------------\n\nRegular expressions are also useful for string replacement. Here’s a\nexample that replaces characters that aren’t letters or numbers with an\nunderscore:\n\n``` r\ncolours \u003c- c(\"scarlet...red\", \"ultramarine.blue\", \"cadmium_red\", \"cobalt blue\", \"cerulean-blue\")\n\n# Replace the first character that isn't a letter or number with an underscore\nstringr::str_replace(colours, \"[^[A-Za-z0-9]]\", \"_\")\n```\n\n    ## [1] \"scarlet_..red\"    \"ultramarine_blue\" \"cadmium_red\"      \"cobalt_blue\"      \"cerulean_blue\"\n\n``` r\n# Replace all characters that aren't a letter or number with an underscore\nstringr::str_replace_all(colours, \"[^[A-Za-z0-9]]\", \"_\")\n```\n\n    ## [1] \"scarlet___red\"    \"ultramarine_blue\" \"cadmium_red\"      \"cobalt_blue\"      \"cerulean_blue\"\n\nIn the first example only the first match in each string has been\nreplaced with an underscore, whereas in the second example all matches\nhave been replaced.\n\n------------------------------------------------------------------------\n\n### Exercise 1\n\nThe column names of a table in an earlier exercise are: `offence_code`,\n`count_2016`, `count_2017`, `count_2018`, `count_2019`, `count_2020`.\n\nUsing only `stringr::str_c()` and `c()`, find the most efficient way to\ncode this from scratch as:\n\n1)  A vector of strings\n2)  A single string, with column names separated by a comma and a space\n\n**Hint:** You may want to create a variable in the first part of the\nquestion, and recycle it for the second part.\n\n### Exercise 2\n\nThe ‘billboard’ data set from the `tidyr` package contains US weekly\nmusic sales data.\n\nExtract its column names with `colnames(tidyr::billboard)`, and return\nan integer corresponding to the number of columns that contain `wk`\n(‘week’) in their names.\n\n**Hint:** You may find the function `stringr::str_detect()` useful here.\n**Hint:** Remember that booleans have numeric value (`TRUE == 1`,\n`FALSE == 0`).\n\n------------------------------------------------------------------------\n\n### Exercise 3\n\nRemove all spaces from the following string:\n\n``` r\nstring \u003c- \"The quick brown fox jumps over the lazy dog.\"\n```\n\n**Hint:** You can remove a matched pattern by replacing it with an empty\nstring (`\"\"`).\n\n------------------------------------------------------------------------\n\n------------------------------------------------------------------------\n\n------------------------------------------------------------------------\n\n# Further Reading\n\n## Further Reading\n\n### General\n\n- [Bonus\n  examples](https://github.com/moj-analytical-services/intro_r_training_extension#bonus-examples)\n- [R for Data Science](https://r4ds.hadley.nz)\n- [Advanced R](https://adv-r.hadley.nz)\n- [Tidyverse website](https://www.tidyverse.org)\n- [Tidyverse style guide](https://style.tidyverse.org/) (has some\n  guidance on choosing function and argument names)\n- [MoJ Analytical Platform\n  Guidance](https://user-guidance.services.alpha.mojanalytics.xyz)\n- [MoJ coding\n  standards](https://moj-analytical-services.github.io/our-coding-standards/)\n\n------------------------------------------------------------------------\n\n### Conditional statements\n\n- [Advanced R - choices\n  section](https://adv-r.hadley.nz/control-flow.html#choices)\n\n### Iteration\n\n- [R for Data Science - iteration\n  chapter](https://r4ds.had.co.nz/iteration.html)\n- [Advanced R - loops\n  section](https://adv-r.hadley.nz/control-flow.html#loops)\n\n### Reshaping data\n\n- [Tidyverse website - pivoting\n  section](https://tidyr.tidyverse.org/dev/articles/pivot.html)\n- [R for Data Science - pivoting\n  chapter](https://r4ds.had.co.nz/tidy-data.html#pivoting)\n\n### Strings and regex\n\n- [R for Data Science - strings\n  chapter](https://r4ds.had.co.nz/strings.html)\n- [stringr and regex\n  cheatsheet](https://evoldyn.gitlab.io/evomics-2018/ref-sheets/R_strings.pdf)\n- [Regex Coffee \u0026 Coding\n  session](https://github.com/moj-analytical-services/Coffee-and-Coding/tree/master/2019-03-13%20Regex)\n\n# Bonus examples\n\nLet’s take a look at a few more examples and tackle some problems that\nwe might encounter as an analyst in MoJ.\n\n## Example 1 - reshaping\n\nLet’s look at an example with some aggregate data based on the\n`offenders` dataset:\n\n``` r\noffenders_summary \u003c- offenders %\u003e%\n  group_by(REGION, SENTENCE) %\u003e%\n  summarise(offender_count = n())\n```\n\n    ## `summarise()` has grouped output by 'REGION'. You can override using the `.groups` argument.\n\n``` r\noffenders_summary\n```\n\n    ## # A tibble: 12 x 3\n    ## # Groups:   REGION [4]\n    ##    REGION SENTENCE    offender_count\n    ##    \u003cchr\u003e  \u003cchr\u003e                \u003cint\u003e\n    ##  1 East   Court_order            211\n    ##  2 East   Prison_\u003c12m            108\n    ##  3 East   Prison_12m+             33\n    ##  4 North  Court_order            219\n    ##  5 North  Prison_\u003c12m             94\n    ##  6 North  Prison_12m+             45\n    ##  7 South  Court_order            235\n    ##  8 South  Prison_\u003c12m            115\n    ##  9 South  Prison_12m+             28\n    ## 10 West   Court_order            191\n    ## 11 West   Prison_\u003c12m            100\n    ## 12 West   Prison_12m+             34\n\n### Transforming from long to wide format\n\nThe above summary dataframe could be described as being in a ‘long’\nformat - where there are minimal columns and lots of rows. This format\ntends not to be used for presenting data, as it is more difficult to\nlook at and interpret. Therefore wider formats are often used to display\ndata, where there are more columns but fewer rows. We can use the\n`pivot_wider()` function from tidyr to help us transform from a long\nformat to a wide format, like so:\n\n``` r\noffenders_summary \u003c- offenders_summary %\u003e%\n  tidyr::pivot_wider(names_from = \"SENTENCE\", values_from = \"offender_count\")\n\noffenders_summary\n```\n\n    ## # A tibble: 4 x 4\n    ## # Groups:   REGION [4]\n    ##   REGION Court_order `Prison_\u003c12m` `Prison_12m+`\n    ##   \u003cchr\u003e        \u003cint\u003e         \u003cint\u003e         \u003cint\u003e\n    ## 1 East           211           108            33\n    ## 2 North          219            94            45\n    ## 3 South          235           115            28\n    ## 4 West           191           100            34\n\nIn the `names_from` argument of `pivot_wider()`, we’ve specifed that we\nwant to create new columns based on the different categories in the\n`SENTENCE` column - so there will be one new column for each of the\nthree categories that appear in `SENTENCE`. Then we use the\n`values_from` argument to specify that we want values from the\n`offender_count` column to go into those new columns.\n\n### Transforming from wide to long format\n\nIn order to reverse the reshaping that we’ve just done, and go back from\na wide format to a long format, we can use the `pivot_longer()`\nfunction:\n\n``` r\noffenders_summary \u003c- offenders_summary %\u003e%\n  tidyr::pivot_longer(cols = -REGION, names_to = \"SENTENCE\", values_to = \"offender_count\")\n\noffenders_summary\n```\n\n    ## # A tibble: 12 x 3\n    ## # Groups:   REGION [4]\n    ##    REGION SENTENCE    offender_count\n    ##    \u003cchr\u003e  \u003cchr\u003e                \u003cint\u003e\n    ##  1 East   Court_order            211\n    ##  2 East   Prison_\u003c12m            108\n    ##  3 East   Prison_12m+             33\n    ##  4 North  Court_order            219\n    ##  5 North  Prison_\u003c12m             94\n    ##  6 North  Prison_12m+             45\n    ##  7 South  Court_order            235\n    ##  8 South  Prison_\u003c12m            115\n    ##  9 South  Prison_12m+             28\n    ## 10 West   Court_order            191\n    ## 11 West   Prison_\u003c12m            100\n    ## 12 West   Prison_12m+             34\n\nThe `cols` argument of `pivot_longer()` has been set to `-REGION`, which\nmeans that all columns apart from `REGION` will be reshaped. Then the\n`names_to` argument is used to specify that we want the names of those\ncolumns to go into a new column called `SENTENCE`, and the `values_to`\nargument is used to specify that we want the values in those column to\ngo into a new column called `offender_count`.\n\n## Example 2\n\n``` r\n# Read data\nprosecutions_and_convictions \u003c- Rs3tools::s3_path_to_full_df(\n  s3_path = \"alpha-r-training/writing-functions-in-r/prosecutions-and-convictions-2018.csv\"\n)\n\n# Filter for Magistrates Court to extract the prosecutions\nprosecutions \u003c- prosecutions_and_convictions %\u003e%\n  filter(`Court.Type` == \"Magistrates Court\")\n```\n\nThe following code is used to prepare a table that will form the basis\nof this example. This table will show the number of prosecutions over\ntime for each offence group.\n\n``` r\n# Create a time series table\ntime_series \u003c- prosecutions %\u003e%\n  group_by(Year, Offence.Type, Offence.Group) %\u003e%\n  summarise(Count = sum(Count)) %\u003e%\n  # Select the past 5 years (to avoid the table being too wide)\n  filter(Year \u003e max(prosecutions$Year) - 5) %\u003e%\n  # Convert from long format to wide format\n  tidyr::pivot_wider(names_from = \"Year\", values_from = \"Count\", values_fill = c(\"Count\" = 0)) %\u003e%\n  arrange(Offence.Type, Offence.Group) %\u003e%\n  ungroup()\n```\n\n    ## `summarise()` has grouped output by 'Year', 'Offence.Type'. You can override using the `.groups` argument.\n\n``` r\n# This removes repeated row labels, to replicate how this data might be displayed in Excel\ntime_series$Offence.Type[duplicated(time_series$Offence.Type)] \u003c- NA\n\ntime_series\n```\n\n    ## # A tibble: 22 x 7\n    ##    Offence.Type       Offence.Group                           `2014` `2015` `2016` `2017` `2018`\n    ##    \u003cchr\u003e              \u003cchr\u003e                                    \u003cint\u003e  \u003cint\u003e  \u003cint\u003e  \u003cint\u003e  \u003cint\u003e\n    ##  1 01 Indictable only 01 Violence against the person            7447   6930   6724   7233   6602\n    ##  2 \u003cNA\u003e               02 Sexual offences                        5289   5743   5610   4941   2930\n    ##  3 \u003cNA\u003e               03 Robbery                                9049   7236   6024   5953   5713\n    ##  4 \u003cNA\u003e               04 Theft Offences                         1726   1465   1265   1345   1097\n    ##  5 \u003cNA\u003e               05 Criminal damage and arson               711    738    647    648    563\n    ##  6 \u003cNA\u003e               06 Drug offences                             0      0     42    211     75\n    ##  7 \u003cNA\u003e               07 Possession of weapons                   729    776    860    776    912\n    ##  8 \u003cNA\u003e               08 Public order offences                    27     68    665    773    842\n    ##  9 \u003cNA\u003e               09 Miscellaneous crimes against society   3648   3054   2930   2763   2277\n    ## 10 \u003cNA\u003e               10 Fraud Offences                          421    460    408    379    173\n    ## # … with 12 more rows\n\nLet’s imagine we received a dataset in the above format, and we wanted\nto calculate the total number of prosecutions over the past 5 years for\neach offence type and group. In the current format, we’d need to sum the\nvalues in the columns `2013` - `2018`. We could do something like this:\n\n``` r\ntotal \u003c- (time_series$`2014` + time_series$`2015` + time_series$`2016` +\n          time_series$`2017` + time_series$`2018`)\n\ntime_series_with_total \u003c- time_series\ntime_series_with_total$Total \u003c- total\n\ntime_series_with_total\n```\n\n    ## # A tibble: 22 x 8\n    ##    Offence.Type       Offence.Group                           `2014` `2015` `2016` `2017` `2018` Total\n    ##    \u003cchr\u003e              \u003cchr\u003e                                    \u003cint\u003e  \u003cint\u003e  \u003cint\u003e  \u003cint\u003e  \u003cint\u003e \u003cint\u003e\n    ##  1 01 Indictable only 01 Violence against the person            7447   6930   6724   7233   6602 34936\n    ##  2 \u003cNA\u003e               02 Sexual offences                        5289   5743   5610   4941   2930 24513\n    ##  3 \u003cNA\u003e               03 Robbery                                9049   7236   6024   5953   5713 33975\n    ##  4 \u003cNA\u003e               04 Theft Offences                         1726   1465   1265   1345   1097  6898\n    ##  5 \u003cNA\u003e               05 Criminal damage and arson               711    738    647    648    563  3307\n    ##  6 \u003cNA\u003e               06 Drug offences                             0      0     42    211     75   328\n    ##  7 \u003cNA\u003e               07 Possession of weapons                   729    776    860    776    912  4053\n    ##  8 \u003cNA\u003e               08 Public order offences                    27     68    665    773    842  2375\n    ##  9 \u003cNA\u003e               09 Miscellaneous crimes against society   3648   3054   2930   2763   2277 14672\n    ## 10 \u003cNA\u003e               10 Fraud Offences                          421    460    408    379    173  1841\n    ## # … with 12 more rows\n\nBut what if we want to re-use the code in the future? We’d need to\ngeneralise it for different years or a different number of years.\nFortunately we can restructure the data to help with this problem.\n\nFirst let’s deal with the empty row labels in the `Offence.Type` column.\nAlthough avoiding repeated row labels looks neater in an Excel table, it\ncan be problematic for analysis. Fortunately we can easily fill the row\nlabels in using the `fill()` function from tidyr:\n\n``` r\ntime_series \u003c- time_series %\u003e% tidyr::fill(Offence.Type)\ntime_series\n```\n\n    ## # A tibble: 22 x 7\n    ##    Offence.Type       Offence.Group                           `2014` `2015` `2016` `2017` `2018`\n    ##    \u003cchr\u003e              \u003cchr\u003e                                    \u003cint\u003e  \u003cint\u003e  \u003cint\u003e  \u003cint\u003e  \u003cint\u003e\n    ##  1 01 Indictable only 01 Violence against the person            7447   6930   6724   7233   6602\n    ##  2 01 Indictable only 02 Sexual offences                        5289   5743   5610   4941   2930\n    ##  3 01 Indictable only 03 Robbery                                9049   7236   6024   5953   5713\n    ##  4 01 Indictable only 04 Theft Offences                         1726   1465   1265   1345   1097\n    ##  5 01 Indictable only 05 Criminal damage and arson               711    738    647    648    563\n    ##  6 01 Indictable only 06 Drug offences                             0      0     42    211     75\n    ##  7 01 Indictable only 07 Possession of weapons                   729    776    860    776    912\n    ##  8 01 Indictable only 08 Public order offences                    27     68    665    773    842\n    ##  9 01 Indictable only 09 Miscellaneous crimes against society   3648   3054   2930   2763   2277\n    ## 10 01 Indictable only 10 Fraud Offences                          421    460    408    379    173\n    ## # … with 12 more rows\n\nNow we need to transform this dataframe into a long format, using\n`pivot_longer()`:\n\n``` r\ntime_series_long \u003c- time_series %\u003e%\n  tidyr::pivot_longer(cols = -c(\"Offence.Type\", \"Offence.Group\"), names_to = \"year\", values_to = \"count\")\n\ntime_series_long\n```\n\n    ## # A tibble: 110 x 4\n    ##    Offence.Type       Offence.Group                  year  count\n    ##    \u003cchr\u003e              \u003cchr\u003e                          \u003cchr\u003e \u003cint\u003e\n    ##  1 01 Indictable only 01 Violence against the person 2014   7447\n    ##  2 01 Indictable only 01 Violence against the person 2015   6930\n    ##  3 01 Indictable only 01 Violence against the person 2016   6724\n    ##  4 01 Indictable only 01 Violence against the person 2017   7233\n    ##  5 01 Indictable only 01 Violence against the person 2018   6602\n    ##  6 01 Indictable only 02 Sexual offences             2014   5289\n    ##  7 01 Indictable only 02 Sexual offences             2015   5743\n    ##  8 01 Indictable only 02 Sexual offences             2016   5610\n    ##  9 01 Indictable only 02 Sexual offences             2017   4941\n    ## 10 01 Indictable only 02 Sexual offences             2018   2930\n    ## # … with 100 more rows\n\nNow we’re ready to find the total for each offence group using\n`group_by()` and `summarise()` from dplyr:\n\n``` r\ntotals \u003c- time_series_long %\u003e%\n  group_by(Offence.Type, Offence.Group) %\u003e%\n  summarise(Total = sum(count))\n```\n\n    ## `summarise()` has grouped output by 'Offence.Type'. You can override using the `.groups` argument.\n\n``` r\ntotals\n```\n\n    ## # A tibble: 22 x 3\n    ## # Groups:   Offence.Type [5]\n    ##    Offence.Type       Offence.Group                           Total\n    ##    \u003cchr\u003e              \u003cchr\u003e                                   \u003cint\u003e\n    ##  1 01 Indictable only 01 Violence against the person          34936\n    ##  2 01 Indictable only 02 Sexual offences                      24513\n    ##  3 01 Indictable only 03 Robbery                              33975\n    ##  4 01 Indictable only 04 Theft Offences                        6898\n    ##  5 01 Indictable only 05 Criminal damage and arson             3307\n    ##  6 01 Indictable only 06 Drug offences                          328\n    ##  7 01 Indictable only 07 Possession of weapons                 4053\n    ##  8 01 Indictable only 08 Public order offences                 2375\n    ##  9 01 Indictable only 09 Miscellaneous crimes against society 14672\n    ## 10 01 Indictable only 10 Fraud Offences                        1841\n    ## # … with 12 more rows\n\nIf we wanted to add these totals to our original dataframe, we can use\n`left_join()` from dplyr:\n\n``` r\ntime_series \u003c- dplyr::left_join(time_series, totals, by=c(\"Offence.Type\", \"Offence.Group\"))\n\ntime_series\n```\n\n    ## # A tibble: 22 x 8\n    ##    Offence.Type       Offence.Group                           `2014` `2015` `2016` `2017` `2018` Total\n    ##    \u003cchr\u003e              \u003cchr\u003e                                    \u003cint\u003e  \u003cint\u003e  \u003cint\u003e  \u003cint\u003e  \u003cint\u003e \u003cint\u003e\n    ##  1 01 Indictable only 01 Violence against the person            7447   6930   6724   7233   6602 34936\n    ##  2 01 Indictable only 02 Sexual offences                        5289   5743   5610   4941   2930 24513\n    ##  3 01 Indictable only 03 Robbery                                9049   7236   6024   5953   5713 33975\n    ##  4 01 Indictable only 04 Theft Offences                         1726   1465   1265   1345   1097  6898\n    ##  5 01 Indictable only 05 Criminal damage and arson               711    738    647    648    563  3307\n    ##  6 01 Indictable only 06 Drug offences                             0      0     42    211     75   328\n    ##  7 01 Indictable only 07 Possession of weapons                   729    776    860    776    912  4053\n    ##  8 01 Indictable only 08 Public order offences                    27     68    665    773    842  2375\n    ##  9 01 Indictable only 09 Miscellaneous crimes against society   3648   3054   2930   2763   2277 14672\n    ## 10 01 Indictable only 10 Fraud Offences                          421    460    408    379    173  1841\n    ## # … with 12 more rows\n\nNow we’ve managed to calculate the total number of prosecutions over the\npast 5 years, without needing to hard-code the names of those years.\nThis means that the code can be re-used in future years without needing\nto be edited.\n\n# Appendix\n\n## Table of operators\n\n| Operator | Definition                    |\n|:--------:|:------------------------------|\n|    ==    | Equal to                      |\n|    !=    | Not equal to                  |\n|    \\\u003e    | Greater than                  |\n|    \\\u003c    | Less than                     |\n|   \\\u003e=    | Greater than or equal to      |\n|   \\\u003c=    | Less than or equal to         |\n|    ǀ     | Or                            |\n|    \u0026     | And                           |\n|    !     | Not                           |\n|   %in%   | The subject appears in a list |\n| is.na()  | The subject is NA             |\n\n## Regex patterns\n\n`[A-Za-z]` or `[:alpha:]` \\| All uppercase and lowercase letters  \n`[0-9]` or `[:digit:]` \\| All numbers  \n`[A-Za-z0-9]` or `[:alnum:]` \\| All letters and all numbers  \n`\\\\s` or `[:space:]` \\| A single space  \n`^a` \\| Begins with ‘a’  \n`a$` \\| Ends with ‘a’  \n`[^a]` \\| Anything other than ‘a’  \n`\\\\b` \\| A word boundary (e.g. a space, punctuation mark or the\nstart/end of a string)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoj-analytical-services%2Fintro_r_training_extension","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmoj-analytical-services%2Fintro_r_training_extension","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoj-analytical-services%2Fintro_r_training_extension/lists"}