--- title: "Chapter 3: Row & Column Operations" toc-title: "arrange,distinct,mutate,select" date: "`r Sys.Date()`" toc: true format: html: code-overflow: wrap toc: true toc-title: "Table of Contents" # hide code? code-fold: false # link at top to globally unhide code code-tools: true # link functions used in code chunks to online docs code-link: true execute: warning: false message: false --- Working with data inevitably leads to the need to filter out subsets, to sort data based on some rule and to find the unique observations. We often need to create new variables as modifications of existing ones. Some variables are often more important than others, thus we need to select them. Finally, renaming and maving variables to within view often makes working with data easier. The tools you learn here make all this possible. They are all contained in `dplyr`, which is included in the `tidyverse`, so as usual: ```{r, message=FALSE} #| label: load_tidyverse library(tidyverse) ``` | Row Functions | Column Functions | |----------|---------:| | filter | mutate | | arrange | select | | distinct | rename | | | relocate| | | select | : Data Transformations {.striped .hover} # Row Functions These functions change which rows are present in the data and keep the columns the same. ## Filter observations based on conditions Subset oberservations (rows) based on a logical condition. Textbook link: [filter](https://r4ds.hadley.nz/data-transform#arrange) ![](filter.png){width=50%} ## Arrange rows by value in ascending order Textbook link: [arrange](https://r4ds.hadley.nz/data-transform#arrange) ![](arrange.png){width=30%} This permutes the ordering of the rows. (**Note the r in arrange, r for rows.**) If you provide more than one column name, each additional column will be used to break ties in the values of preceding columns. To arrange in descending order use `desc()`. ```{r eval=FALSE} #| label: arrange # arrange by bill length in ascending order penguins |> arrange(bill_len) # arrange by bill length in descending order penguins |> arrange(-bill_len) # arrange by two variables penguins |> arrange(bill_len, bill_dep) # penguins |> arrange(bill_len/bill_dep) ``` Pipe to `head()` if you only want to see the first few results: ```{r} #| label: arrange_head # arrange by two variables penguins |> arrange(bill_len, bill_dep) |> head() # arrange by a mathematical combination of variables penguins |> arrange(bill_len/bill_dep) |> head() ``` ## Keep distinct rows [](https://r4ds.hadley.nz/data-transform#distinct) Use `distinct` on several column names to find unique combinations. The `.keep_all = TRUE` argument is used to retain all columsns ```{r} #| label: distinct # this keeps all unique (origin, dest) pairs pak::pkg_install("nycflights13") library(nycflights13) flights |> distinct(origin,dest) # combine with other dplyr functions flights |> distinct(origin,dest) |> arrange(origin) ``` # Column functions ## Create new variables ![](mutate.png){width=30%} Textbook link: [mutate](https://r4ds.hadley.nz/data-transform#sec-mutate) Add new variables (columns), usually via a formula involving existing ones. Use these helper functions to relocate or newly created variables. (i) `.before = 1` (ii) `.after = some_var_name` ```{r} #| label: mutate library(palmerpenguins) penguins |> mutate(bill_ratio = bill_length_mm / bill_depth_mm, .after = island) ``` ## Select The function `select()` is used to choose which columns you want to keep. You can use the helper functions `starts_with()`, `ends_with()`, `contains()`, or `num_range()` to select columns based on their names. You can also use `where()` to select columns based on their type (numeric, factor, character). This is useful if you have too many columns and you with choose which columns you wish to view. Textbook link: [select](https://r4ds.hadley.nz/data-transform#sec-select) ![](select.png){width=30%} ### Select tips & helper functions i) ***use : to select a range*** ```{r} #| label: select_range penguins |> select(species:bill_depth_mm) |> head() penguins |> select(1:3) |> head() ``` ii) ***use ! to exclude*** ```{r} #| label: select_exclude penguins |> select(-c(species, island)) |> head() ``` iii) ***use `where` with `is.factor`, or `is.numeric`, or `is.character`*** Example ```{r} #| label: select_where # Select all numeric columns penguins |> select(where(is.numeric)) |> head() # general rule of thumb: use `where()` inside `select()`, `mutate(across(...))` or `pick()` # and apply where() to is.numeric, is.factor, or is.character ``` iv) ***`starts_with("abc")`: matches names that begin with “`abc`”.*** Example ```{r} #| label: select_starts_with" # Select columns that start with "bill" penguins |> select(starts_with("bill")) |> head() # general rule of thumb: use `starts_with()`, `ends_with()`, `contains()`, or `num_range()` to select columns based on their names # you do not combine these with `where()`, they are used directly inside `select()` ``` v) ***`ends_with("xyz")`: matches names that end with “`xyz`”.*** vi) ***`contains("ijk")`: matches names that contain “`ijk`”.*** vii) ***`num_range("x", 1:3)`: matches `x1, x2` and `x3`.*** Example ```{r} #| label: select_num_range # select a range of columns df <- penguins |> mutate( b1 = round(bill_length_mm), b2 = bill_depth_mm^2, b3 = bill_length_mm + bill_depth_mm, .before = 1) df |> select(num_range("b", c(1,3))) |> head() ``` viii) ***To drop colummns: `select(data,-c(this_col,that_col))`*** Example ```{r} #| label: select_drop #| echo: true penguins |> select(-c(species, island)) |> head() ``` ix) ***use regular expressions: `select(matches("^b.*mm"))`*** Note: `^b.*mm` is a regular expression that matches a `b` followed by anything `.` any number times `*` followed by `mm`. See [regex101.com](https://regex101.com/) ### How `where()` works - You pass a function (like is.numeric, is.factor, or is.character) into where(). - It evaluates that function on every column in your data frame. - It keeps only the columns where the function returns TRUE #### Common Use Cases You typically use `where()` inside `select()`, `mutate(across(...))` or `pick()` ***Select all numeric columns*** ```{r} #| label: select_numeric df %>% select(where(is.numeric)) ``` ***Apply a transformation to all factor columns*** ```{r} #| label: mutate_factor_to_character df %>% mutate(across(where(is.factor), as.character)) ``` ***Use custom conditions with formula shortcuts*** ```{r} #| label: select_numeric_mean_gt_10 df %>% select(where(~ is.numeric(.x) & mean(.x, na.rm = TRUE) > 10)) ``` ### Make sure you know how to: Use `select()` and these helper functions to create various subsets of penguins data. ## [rename](https://r4ds.hadley.nz/data-transform#rename) Explicitly rename variables. Do so in bulk with `janitor::clean_names` ```{r} #| label: rename penguins |> rename(bill_length = bill_length_mm, bill_depth = bill_depth_mm) ``` ## [relocate](https://r4ds.hadley.nz/data-transform#relocate) Permute the ordering of columns (notice the c in relocate, c for columns) You can also achieve this with `select()` ```{r eval=FALSE} relocate(penguins, sex, year) # versus select(penguins, sex,year, everything()) ``` # Assignment-4 - Due in the Samba folder - Wednesday, 9/16 - Complete the following exercises: - Row verb exercises: - Column verb exercises: