How to summarize data

summarize(), group_by(), slice_(), across()

Creating summaries of data is a very common practice in data science. One way to get a summary is to use the base R summary()

mtcars |> summary()

      mpg             cyl             disp             hp       
 Min.   :10.40   Min.   :4.000   Min.   : 71.1   Min.   : 52.0  
 1st Qu.:15.43   1st Qu.:4.000   1st Qu.:120.8   1st Qu.: 96.5  
 Median :19.20   Median :6.000   Median :196.3   Median :123.0  
 Mean   :20.09   Mean   :6.188   Mean   :230.7   Mean   :146.7  
 3rd Qu.:22.80   3rd Qu.:8.000   3rd Qu.:326.0   3rd Qu.:180.0  
 Max.   :33.90   Max.   :8.000   Max.   :472.0   Max.   :335.0  
      drat             wt             qsec             vs        
 Min.   :2.760   Min.   :1.513   Min.   :14.50   Min.   :0.0000  
 1st Qu.:3.080   1st Qu.:2.581   1st Qu.:16.89   1st Qu.:0.0000  
 Median :3.695   Median :3.325   Median :17.71   Median :0.0000  
 Mean   :3.597   Mean   :3.217   Mean   :17.85   Mean   :0.4375  
 3rd Qu.:3.920   3rd Qu.:3.610   3rd Qu.:18.90   3rd Qu.:1.0000  
 Max.   :4.930   Max.   :5.424   Max.   :22.90   Max.   :1.0000  
       am              gear            carb      
 Min.   :0.0000   Min.   :3.000   Min.   :1.000  
 1st Qu.:0.0000   1st Qu.:3.000   1st Qu.:2.000  
 Median :0.0000   Median :4.000   Median :2.000  
 Mean   :0.4062   Mean   :3.688   Mean   :2.812  
 3rd Qu.:1.0000   3rd Qu.:4.000   3rd Qu.:4.000  
 Max.   :1.0000   Max.   :5.000   Max.   :8.000

A more useful tool is in the tidyverse. We begin with

group_by()

The main way to summarize data begins with group_by().

It shuffles a dataset into groups, so that subsequent analysis is done “by group”.

# important: group_by does not change how the data looks
library(tidyverse)
library(nycflights13)
flights |> 
  group_by(year,month,day) |> 
  summarize(mx = max(arr_delay,na.rm=TRUE)) |> 
  arrange((mx)) |> 
  print(n=30)

# A tibble: 365 × 4
# Groups:   year, month [12]
    year month   day    mx
   <int> <int> <int> <dbl>
 1  2013     5    26   102
 2  2013     6    20   130
 3  2013     9     7   135
 4  2013    11    19   140
 5  2013     9    24   142
 6  2013    11    30   144
 7  2013     2    28   145
 8  2013     9     4   148
 9  2013     9     9   153
10  2013     9    15   157
11  2013    11    15   159
12  2013     9    14   161
13  2013     1    19   162
14  2013     3    26   166
15  2013     2     4   169
16  2013    10    31   169
17  2013     9    23   170
18  2013    10     4   171
19  2013    10    21   173
20  2013     4     3   174
21  2013     1     6   175
22  2013     3    11   180
23  2013    11    13   180
24  2013    11    16   182
25  2013     1     8   184
26  2013     1    15   187
27  2013     8    26   187
28  2013     3    30   189
29  2013    10     5   189
30  2013     5     5   190
# ℹ 335 more rows

group_by() does not remove or add data, it just adds a layer of grouping that is used by subsequent functions

# important: group_by does not change how the data looks
diamonds |> group_by(cut)

# A tibble: 53,940 × 10
# Groups:   cut [5]
   carat cut       color clarity depth table price     x     y     z
   <dbl> <ord>     <ord> <ord>   <dbl> <dbl> <int> <dbl> <dbl> <dbl>
 1  0.23 Ideal     E     SI2      61.5    55   326  3.95  3.98  2.43
 2  0.21 Premium   E     SI1      59.8    61   326  3.89  3.84  2.31
 3  0.23 Good      E     VS1      56.9    65   327  4.05  4.07  2.31
 4  0.29 Premium   I     VS2      62.4    58   334  4.2   4.23  2.63
 5  0.31 Good      J     SI2      63.3    58   335  4.34  4.35  2.75
 6  0.24 Very Good J     VVS2     62.8    57   336  3.94  3.96  2.48
 7  0.24 Very Good I     VVS1     62.3    57   336  3.95  3.98  2.47
 8  0.26 Very Good H     SI1      61.9    55   337  4.07  4.11  2.53
 9  0.22 Fair      E     VS2      65.1    61   337  3.87  3.78  2.49
10  0.23 Very Good H     VS1      59.4    61   338  4     4.05  2.39
# ℹ 53,930 more rows

Ungrouping & `.by()`

Occasionally you may want to remove grouping. Use ungroup() to do this. You may wish to ungroup because you want to do a summary of the entire dataset, not by group. You may also wish to ungroup because you want to do a summary of a different grouping.

To remove grouping use ungroup(), and to do “in-line” grouping on a per-operation basis you can use .by()

library(palmerpenguins)
penguins |> summarize(
  mx_bd = max(bill_depth_mm), 
  n = n(), 
  .by = species)

# A tibble: 3 × 3
  species   mx_bd     n
  <fct>     <dbl> <int>
1 Adelie     NA     152
2 Gentoo     NA     124
3 Chinstrap  20.8    68

using `summarize()` with `group_by()`

group_by is often used in tandem with summarize

summarize() transforms your data table into one or more rows, where each row constitutes a summary of the corresponding data.

Find the max of a single variable.

library(tidyverse)
library(palmerpenguins)
penguins |> summarize(mx_bd = max(bill_depth_mm))

# A tibble: 1 × 1
  mx_bd
  <dbl>
1    NA

We notice `NA values in the bill_depth_mm variable. We can remove them with na.rm = TRUE.

Find the max of single variable within each group.

penguins |> group_by(species) |>
  summarize(mx_bd = max(bill_depth_mm, na.rm = TRUE))

# A tibble: 3 × 2
  species   mx_bd
  <fct>     <dbl>
1 Adelie     21.5
2 Chinstrap  20.8
3 Gentoo     17.3

Improve your summary with `n = n()`. Every summary should include this.

penguins |> group_by(species) |>
  summarize(mx_bd = max(bill_depth_mm, na.rm = TRUE), n = n())

# A tibble: 3 × 3
  species   mx_bd     n
  <fct>     <dbl> <int>
1 Adelie     21.5   152
2 Chinstrap  20.8    68
3 Gentoo     17.3   124

Exercise: What’s wrong with this code?

penguins |> group_by(species) |>
  summarize(mx_bd = max(bill_depth_mm, na.rm = TRUE, n = n()))

# A tibble: 3 × 2
  species   mx_bd
  <fct>     <dbl>
1 Adelie      152
2 Chinstrap    68
3 Gentoo      124

Find the number of observations (rows) within each group.

penguins |> group_by(species) |> summarize(n = n())

# A tibble: 3 × 2
  species       n
  <fct>     <int>
1 Adelie      152
2 Chinstrap    68
3 Gentoo      124

cut()

Here we show how to use cut() to categorize continuous data into groups. This is often useful for summarizing data.

Explore the diamonds dataset by summarizing additional values.
Arrange the output by cut.
Interpret the output. Does anything surprise you?

  diamonds |> group_by(carat) |>
    summarize(
      mean_price = mean(price),
    )

# A tibble: 273 × 2
   carat mean_price
   <dbl>      <dbl>
 1  0.2        365.
 2  0.21       380.
 3  0.22       391.
 4  0.23       486.
 5  0.24       505.
 6  0.25       551.
 7  0.26       551.
 8  0.27       575.
 9  0.28       580.
10  0.29       601.
# ℹ 263 more rows

Grouping numerical values & summarizing

Create a new variable in the `diamonds` data using `cut()` that

gives the relative size, eg., “small”, “medium”, etc. The quantile() function may be helpful:

quantile(diamonds$carat)

  0%  25%  50%  75% 100% 
0.20 0.40 0.70 1.04 5.01

Group the data on this newly created variable and create a summary of the price (mean, max, min, etc.) for each group.

`across()`

Apply a function to several variables in a data frame.

# change character variables to factors
mpg |> mutate(across(where(is.character),
                     as.factor)) |> glimpse()

Rows: 234
Columns: 11
$ manufacturer <fct> audi, audi, audi, audi, audi, audi, audi, audi, audi, aud…
$ model        <fct> a4, a4, a4, a4, a4, a4, a4, a4 quattro, a4 quattro, a4 qu…
$ displ        <dbl> 1.8, 1.8, 2.0, 2.0, 2.8, 2.8, 3.1, 1.8, 1.8, 2.0, 2.0, 2.…
$ year         <int> 1999, 1999, 2008, 2008, 1999, 1999, 2008, 1999, 1999, 200…
$ cyl          <int> 4, 4, 4, 4, 6, 6, 6, 4, 4, 4, 4, 6, 6, 6, 6, 6, 6, 8, 8, …
$ trans        <fct> auto(l5), manual(m5), manual(m6), auto(av), auto(l5), man…
$ drv          <fct> f, f, f, f, f, f, f, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, r, …
$ cty          <int> 18, 21, 20, 21, 16, 18, 18, 18, 16, 20, 19, 15, 17, 17, 1…
$ hwy          <int> 29, 29, 31, 30, 26, 26, 27, 26, 25, 28, 27, 25, 25, 25, 2…
$ fl           <fct> p, p, p, p, p, p, p, p, p, p, p, p, p, p, p, p, p, p, r, …
$ class        <fct> compact, compact, compact, compact, compact, compact, com…

You can also use across inside of summarise.

# compute the grouped-mean of all numeric variables
diamonds |> group_by(cut) |>
  summarise(across(where(is.numeric),
            mean))

# A tibble: 5 × 8
  cut       carat depth table price     x     y     z
  <ord>     <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 Fair      1.05   64.0  59.1 4359.  6.25  6.18  3.98
2 Good      0.849  62.4  58.7 3929.  5.84  5.85  3.64
3 Very Good 0.806  61.8  58.0 3982.  5.74  5.77  3.56
4 Premium   0.892  61.3  58.7 4584.  5.97  5.94  3.65
5 Ideal     0.703  61.7  56.0 3458.  5.51  5.52  3.40

# compute the mean of several variable
mpg |> group_by(manufacturer) |>
  summarize(
    across(where(is.numeric) & contains("y") & !contains("year"),
           mean))

# A tibble: 15 × 4
   manufacturer   cyl   cty   hwy
   <chr>        <dbl> <dbl> <dbl>
 1 audi          5.22  17.6  26.4
 2 chevrolet     7.26  15    21.9
 3 dodge         7.08  13.1  17.9
 4 ford          7.2   14    19.4
 5 honda         4     24.4  32.6
 6 hyundai       4.86  18.6  26.9
 7 jeep          7.25  13.5  17.6
 8 land rover    8     11.5  16.5
 9 lincoln       8     11.3  17  
10 mercury       7     13.2  18  
11 nissan        5.54  18.1  24.6
12 pontiac       6.4   17    26.4
13 subaru        4     19.3  25.6
14 toyota        5.12  18.5  24.9
15 volkswagen    4.59  20.9  29.2

Here’s another use of across

mtcars |> mutate(across(c(1,5), round)) |> head()

                  mpg cyl disp  hp drat    wt  qsec vs am gear carb
Mazda RX4          21   6  160 110    4 2.620 16.46  0  1    4    4
Mazda RX4 Wag      21   6  160 110    4 2.875 17.02  0  1    4    4
Datsun 710         23   4  108  93    4 2.320 18.61  1  1    4    1
Hornet 4 Drive     21   6  258 110    3 3.215 19.44  1  0    3    1
Hornet Sportabout  19   8  360 175    3 3.440 17.02  0  0    3    2
Valiant            18   6  225 105    3 3.460 20.22  1  0    3    1

`slice`

The slice() functions are used to select rows from a data frame. They are often used in conjunction with group_by() and summarize(). The slice functions are shown below.

The slice_ functions

df |> slice_head(n = 1) takes the first row from each group.
df |> slice_tail(n = 1) takes the last row in each group.
df |> slice_min(x, n = 1) takes the row with the smallest value of column x.
df |> slice_max(x, n = 1) takes the row with the largest value of column x.
df |> slice_sample(n = 1) takes one random row.

bind_rows

This has nothing to do with grouping, but it is an operation on a table.

Glue two tables together using bind_rows. Experiment to learn how different variables are handled.