---
title: "Working with Numbers"
subtitle: "using dplyr"
format: 
  html:
    code-fold: FALSE
toc: TRUE
theme: flatly
warning: FALSE
message: FALSE
---

Standard techniques for working with numerical data in R using the `dplyr` package.

## Make Numbers From  Strings


### `parse_double`

```{r}
library(tidyverse)
x <- c("1.2", "5.6", "1e3")
parse_double(x)
#> [1]    1.2    5.6 1000.0
```

For numbers with formatting (like currency symbols, commas, or percentage signs), use `parse_number()`:

```{r}

x <- c("$1,234", "USD 3,513", "59%")
parse_number(x)
#> [1] 1234 3513   59
```

A (perhaps surprising) amount of data science can be done with simple tools like counts & summaries.

## Counts

Often used with `group_by()`, `count()` is a convenient way to get the number of observations in each group.

### `count()`

```{r}
# how many rows are in the diamonds dataset
count(diamonds)

# how many fair diamonds are there?
count(diamonds, cut = "Fair")

# note, the code above does not throw an error, but it does not do what you want.  It's an example of something an AI might suggest that is incorrect.  The correct code is below:
count(diamonds, cut)

# or if you really ONLY wanted the "Fair" cut diamonds
count(diamonds, cut) |> filter(cut == "Fair")
```


To count unique values, or unique combinations `dplyr` provides `n_distinct()`.

### `n_distinct()`

```{r}
# how many unique cuts are there?
n_distinct(diamonds$cut)

# it's often used inside summarize()
# number of models per manufacturer
mpg |> group_by(manufacturer) |> 
  summarize(n_models = n_distinct(model)) |> 
  arrange(desc(n_models))
```

## Numeric Transformations

Below are some common ways to produce new numbers from existing ones.

### recycling
R recycles vectors.  It can be a useful feature, but can also lead to unexpected results.  For example:

```{r}
x <- seq(1,20,by = 2)
y <- c(1,2)

# explain x/y
```

### Max / Min

The arithmetic functions work with pairs of variables. Two closely related functions are `pmin()` and `pmax()`, which when given two or more variables will return the smallest or largest value in each row:

```{r}
df <- tribble(
  ~x, ~y,
  1,  3,
  5,  2,
  7, NA,
)

# notice the difference between min/max and pmin/pmax
df |> 
  mutate(
    min = min(x, y, na.rm = TRUE),
    max = max(x, y, na.rm = TRUE),
    row_min = pmin(x, y, na.rm = TRUE),
    row_max = pmax(x, y, na.rm = TRUE)
  )
```

## Modular arithmetic

Look at the format of `dep_time`.  We can see that `517` means the time `5:17`.  Use modular arithmetic `%%` (the remainder after division) and integer division `%/%` (divide, but ignore remainder) to produce new variables of hour and minute. 

```{r, echo = FALSE}
library(nycflights13)
library(tidyverse)
flights |> glimpse()
```

```{r}

# divide but drop remainder 
hour <- flights$sched_dep_time %/% 100

# the remainder of 517 after dividing by 100 is 17
min <- flights$sched_dep_time %% 100
```
Recall, to find the proportion of cancelled flights per month we can do:

```{r}
flights |> group_by(month) |> 
  summarize(prop_cancelled = mean(is.na(dep_time)))
```
Use this idea with our new arithmetic tools to see how the proportion of cancelled flights varies throughout the day.

```{r}
flights |> 
  # create a variable hour from sched_dep_time and group by it.
  group_by(hour = sched_dep_time %/% 100) |> 
  summarize(prop_cancelled = mean(is.na(dep_time)), n = n()) |> 
  filter(hour > 1) |> 
  ggplot(aes(x = hour, y = prop_cancelled)) +
  geom_line(color = "grey") + 
  geom_point(aes(size = n)) + 
  labs(title = 
         "Proportion of Cancelled Flights Throughout the Day") + 
  theme_minimal()

```

## Rounding

`round()` uses what’s known as “round half to even” or Banker’s rounding: if a number is half way between two integers, it will be rounded to the even integer. This is a good strategy because it keeps the rounding unbiased: half of all 0.5s are rounded up, and half are rounded down.

```{r}
round(c(1.5, 2.5))
```

Here's some common uses of rounding.

```{r}
x <- 123.456
# Round down to nearest two digits
floor(x / 0.01) * 0.01

# Round up to nearest two digits
ceiling(x / 0.01) * 0.01

# Round to nearest multiple of 4
round(x / 4) * 4

# Round to nearest 0.25
round(x / 0.25) * 0.25
```

## cut

Use `cut()` to break up (aka bin) a numeric vector into discrete buckets:

```{r}
x <- c(1, 2, 5, 10, 15, 20)
cut(x, breaks = c(0, 5, 10, 15, 20))
```

The breaks don’t need to be evenly spaced:

```{r}
cut(x, breaks = c(0, 5, 10, 100))
```


You can optionally supply your own labels. Note that there should be one less labels than breaks.

```{r}
cut(x, 
  breaks = c(0, 5, 10, 15, 20), 
  labels = c("sm", "md", "lg", "xl")
)
```

**Exercise:** Add a new variable to the `diamonds` data describing the price of each diamond.

## Cumulative and rolling aggregates

Base R provides `cumsum(), cumprod(), cummin(), cummax()` for running, or cumulative, sums, products, mins and maxes. dplyr provides `cummean()` for cumulative means. Cumulative sums tend to come up the most in practice:

```{r}
x <- 1:10
cumsum(x)
#>  [1]  1  3  6 10 15 21 28 36 45 55
```

## Ranks

To assign a ranking use `min_rank()`

```{r}
x <- c(10, 2, 12, 3, 4, NA)
min_rank(x)
```
  
Note that the smallest values get the lowest ranks; use desc(x) to give the largest values the smallest ranks:

```{r}
min_rank(desc(x))
```

### row number
`row_number()` can also be used without any arguments when inside a dplyr verb. In this case, it’ll give the number of the “current” row. When combined with `%%` or `%/%` this can be a useful tool for dividing data into similarly sized groups:

```{r}
df <- tibble(id = 1:10)

df |> 
  mutate(
    row0 = row_number() - 1,
    three_groups = row0 %% 3,
    three_in_each_group = row0 %/% 3
  )

```

**Exercise:** Use this idea to place *randomly* place everyone in this room into groups of 2.  

### Consecutive Identifiers

Sometimes you want to start a new group every time some event occurs. For example, when you’re looking at website data, it’s common to want to break up events into sessions, where you begin a new session after gap of more than x minutes since the last activity. For example, imagine you have the times when someone visited a website:

```{r}
events <- tibble(
  time = c(0, 1, 2, 3, 5, 10, 12, 15, 17, 19, 20, 27, 28, 30)
)
```

And you’ve computed the time between each event, and figured out if there’s a gap that’s big enough to qualify:

```{r}
events <- events |> 
  mutate(
    diff = time - lag(time, default = first(time)),
    has_gap = diff >= 5
  )
events
```

Use `cumsum()` to go from the boolean vector above to a numeric vector that you can `group_by()`.

```{r}
events |> mutate(
  group = cumsum(has_gap)
)
```
Now you can use `group_by(group)` to analyze each session separately.
```{r}
events |> 
  mutate(
    group = cumsum(has_gap)
  ) |> 
  group_by(group) |> 
  summarize(
    start = min(time),
    end = max(time),
    n_events = n()
  )
```

### max, min & quantiles

`quantile()`  is a generalization of the median: `quantile(x, 0.25)` will find the value of x that is greater than 25% of the values, `quantile(x, 0.5)` is equivalent to the median, and `quantile(x, 0.95)` will find the value that’s greater than 95% of the values.

For the flights data, you might want to look at the 95% quantile of delays rather than the maximum, because it will ignore the 5% of most delayed flights which can be quite extreme.

```{r}
flights |>
  group_by(year, month, day) |>
  summarize(
    max = max(dep_delay, na.rm = TRUE),
    q95 = quantile(dep_delay, 0.95, na.rm = TRUE),
    num_badly_delayed = sum(dep_delay > q95, na.rm = TRUE),
    .groups = "drop"
  )
```

### standardizing data (normalizing)

1. `x / sum(x)` calculates the proportion of a total.

2. `(x - mean(x)) / sd(x)` computes a Z-score (standardized to mean 0 and sd 1).

3. `(x - min(x)) / (max(x) - min(x))` standardizes to range [0, 1].

4. `x / first(x)` computes an index based on the first observation.

5. `x / lag(x)` computes period-over-period growth rates.

6. `x / lead(x)` computes period-over-period decay rates.

7. `x / nth(x, n)` computes an index based on the nth observation.


