---
title: "Load & Wrangle Zillow Data"
subtitle: "Intro loops & functions"
theme: flatly 
date: today
toc: true
format:
  html: 
    code-overflow: wrap
    toc: true
    toc-title: "Contents"
    # hide code?

execute:
  warning: false
  message: false
---

## Library load

```{r library}
library(tidyverse)
library(tidyquant) # contains coord_x_datetime for dates as coordinates
library(scales)
```

This data, from [Zillow](https://www.zillow.com/research/data/), comes to us in two files; one associated with yts (zillow home value index) and one with nrin (total monthly payment).  We well join the datasets together so that each house is an observation and nrin and yts are variables.  

## 0. load data
```{r loading data}


# Years to Save: A measure of the number of years it would take the median household to save for a 20% down payment on a home, assuming they are able to save 10% of their income into a simple savings account accruing no interest. This is equivalent to the number of years it would take the median household to save for a 10% down payment, assuming a 5% savings rate.
yts <- read_csv("https://euclid.nmu.edu/~joshthom/teaching/dat309/week8/Metro_years_to_save_downpayment_0.20_uc_sfrcondo_tier_0.33_0.67_sm_sa_month.csv") |> janitor::clean_names()

# New Renter Income Needed: An estimate of the household income required to spend less than 30% of monthly income to newly lease the typical rental.
nrin <- read_csv("https://euclid.nmu.edu/~joshthom/teaching/dat309/week8/Metro_new_renter_income_needed_uc_sfrcondomfr_sm_sa_month.csv") |> janitor::clean_names()

print(yts, n = 5)
print(nrin, n = 5) 
```
## 1. rename x's with yts and nrin, respectively

First, we rename variables to make it easier to interpret the nrin & yts price variables.  We have `rename_with` use the `str_replace` function to perform a renaming of many variables at once.  

  - The `~` syntax makes `str_replace` into a function whose argument.
  - The data is sent to the `.` spot.  
  - Thy syntax is str_replace(data, thing_to_replace, new_string)

```{r rename}
yts <- yts |> rename_with(~str_replace(.,"x","yts_"))
nrin <- nrin |> rename_with(~str_replace(.,"x","nrin_"))
```


## 2. use names() and head() to note the 1st 10 variables
```{r prepare to merge (join)}
nrin |> names() |> head(n=10)
yts |> names() |> head()
```

## 3. Joining 101: 

There are several ways of joining two data sets together, some are illustrated below.

![joins](https://tavareshugo.github.io/r-intro-tidyverse-gapminder/fig/07-dplyr_joins.svg)


### join two data sets with `full_join`

Joining two data frames using "left_join" or "right_join" requires a variable common to both data frames and keeps data.  We use `full_join` to bring in new (non-redundant) variables.

Learn more about joining [here.](https://dplyr.tidyverse.org/reference/mutate-joins.html#ref-examples) 


```{r join}
nrin_yts <- full_join(yts,nrin)
```


## 5. pivot to get plotable data
```{r pivot}
# 
# ytsing price & nrining price are variables 
zillow <- pivot_longer(nrin_yts,
                      cols = starts_with("yts"),
                      names_to = "yts_date",
                      values_to = "years_to_save") |> # before buying an avg house
  pivot_longer(
    cols = starts_with("nrin"),
    names_to = "nrin_date",
    values_to = "income_needed") # to rent @ < 30% 

zillow <- zillow |> 
  mutate(
    nrin_date = as_datetime(str_remove(nrin_date,"nrin_")),
    yts_date = as_datetime(str_remove(yts_date,"yts_"))
    ) |> 
  relocate(nrin_date,.before=1) |>
  drop_na()
```

## 7. filter to get some reasonable sized data
```{r function}
# parameter "name" : vector of strings of state names: c("CO","MI")
# parameter "number" : the number of regions to pull from each states
get_states <- function(name, number){
  
  # empty dataset
  d <- {}
  for (i in 1:length(name)){
    # print the name of the state for sanity
    print(name[i])
    
    # get all data for the given name
    my_states <- filter(zillow, state_name %in% name[i])
    
    # grab a few region_names from each state
    some_names <- distinct(my_states,region_name) |> 
      slice_sample(n = number)
    
    some_regions <- filter(my_states, 
                           region_name %in% pull(some_names))
    # could also do some_names[[1]] to extract the vector from the tibble
    
    d <- bind_rows(d,some_regions)
  }
  return(d)
}

zil_sub <- get_states(c("NC","MI","NY","WA", "WY"),3)
```

## 8. plot years to save before buyint average home for each region
```{r plot p}
p <- ggplot(zil_sub, aes(x = yts_date, y = years_to_save,
                         group = region_name,
                         color = region_name)) + geom_line()

print(p)
``` 
## 9. Plot income needed so that an average rental costs less than 30% of your income for each region

```{r plotting}
# option 1: plot alongside the yts plot
q <- ggplot(zil_sub, aes(x = nrin_date, y = income_needed,
                         group = region_name,
                         color = region_name)) + geom_line()

print(q)
```

Note that the  `as_date()` or `as_datetime()` functions can be used to cast strings into variables of type `date` or `dttm`

```{r}
str <- "2019-01-01 00:14:15"
as_datetime(str)
```

This works a bit ebetter:
```{r plotting better}
p <- ggplot(zil_sub, aes(x = nrin_date, y = income_needed,
                         group = region_name,
                         color = region_name,
                         linetype = state_name)) +
      geom_line()

```

This is even better: 
```{r even better}
# zoom to the interesting region 
p + scale_x_date(date_breaks = "2 years")
#p + coord_x_datetime(xlim = c("2018-01-01", "2023-12-31")) 
```

```{r better still}
# even better still
p + scale_x_date(date_breaks = "1 year", date_labels= "%Y")
```


Zoom in on post 2018.  Here we use `coord_x_date` to showcase how to zoom without losing data.  (The `coord_` functions allow for zooming without removing data that might be used to compute a statistic) 

```{r zoom}
p + scale_x_date(
  date_breaks = "1 year", 
  date_labels= "%Y") +
  theme(axis.text.x = element_text(size = 5)) + 
  coord_x_date(xlim = c("2018-01-01","2024-01-01"))
```

Get better formatting on the y-axis
```{r y axis}
p + scale_x_date(
      date_breaks = "1 year", 
      date_labels= "%Y") + 
    theme(axis.text.x = element_text(size = 5)) +
    labs(x = "Year", y = "Income Needed to Buy Average Home") +
    scale_y_continuous(label = label_dollar(prefix = "$"))
```


## 10. zoom to the interesting region
```{r zooming}
p + scale_x_date(
      date_breaks = "1 year", 
      date_labels= "%Y",
      limits = 
      c(as_date("2018-01-01"), as_date("2023-12-31")))  +
    theme(axis.text.x = element_text(size = 5)) +
    scale_y_continuous(label = label_dollar(prefix = "$")) + 
    labs(title = "Income Needed",
    subtitle = "To Lease an Average Rental")
```
## Save data for future work

This saves the variable `zil_sub` to the filename indicated below.  Load it back up in R with `load("zillow-data.Rdata")`.

```{r save}
#| eval: FALSE
write_rds(zil_sub, file = "~/t/dat309/week7/zillow-data.Rdata")
```
