---
title: "Load & Wrangle Zillow Data"
subtitle: "Intro loops & functions"
theme: flatly 
date: today
toc: true
format:
  html: 
    code-overflow: wrap
    toc: true
    toc-title: "Contents"
    # hide code?

execute:
  warning: false
  message: false
---

## Library load

```{r library}
library(tidyverse)
library(tidyquant) # contains coord_x_datetime for dates as coordinates
library(scales)
```

This data, from [Zillow](https://www.zillow.com/research/data/), comes to us in two files; one associated with selling price and one with listing price.  We well join the datasets together so that each house is an observation and list price and sell price are variables.  

## 0. load data
```{r loading data}
sell <- read_csv("https://euclid.nmu.edu/~joshthom/Teaching/DAT309/Week9/Metro_invt_fs_uc_sfrcondo_sm_month_sell.csv") |> janitor::clean_names()

list <- read_csv("http://euclid.nmu.edu/~joshthom/Teaching/DAT309/Week9/Metro_new_listings_uc_sfrcondo_sm_month_list.csv") |> janitor::clean_names()

print(sell, n = 5)
print(list, n = 5) 
```

## 1. rename x's with sell and list, respectively

First, we rename variables to make it easier to interpret the list & sell price variables.  We have `rename_with` use the `str_replace` function to perform a renaming of many variables at once.  

  - The `~` syntax makes `str_replace` into a function whose argument.
  - The data is sent to the `.` spot.  
  - Thy syntax is str_replace(data, thing_to_replace, new_string)

```{r rename}
sell <- sell |> rename_with(~str_replace(.,"x","sell_"))
list <- list |> rename_with(~str_replace(.,"x","list_"))
```


## 2. use names() and head() to note the 1st 10 variables
```{r prepare to merge (join)}
list |> names() |> head(n=10)
sell |> names() |> head()
```

## 3. Joining 101: 

There are several ways of joining two data sets together, some are illustrated below.

![joins](https://tavareshugo.github.io/r-intro-tidyverse-gapminder/fig/07-dplyr_joins.svg)


### join two data sets with `full_join`

Joining two data frames using "left_join" or "right_join" requires a variable common to both data frames and keeps data.  We use `full_join` to bring in new (non-redundant) variables.

Learn more about joining [here.](https://dplyr.tidyverse.org/reference/mutate-joins.html#ref-examples) 


```{r join}
list_sell <- full_join(sell,list)
```


## 5. pivot to get plotable data
```{r pivot}
# 
# selling price & listing price are variables 
zillow <- pivot_longer(list_sell,
                      cols = starts_with("sell"),
                      names_to = "sell_date",
                      values_to = "sell_price") |> 
  pivot_longer(
    cols = starts_with("list"),
    names_to = "list_date",
    values_to = "list_price")

zillow <- zillow |> 
  mutate(
    list_date = as_datetime(str_remove(list_date,"list_")),
    sell_date = as_datetime(str_remove(sell_date,"sell_"))
    ) |> 
  relocate(list_date,.before=1)
```

## 7. filter to get some reasonable sized data
```{r function}
# parameter "name" : vector of strings of state names: c("CO","MI")
# parameter "number" : the number of regions to pull from each states
get_states <- function(name, number){
  
  # empty dataset
  d <- {}
  for (i in 1:length(name)){
    # print the name of the state for sanity
    print(name[i])
    
    # get all data for the given name
    my_states <- filter(zillow, state_name %in% name[i])
    
    # grab a few region_names from each state
    some_names <- distinct(my_states,region_name) |> 
      slice_sample(n = number)
    
    some_regions <- filter(my_states, 
                           region_name %in% pull(some_names))
    # could also do some_names[[1]] to extract the vector from the tibble
    
    d <- bind_rows(d,some_regions)
  }
  return(d)
}

zil_sub <- get_states(c("CO","MI","AL","FL"),3)
```

## 8. plot prices for each region
```{r plot p}
p <- ggplot(zil_sub, aes(x = sell_date, y = sell_price,
                         group = region_name,
                         color = region_name)) + geom_line()

print(p)
``` 
## 9. plot list prices for each region

```{r plotting}
# option 1: plot alongside the sell plot
q <- ggplot(zil_sub, aes(x = list_date, y = list_price,
                         group = region_name,
                         color = region_name)) + geom_line()

print(q)
```


This works better:
```{r plotting better}
zz <- mutate(zil_sub, 
             sell_date = as_date(sell_date),
             list_date = as_date(list_date))
             
p <- ggplot(zz, aes(x = sell_date, y = sell_price,
                         group = region_name,
                         color = region_name)) + 
  geom_line(linetype = 1) + 
  geom_line(aes(x = list_date, y = list_price,
                group = region_name,
                color = region_name), linetype = 2)

```

This is even better: 
```{r even better}
# zoom to the interesting region 
p + scale_x_date(date_breaks = "2 years")
#p + coord_x_datetime(xlim = c("2018-01-01", "2023-12-31")) 
```

```{r better still}
# even better still
p + scale_x_date(date_breaks = "1 year", date_labels= "%Y")
```


Zoom in on post 2018 
```{r zoom}
p + scale_x_date(date_breaks = "1 year", date_labels= "%Y") + theme(axis.text.x = element_text(size = 5))
```

Get better formatting on the y-axis
```{r y axis}
p + scale_x_date(
      date_breaks = "1 year", 
      date_labels= "%Y") + 
    theme(axis.text.x = element_text(size = 5)) +
    scale_y_continuous(label = label_dollar(prefix = "$"))
```


## 10. zoom to the interesting region
```{r zooming}
p + scale_x_date(
      date_breaks = "1 year", 
      date_labels= "%Y",
      limits = 
      c(as_date("2018-01-01"), as_date("2023-12-31")))  +
    theme(axis.text.x = element_text(size = 5)) +
    scale_y_continuous(label = label_dollar(prefix = "$")) + 
    labs(title = "Selling Price vs. Listing Price",
    subtitle = "Dotted line is listing price and solid line is selling price")
```
## Save data for future work

This saves the variable `zil_sub` to the filename indicated below.  Load it back up in R with `load("zillow-data.Rdata")`.

```{r save}
#| eval: FALSE
save(zil_sub, file = "zillow-data.Rdata")
```

