  ---
title: "Exploring a real dataset"
subtitle: "Exploratory Data Analysis"
format:
  html: 
    code-overflow: wrap
    toc: true
    toc-title: "Table of Contents"
include-in-header:
  - text: |
      <style>
      .cell-output-stdout code {
        word-break: break-wor !important;
        white-space: pre-wrap !important;
      }
      </style>
execute:
  warning: false
  message: false
---

This set of exercises exposes you to several important aspects of data science. 

1. This is a big dataset (relatively speaking)
2. It's been touched by many hands, and the artifacts show.  
3. It involves download, import as well as installing tools specific for this dataset.

# ATUS Data

Learn more and extract different data below: [American Time Use Survey](https://www.atusdata.org/atus/)

Our ATUS data concerns sex, age, race, stress & work

## Download

*Download* the `codebook`, the `ddi` file and the `zip` file. [ddi](http://euclid.nmu.edu/~joshthom/teaching/dat309/week2/ATUS2/atus_00002.xml) [codebook](http://euclid.nmu.edu/~joshthom/teaching/dat309/week2/ATUS2/codebook2.txt) [.zip](http://euclid.nmu.edu/~joshthom/teaching/dat309/week2/ATUS2/atus_00002.dat.zip)

*Prepare* to download the data into your Dat309 folder (not Samba!), or a suitable sub-directory. Make sure RStudio can see your newly downloaded files.  Click `Session > Set Working Directory > Choose Directory` 

![working director](setwd.png)

Or, do it by hand using these commands. 
```{r directories}
#| eval: FALSE
# get the working diretory
getwd()
# set the working directory
# note: Windows users, use forward slash \
setwd("enter your directory here")
```


Use the codebook to parse meaning from the variables. You need the `ipumsr` package to load the data into R.

```{r}
#| eval: FALSE

# haven contains functions that covert the data into an r-friendly format
install.packages("haven")
install.packages("ipumsr")
```

```{r load data}
#| eval: TRUE
# load ATUS data
# The data requires the IPUMSr package
library(haven)
library(ipumsr)
ddi <- read_ipums_ddi("ATUS2/atus_00002.xml")
data <- read_ipums_micro(ddi)
detach("package:haven", unload=TRUE)
```

## Exploratory Questions

(i) What is the size of the data? How many rows & columns? (Use `dim()`, for dimension.)
(ii) What do the numbers mean in the rows?
(iii) What are the variable names and what do they mean?

The (very useful) `clean_names()` function makes the variable names a bit easier to read.

```{r filtering}
# filter to get stress data
#| eval: TRUE
library(tidyverse)
library(janitor)
ds <- clean_names(data)
ds <- filter(ds,scstress < 10)
```

The variable pertaining to "kind of job" was poorly named, so we change it.

```{r rename}
# rename work variable 
#| eval: FALSE
ds <- filter(ds,scstress < 10)
ds <- rename(ds,"job" = occ2_cps8)

```

## Filter

Choose a few jobs so the data isn't so big. Learn what how the numbers relate to jobs in the [codebook](ATUS2/codebook2.txt).

```{r filter and plot}
#| eval: FALSE
dmsf <- filter(ds,job == 120 | job == 122 |job == 132)
ggplot(dmsf,aes(x=scstress)) + geom_bar()
```

## Visualizing Distributions & Relationships

[See examples in text](https://r4ds.hadley.nz/data-visualize#visualizing-distributions) 

1. Categorical distribution: bar plot 
2. Numerical distribution: histogram, density plot
3. Numerical distribution vs. categorical variable: box plot
4. Two categorical variables: barplot filled with color or the same with `position = "fill"` in the geom.
5. Two-Three numerical variables: scatter plot with colors mapped to a variable. (see [text](https://r4ds.hadley.nz/data-visualize#three-or-more-variables))

6. Just a few categorical variables? Try [faceting](https://r4ds.hadley.nz/layers.html#facets):

**Exercise**
1. Improve the plot below with `fill = as_factor(job)` from the `haven` package.  
2. Add a `position = "fill"` to the geom_bar().  Does it help?

```{r}
dmsf <- filter(ds,job == 120 | job == 122 |job == 132)
ggplot(dmsf,aes(x=scstress, fill = job)) + geom_bar() +
  labs(fill = "Job")
```



## Factors

(i) Remove `factor()` from the `fill = factor(job)` and observe the result.
(ii) Replace it with `fill = haven::as_factor(job)` library) and observe.  The syntax above is a way of using the `as_factor` function without loading the Haven library that has recently caused problems with data-typing.

## Exercises: Due: Tuesday 9/9
(i) Create a plot indicating how the `happy` and `stress` variables correlated.   
(ii) Textbook: (https://r4ds.hadley.nz/data-visualize#exercises-2) 
