Exploratory Data Analysis in R

This repo houses lecture notes used in an Exploratory Data Analysis in R course taught to undergraduates at Colby College. The course assumes little to no background in quantitative analysis nor in computer programming and was first taught in Spring, 2015. The course introduces students to data manipulation in R, data exploration (in the spirit of John Tukey’s EDA) and the R markdown language. Many of the visualization tools are adopted from William Cleveland’s Data Visualization book.

Week 1

Course introduction

The R vs RStudio

The R/RStudio environments

Command line vs. script file
The assignment operator
Packages
- Base packages
- Installing packages from CRAN
- Installing packages from GitHub
- Using a package in a R session
Getting a session’s info
A brief video intro to the RStudio environment

Maintaining a tidy project environment

Some useful links

For R programming questions, a wonderful (peer-reviewed) forum is http://stackoverflow.com/ (add the [r] label in the search box to limit the search to R related topics).
For statistical questions, search the http://stats.stackexchange.com forum (here too add the [r] label in the search box to limit the search to R related topics)
A very nice repository of R related blogs is http://R-bloggers.com
If you are within the UvA domain, you can access the Use R! online books series at http://link.springer.com/bookseries/6991

Week 2

Data object type and structure

Data types (aka mode) in R: numeric, character, factor, logical, dates
Data objects in R: vector, data frame, lists
Coercing data from one type to another

How to read and create data files in R

Reading/writing files: comma delimited (CSV), tab delimited, R files
Data can be loaded from a file residing on your local drive or on the web
Importing files from other formats such as Excel, STATA and SAS

Working with Date objects

Creating date/time objects using lubridate
Extracting data information
Concatenating vectors with paste
Operating on dates
Formatting date objects

Extracting and replacing values using base functions

The subset function
Extracting values using indices
Replacing values using indices

Relational and boolean operations

The relational operators: ==, !=, <, <=, > and >=
The boolean operators &, | and !
The identical() function
The match operator %in%

Example: Cleaning a buoy data table

Exploring the table with dim, names, head, str and summary
Deleting columns
Replacing values with NA

Week 3

Manipulating dataframes with `dplyr`

Subsetting by row values: filter
Subsetting columns: select
Computing and/or adding columns: mutate
Sorting by column values: arrange
Combining operations using the pipe %>% operator
Conditional statements ifelse, if_else, case_when and recode

Working with string objects

Finding patterns in a string
Modifying strings
Extracting parts of a string

Week 4

Grouping and summarizing tables

grouping by variable(s): group_by
summarize: summarise

Tidying/reshaping tables using `tidyr`

From long to wide: spread
From wide to long: gather
Combining columns: unite
Splitting columns: separate
Adding missing records: complete

Joining data tables

Left join
Right join
Inner join
Full join

Example of a few data manipulation workflows

These are data objects created for use in subsequent tutorials

Week 5

The base plotting environment

Point and line plots
Boxplots
Histograms
Density plot
Point and line symbols
Exporting to files (tiff and PDF)

The `lattice` plotting environment (optional)

Conditioning on factors
Displaying univariate data
Displaying multivariate data
Customizing trellis plots

The `ggplot` plotting environment

Line geometry geom_line
Point geometry geom_point
Boxplot geometry geom_boxplot
Histogram geometry geom_histograms
Violin plot geometry geom_violin
Combining geometries and layers
Faceting plots (trellis plots)

Manipulating colors in R

The rgb() function

Week 6

The R Markdown language

R Markdown basics
An R markdown file example: Buoy data report
Additional resources (external site)

Visualizing univariate data

Boxplots
Histograms
Quantile plots

Comparing univariate data distributions

Side-by-side boxplots
Quantile-quantile plots (q-q)
Tukey mean-difference plots (m-d)

Week 7

The theoretical q-q plot

Imposing a structure: the normal distribution

Week 8

Fits and residuals

Fitting univariate data
Extracting the residuals
Residual-fit spread plot

Spread-location plot

Detecting changes in the spread
Interpreting and s-l plot

Week 9

Re-expressing data

Log transformation
Box-Cox family of transformations
How quantile plots behave in the face of skewed data

Letter value summaries

Constructing letter value summaries
Interpreting letter value summaries

The Two R’s of EDA

Robustness
Re-expression

Week 10

Bivariate analysis

Scatter plots
Fitting the data
- Parametric fit
- LOESS fit
Residuals
- Residual dependence plot
- spread-location plot

Resistant lines

Robust lines
- Tukey’s 3-point summary
- Bisquare estimation method
Robust loess

Week 11

The third R of EDA: Residuals

Exploring atmospheric CO2 data

Detecting discontinuities in the data

Slicing data
Changepoint detection

Week 12

Two-way tables

Median polish
Mean polish and the two-way ANOVA analysis

Week 13

Creating maps in R

Using maps package datasets
Loading custom shapefiles
continuous vs. discrete color schemes

Relational databases

Querying with SQL
Querying with dplyr

Software version

This site was built with R 4.0.4 and the following packages:

##  package      * version  date       lib source        
##  assertthat     0.2.1    2019-03-21 [1] CRAN (R 4.0.0)
##  brio           1.1.1    2021-01-20 [1] CRAN (R 4.0.3)
##  callr          3.5.1    2020-10-13 [1] CRAN (R 4.0.3)
##  cellranger     1.1.0    2016-07-27 [1] CRAN (R 4.0.0)
##  cli            2.4.0    2021-04-05 [1] CRAN (R 4.0.4)
##  colorspace     2.0-0    2020-11-11 [1] CRAN (R 4.0.3)
##  cpp11          0.2.6    2021-01-29 [1] CRAN (R 4.0.3)
##  crayon         1.4.1    2021-02-08 [1] CRAN (R 4.0.3)
##  desc           1.2.0    2018-05-01 [1] CRAN (R 4.0.0)
##  diffobj        0.3.3    2021-01-07 [1] CRAN (R 4.0.3)
##  digest         0.6.27   2020-10-24 [1] CRAN (R 4.0.3)
##  dplyr          1.0.5    2021-03-05 [1] CRAN (R 4.0.5)
##  ellipsis       0.3.1    2020-05-15 [1] CRAN (R 4.0.3)
##  evaluate       0.14     2019-05-28 [1] CRAN (R 4.0.0)
##  fansi          0.4.2    2021-01-15 [1] CRAN (R 4.0.3)
##  farver         2.0.3    2020-01-16 [1] CRAN (R 4.0.0)
##  generics       0.1.0    2020-10-31 [1] CRAN (R 4.0.3)
##  ggplot2        3.3.3    2020-12-30 [1] CRAN (R 4.0.3)
##  glue           1.4.2    2020-08-27 [1] CRAN (R 4.0.3)
##  gtable         0.3.0    2019-03-25 [1] CRAN (R 4.0.0)
##  hms            1.0.0    2021-01-13 [1] CRAN (R 4.0.3)
##  isoband        0.2.3    2020-12-01 [1] CRAN (R 4.0.3)
##  jsonlite       1.7.2    2020-12-09 [1] CRAN (R 4.0.3)
##  labeling       0.4.2    2020-10-20 [1] CRAN (R 4.0.3)
##  lattice        0.20-41  2020-04-02 [1] CRAN (R 4.0.3)
##  lifecycle      1.0.0    2021-02-15 [1] CRAN (R 4.0.4)
##  lubridate      1.7.10   2021-02-26 [1] CRAN (R 4.0.4)
##  magrittr       2.0.1    2020-11-17 [1] CRAN (R 4.0.3)
##  MASS           7.3-53.1 2021-02-12 [1] CRAN (R 4.0.4)
##  Matrix         1.3-2    2021-01-06 [2] CRAN (R 4.0.4)
##  mgcv           1.8-34   2021-02-16 [1] CRAN (R 4.0.4)
##  munsell        0.5.0    2018-06-12 [1] CRAN (R 4.0.0)
##  nlme           3.1-152  2021-02-04 [2] CRAN (R 4.0.4)
##  pillar         1.5.1    2021-03-05 [1] CRAN (R 4.0.5)
##  pkgbuild       1.2.0    2020-12-15 [1] CRAN (R 4.0.3)
##  pkgconfig      2.0.3    2019-09-22 [1] CRAN (R 4.0.0)
##  pkgload        1.2.0    2021-02-23 [1] CRAN (R 4.0.4)
##  praise         1.0.0    2015-08-11 [1] CRAN (R 4.0.0)
##  prettyunits    1.1.1    2020-01-24 [1] CRAN (R 4.0.0)
##  processx       3.4.5    2020-11-30 [1] CRAN (R 4.0.2)
##  progress       1.2.2    2019-05-16 [1] CRAN (R 4.0.0)
##  ps             1.5.0    2020-12-05 [1] CRAN (R 4.0.3)
##  purrr          0.3.4    2020-04-17 [1] CRAN (R 4.0.0)
##  R6             2.5.0    2020-10-28 [1] CRAN (R 4.0.3)
##  RColorBrewer   1.1-2    2014-12-07 [1] CRAN (R 4.0.3)
##  Rcpp           1.0.6    2021-01-15 [1] CRAN (R 4.0.2)
##  readxl         1.3.1    2019-03-13 [1] CRAN (R 4.0.0)
##  rematch        1.0.1    2016-04-21 [1] CRAN (R 4.0.0)
##  rematch2       2.1.2    2020-05-01 [1] CRAN (R 4.0.2)
##  rlang          0.4.10   2020-12-30 [1] CRAN (R 4.0.3)
##  rprojroot      2.0.2    2020-11-15 [1] CRAN (R 4.0.3)
##  rstudioapi     0.13     2020-11-12 [1] CRAN (R 4.0.3)
##  scales         1.1.1    2020-05-11 [1] CRAN (R 4.0.0)
##  stringi        1.5.3    2020-09-09 [1] CRAN (R 4.0.3)
##  stringr        1.4.0    2019-02-10 [1] CRAN (R 4.0.0)
##  testthat       3.0.2    2021-02-14 [1] CRAN (R 4.0.4)
##  tibble         3.1.0    2021-02-25 [1] CRAN (R 4.0.4)
##  tidyr          1.1.2    2020-08-27 [1] CRAN (R 4.0.3)
##  tidyselect     1.1.0    2020-05-11 [1] CRAN (R 4.0.0)
##  utf8           1.2.1    2021-03-12 [1] CRAN (R 4.0.5)
##  vctrs          0.3.7    2021-03-29 [1] CRAN (R 4.0.5)
##  viridisLite    0.3.0    2018-02-01 [1] CRAN (R 4.0.0)
##  waldo          0.2.4    2021-02-11 [1] CRAN (R 4.0.3)
##  withr          2.4.1    2021-01-26 [1] CRAN (R 4.0.3)
## 
## [1] C:/Users/vanloon/Documents/R/win-library/4.0
## [2] C:/Program Files/R/R-4.0.4/library