gadenbuie/tidyexplain

🤹‍♀ Animations of tidyverse verbs using R, the tidyverse, and gganimate

Tidy Animated Verbs

Garrick Aden-Buie – @grrrck – garrickadenbuie.com

Thanks to contributions from …

• Tyler Grant Smith contributed set operations animations.
• Lukas Wallrich and Kelsey Gonzalez helped create animations of tidyr’s pivoting functions.

Animations:

• Mutating Joins — inner_join(), left_join(), right_join(), full_join()

• Filtering Joins — semi_join(), anti_join()

• Set Operations — union(), union_all(), intersect(), setdiff()

• Tidy Data — pivot_wider() and pivot_longer(), spread() and gather()

• Learn more about

  • Using the animations and images
  • Relational Data
  • gganimate

Background

Usage

Please feel free to use these images for teaching or learning about action verbs from the tidyverse. You can directly download the original animations or static images in svg or png formats, or you can use the scripts to recreate the images locally.

Currently, the animations cover the dplyr two-table verbs and I’d like to expand the animations to include more verbs from the tidyverse. Suggestions are welcome!

Relational Data

The Relational Data chapter of the R for Data Science book by Garrett Grolemund and Hadley Wickham is an excellent resource for learning more about relational data.

The dplyr two-table verbs vignette and Jenny Bryan’s Cheatsheet for dplyr join functions are also great resources.

gganimate

The animations were made possible by the newly re-written gganimate package by Thomas Lin Pedersen (original by Dave Robinson). The package readme provides an excellent (and quick) introduction to gganimate.

Dynamic Animations

Thanks to an initial push by David Zimmermann, we have begun work towards functions that generate dynamic animations from users’ actual data. Please visit the pkg branch of the tidyexplain repository for more information (or to contribute!).

Mutating Joins

A mutating join allows you to combine variables from two tables. It first matches observations by their keys, then copies across variables from one table to the other.
R for Data Science: Mutating joins

x
#> # A tibble: 3 × 2
#>      id x    
#>   <int> <chr>
#> 1     1 x1   
#> 2     2 x2   
#> 3     3 x3
y
#> # A tibble: 3 × 2
#>      id y    
#>   <int> <chr>
#> 1     1 y1   
#> 2     2 y2   
#> 3     4 y4

Inner Join

All rows from x where there are matching values in y, and all columns from x and y.

inner_join(x, y, by = "id")
#> # A tibble: 2 × 3
#>      id x     y    
#>   <int> <chr> <chr>
#> 1     1 x1    y1   
#> 2     2 x2    y2

Left Join

All rows from x, and all columns from x and y. Rows in x with no match in y will have NA values in the new columns.

left_join(x, y, by = "id")
#> # A tibble: 3 × 3
#>      id x     y    
#>   <int> <chr> <chr>
#> 1     1 x1    y1   
#> 2     2 x2    y2   
#> 3     3 x3    <NA>

Left Join (Extra Rows in y)

… If there are multiple matches between x and y, all combinations of the matches are returned.

y_extra # has multiple rows with the key from `x`
#> # A tibble: 4 × 2
#>      id y    
#>   <dbl> <chr>
#> 1     1 y1   
#> 2     2 y2   
#> 3     4 y4   
#> 4     2 y5
left_join(x, y_extra, by = "id")
#> # A tibble: 4 × 3
#>      id x     y    
#>   <dbl> <chr> <chr>
#> 1     1 x1    y1   
#> 2     2 x2    y2   
#> 3     2 x2    y5   
#> 4     3 x3    <NA>

Right Join

All rows from y, and all columns from x and y. Rows in y with no match in x will have NA values in the new columns.

right_join(x, y, by = "id")
#> # A tibble: 3 × 3
#>      id x     y    
#>   <int> <chr> <chr>
#> 1     1 x1    y1   
#> 2     2 x2    y2   
#> 3     4 <NA>  y4

Full Join

All rows and all columns from both x and y. Where there are not matching values, returns NA for the one missing.

full_join(x, y, by = "id")
#> # A tibble: 4 × 3
#>      id x     y    
#>   <int> <chr> <chr>
#> 1     1 x1    y1   
#> 2     2 x2    y2   
#> 3     3 x3    <NA> 
#> 4     4 <NA>  y4

Filtering Joins

Filtering joins match observations in the same way as mutating joins, but affect the observations, not the variables. … Semi-joins are useful for matching filtered summary tables back to the original rows. … Anti-joins are useful for diagnosing join mismatches.
R for Data Science: Filtering Joins

Semi Join

All rows from x where there are matching values in y, keeping just columns from x.

semi_join(x, y, by = "id")
#> # A tibble: 2 × 2
#>      id x    
#>   <int> <chr>
#> 1     1 x1   
#> 2     2 x2

Anti Join

All rows from x where there are not matching values in y, keeping just columns from x.

anti_join(x, y, by = "id")
#> # A tibble: 1 × 2
#>      id x    
#>   <int> <chr>
#> 1     3 x3

Set Operations

Set operations are occasionally useful when you want to break a single complex filter into simpler pieces. All these operations work with a complete row, comparing the values of every variable. These expect the x and y inputs to have the same variables, and treat the observations like sets.
R for Data Science: Set operations

x
#> # A tibble: 3 × 2
#>   x     y    
#>   <chr> <chr>
#> 1 1     a    
#> 2 1     b    
#> 3 2     a
y 
#> # A tibble: 2 × 2
#>   x     y    
#>   <chr> <chr>
#> 1 1     a    
#> 2 2     b

Union

All unique rows from x and y.

union(x, y)
#> # A tibble: 4 × 2
#>   x     y    
#>   <chr> <chr>
#> 1 1     a    
#> 2 1     b    
#> 3 2     a    
#> 4 2     b

union(y, x)
#> # A tibble: 4 × 2
#>   x     y    
#>   <chr> <chr>
#> 1 1     a    
#> 2 2     b    
#> 3 1     b    
#> 4 2     a

Union All

All rows from x and y, keeping duplicates.

union_all(x, y)
#> # A tibble: 5 × 2
#>   x     y    
#>   <chr> <chr>
#> 1 1     a    
#> 2 1     b    
#> 3 2     a    
#> 4 1     a    
#> 5 2     b

Intersection

Common rows in both x and y, keeping just unique rows.

intersect(x, y)
#> # A tibble: 1 × 2
#>   x     y    
#>   <chr> <chr>
#> 1 1     a

Set Difference

All rows from x which are not also rows in y, keeping just unique rows.

setdiff(x, y)
#> # A tibble: 2 × 2
#>   x     y    
#>   <chr> <chr>
#> 1 1     b    
#> 2 2     a

setdiff(y, x)
#> # A tibble: 1 × 2
#>   x     y    
#>   <chr> <chr>
#> 1 2     b

Tidy Data

Tidy data follows the following three rules:

1. Each variable has its own column.
2. Each observation has its own row.
3. Each value has its own cell.

Many of the tools in the tidyverse expect data to be formatted as a tidy dataset and the tidyr package provides functions to help you organize your data into tidy data.

wide
#> # A tibble: 2 × 4
#>      id x     y     z    
#>   <int> <chr> <chr> <chr>
#> 1     1 a     c     e    
#> 2     2 b     d     f
long
#> # A tibble: 6 × 3
#>      id key   val  
#>   <int> <chr> <chr>
#> 1     1 x     a    
#> 2     2 x     b    
#> 3     1 y     c    
#> 4     2 y     d    
#> 5     1 z     e    
#> 6     2 z     f

Pivot Wider and Longer

pivot_wider() and pivot_longer() were introduced in tidyr version 1.0 (released in September 2019). They provide a more consistent and more powerful approach to changing the fundamental shape of the data and are “modern alternatives to spread() and gather().

Here we show the very basic mechanics of pivoting, but there’s much more that the pivot functions can do. You can learn more about them in the Pivoting vignette in tidyr.

pivot_wider(data, names_from = key, values_from = val)

pivot_wider() “widens” data, increasing the number of columns and decreasing the number of rows.

pivot_longer(data, cols = x:y, names_to = "key", values_to = "val")

pivot_longer() “lengthens” data, increasing the number of rows and decreasing the number of columns.

Spread and Gather

spread(data, key, value)

Spread a key-value pair across multiple columns. Use it when an a column contains observations from multiple variables.

gather(data, key = "key", value = "value", ...)

Gather takes multiple columns and collapses into key-value pairs, duplicating all other columns as needed. You use gather() when you notice that your column names are not names of variables, but values of a variable.

gather(wide, key, val, x:z)
#> # A tibble: 6 × 3
#>      id key   val  
#>   <int> <chr> <chr>
#> 1     1 x     a    
#> 2     2 x     b    
#> 3     1 y     c    
#> 4     2 y     d    
#> 5     1 z     e    
#> 6     2 z     f
spread(long, key, val)
#> # A tibble: 2 × 4
#>      id x     y     z    
#>   <int> <chr> <chr> <chr>
#> 1     1 a     c     e    
#> 2     2 b     d     f

tidyexplain's Issues

Publish animations on companion github pages...

... so the animation becomes visible (README.md doesn't support animations I think)

Move default branch from `master` to `main`

The default branch of this repository is being moved from master to main. This issue will be closed when the default branch has been changed.

For help and guidance with updating the default branch of your local or forked copy of this repo, please see Renaming the default branch.

Search for files that may potentially reference master.

^{ID: 91e77a361e4e}

tidyr :: spread() and gather()

https://twitter.com/CMastication/status/1029575831943868416

tidyr :: pivot_wider() and pivot_longer()

Animation request for pivot_wider() and pivot_longer().

Background: “Pivotting” which converts between long and wide forms. tidyr 1.0.0 introduces pivot_longer() and pivot_wider(), replacing the older spread() and gather() functions. See vignette("pivot") for more details.

Link: https://tidyr.tidyverse.org/

`union()` animations are broken

The latest animations aren't working as expected:

commit	union(x, y)	union(y, x)
280f621
69cb816

Variable width and height

If we have variable width of each cell, we can also animate nest/unnest nicely.
Not sure how to do it yet :)

Migrate README to pkgdown site with vignettes

A major advantage of moving to the package structure is that we can use pkgdown to create a slick website to replace the README. In addition to documenting the package functions, each major section of the README can be an individual vignette/article.

Fix semi join

Not sure what happened but the y side keys don’t end up in the right place.

tidyr :: separate() and extract()

tidyr::unite animation request

Package cleanup

• Replace few stringr calls with base R
• Qualify package namespaces for non-imported functions
• De-pipe single command pipes
• Remove final-line return() statements in functions

Fonts

I used Fira Mono (title) and Fira Sans (cell text) originally, but I think "Fira Sans Condensed" is a better choice for cell text.

I also used sysfonts to grab fonts from Google and showtext to ensure they are handled consistently across systems (in particular where the fonts aren't already installed).

We should probably invest some code or time to make sure that either we have good defaults that will work wherever ggplot2 does or that we have some code in place to make sure that our default fonts are available.

tidyr :: nest and unnest

https://twitter.com/dvaughan32/status/1032749426186637312?s=21

Fix tidyr spread/gather static image

Font sizes aren't right

Migrate to package structure

Once the dust settles on the twitter poll...

Rename this repo. It seems that everything will redirect but may need to investigate/resolve if not.

I'd like to merge #10 into a new pkg branch that we can use for the package development. This way we can push fixes and improvements there while leaving the current animations and documentation on the master branch. Once everything is ready to go with the package, we can merge to master.

To facilitate communication, I'd like to still use feature branches and PRs for any major feature additions or updates. Minor bugfixes can be pushed directly to pkg. Branch names like pkg-add-filter-animation are great. Let me know if there are any questions with this workflow.

I'll handle getting #10 into the pkg branch in this repo once it's ready.

tidyr :: expand, crossing, and complete

Show complete pivot_longer() in intermediate step

Feedback follow up to #35: I originally wanted to de-emphasize pivot_longer() to highlight the cols, names_to and values_to arguments in the intermediate step between wide and long formats, but using ggtext for that feature breaks the monospace code formatting. I settled on temporarily removing pivot_longer( to draw attention, but because the text just disappears it doesn't work so well.

Would love to have available an animation for dplyr::group_by() %>% dplyr::summarise() %>% dplyr::ungroup

Title says it all. I might even get to this myself. More of a reminder to myself that I didn't see it on the website and had a need for it for teaching.

Quote/unquote inputs to animate_<type>_join helpers

to pass inputs to animate_join() so that actual names can be used in plot annotations

Make use of `...` consistent and documentable

We need to ensure that arguments passed through ... are documentable (i.e. are inherited from another function) and that their scope is relatively limited.

Currently, nearly every function takes ... which means that we don't have issues with extra arguments being passed to functions that don't take them.

This short reprex demonstrates the issue with removing ... from existing functions.

f <- function(x = "a", ...) paste(x)

g <- function(y = "b", ...) paste(y)

h <- function(z, ...) paste(z, g(...), f(...))

# Doesn't have ... to capture unused arguments
f2 <- function(x = "a") paste(x)

h2 <- function(z, ...) paste(z, g(...), f2(...))

h("c", y = "e")
#> [1] "c e a"
h2("c", y = "e")
#> Error in f2(...): unused argument (y = "e")

In the above example, we would need to add x as a parameter to h2() or do some work to separate the arguments in the .... In the end, though, the effort will be worth it because documentation will be much clearer and the user will be able to figure out from autocompletion, etc, what parameters get passed to which functions.