Working through data instructions

[jonkeane]

Working through data instructions

[jonkeane]

Any opinions on whether we:

• just mention out right we are using a mish mash of methods (we have 3 methods: arrow::open_dataset(), curl::multidownload() and download.file())
• change the one optional curl::multidownload() one to download.file() (the Seattle CSV file)
• change the various download.file() ones to curl::multidownload() (reduces to 2 methods)

IMHO the second or third, maybe leaning towards the second since it's base and means you don't need to also ask folks to install curl (surely they will have it anyway, but still one less thing to need to on). but the third would reduce the number of variants in cases where they "aren't" "necessary"

[stephhazlitt]

Thanks @jonkeane, excellent feedback.

For open_dataset(), I don't think that is a 12.0.1 vs 13.0.0 issue, but IIRC you can get that strange error when you don't have dplyr loaded. Can you try this:

library(arrow)
library(dplyr)

data_path <- here::here("data/nyc-taxi") # Or set your own preferred path

open_dataset("s3://voltrondata-labs-datasets/nyc-taxi") |>
    filter(year %in% 2012:2021) |> 
    write_dataset(data_path, partitioning = c("year", "month"))

[jonkeane]

For open_dataset(), I don't think that is a 12.0.1 vs 13.0.0 issue, but IIRC you can get that strange error when you do't have dplyr loaded. Can you try this: [...]

That does seem to be working now, yeah

[stephhazlitt]

Excellent point about the mish-mash of methods. Of course, the reason is either an intentional one (providing a couple of options for getting the big data and one that uses arrow, so we can talk about using arrow with cloud) or a legacy one (because that is the way R4DS gets the data 😆).

Any opinions on whether we:

• just mention out right we are using a mish mash of methods (we have 3 methods: arrow::open_dataset(), curl::multidownload() and download.file())
• change the one optional curl::multidownload() one to download.file() (the Seattle CSV file)
• change the various download.file() ones to curl::multidownload() (reduces to 2 methods)

[stephhazlitt]

For open_dataset(), I don't think that is a 12.0.1 vs 13.0.0 issue, but IIRC you can get that strange error when you do't have dplyr loaded. Can you try this: [...]

That does seem to be working now, yeah

OK, so I will add library(dplyr) in then—great catch.

[stephhazlitt]

Updating with feedback WIP: #3

[thisisnic]

Closing this now as I just merged the PR, but let me know if we've missed anything. Thanks for testing these out @jonkeane and making the updates @stephhazlitt !