posit-conf-2023/arrow

Materials for the workshop "Big Data in R with Arrow" at posit::conf(2023)

arrow's Issues

Working through data instructions

Using the instructions at https://posit-conf-2023.github.io/arrow/setup.html

I don't have an entirely fresh environment, but it's not been used too much for analysis either, so hopefully is semi representative of folk's setups.

• https://posit-conf-2023.github.io/arrow/setup.html#nyc-taxi-data uses {here} be that is not requested to be installed anywhere. Same with {curl} though that was installed on my system already.
• when I tried the open_dataset() ... write_dataset() I get: Error in attr(data, "tsp") <- c(start, end, frequency) : object is not a matrix I thought there was something about 13 and this, so maybe this is already known + fixed in 13 (I'm on 12 right now)
• I was able to get the full nyc data using option 2 in about 40 minutes (on a 250 mb down connection)

One general note, it seemed a little bit odd to use curl::multi_download() in some places and then download.file() in others. Maybe an explanation of why?

FWIW, for the seattle library checkouts by title, curl::multidownload() was only a but faster:

   user  system elapsed 
 98.517 142.213 438.774 

compared to download.file() on the same file:

   user  system elapsed 
 68.629 150.350 592.596 

[Useful Resource] SO thread on difference btw parquet & arrow/feather file formats

https://stackoverflow.com/questions/56472727/difference-between-apache-parquet-and-arrow

Remove references to Slido throughout materials

Decision made to use Discord channel (or just 🙋🏻‍♀️) instead.

Add Final Details to Housekeeping Slides

• wifi + password
• restroom + specialty room locations
• confirm all urls are working

arrow::schema() function does not work to extract schema from an arrow dataset

On "3_data_engineering.qmd", "schema(seattle_csv)" has an error:

Error in map2(.x, vec_index(.x), .f, ...) : 
Caused by error:
! 1 must be a DataType, not FileSystemDatasetDatasetArrowObjectR6
16.
stop(fallback) 
15.
signal_abort(cnd, .file) 
14.
rlang::abort(message, ..., call = call, use_cli_format = TRUE, 
.frame = .frame)
13.
cli::cli_abort(message, location = i, name = name, parent = cnd, 
call = error_call, class = "purrr_error_indexed")
12.
h(simpleError(msg, call)) 
11.
.handleSimpleError(function (cnd) 
{
if (i == 0L) {
} ...
10.
stop(name, " must be a DataType, not ", class(type), call. = FALSE) 
9.
.f(.x[[i]], .y[[i]], ...) 
8.
call_with_cleanup(map2_impl, environment(), .type, .progress, 
n, names, i)
7.
withCallingHandlers(expr, error = function(cnd) {
if (i == 0L) {
}
else { ...
6.
with_indexed_errors(i = i, names = names, error_call = .purrr_error_call, 
call_with_cleanup(map2_impl, environment(), .type, .progress, 
n, names, i))
5.
map2_("list", .x, .y, .f, ..., .progress = .progress) 
4.
map2(.x, vec_index(.x), .f, ...) 
3.
imap(.list, as_type) 
2.
Schema__from_list(imap(.list, as_type)) 
1.
schema(seattle_csv) 

My arrow version is 12.0.1.1.

It seems that schema() only creates a schema, but does not extract one anymore.