Error when setting row.idx

[pbreheny]

Error when setting row.idx

[privefl]

What is size, X, y, ...?

[dbstern]

Sorry, I forgot the line
size = length(y)

Not sure if this answers your question but

str(x)
Formal class 'big.matrix' [package "bigmemory"] with 1 slot
..@ address:
str(y)
logi [1:21989] FALSE TRUE FALSE FALSE FALSE FALSE ...

[privefl]

I mean, could you provide a reproducible example with some example data so that we can run your code and see the error.

[dbstern]

x <- as.big.matrix(matrix(rnorm(n= 21989,2790), nrow = 21989))
y <- sample(c(0,1), size = 21989, replace = T)
train <- sample(c(T,F), length(y), c(.5,.5), replace = TRUE)
fit <- cv.biglasso(X = x, y = y, row.idx = which(train), penalty = "lasso", family = "binomial", nfolds = 10)
Error in [<-(*tmp*, cv.ind == i, 1:res$nl, value = res$loss) :
(subscript) logical subscript too long

[privefl]

If we run the code step by step after using debugonce(cv.biglasso), we see a first problem where cv.ind is defining folds for the whole sample size, instead of only the indices of training set.

Specifying cv.ind = sample(rep_len(1:10, sum(train))) returns another error.

[dbstern]

I used deepcopy to workaround this bug, but this was taking too much memory and the server would sometimes crashes. When I use deepcopy on a big.matrix, a corresponding file is created at "dev/shm" (I tried changing the backingfile to my external hd but wasn't successful).

I found that the bigstatsr package makes it easier to deal with these problems (althought I'm not sure if what I'm doing is ok). The code is now something like this:

exthd_path <- "."
file <- file.path(exthd_path,"test.txt")
x <- as.big.matrix(matrix(rnorm(n= 21989,2790), nrow = 21989))
write.big.matrix(x, filename = file, row.names = FALSE, col.names = T, sep = " ")
rm(x); gc()

y <- sample(c(0,1), size = 21989, replace = T)
train <- sample(c(T,F), length(y), c(.5,.5), replace = TRUE)
x <- big_read(file, select = 1)
xtrain <- big_copy(x, ind.row = which(train), backingfile = paste0(bigstatsr::sub_bk(x$bk),"-train"))
fit <- cv.biglasso(X = xtrain$bm(), y = y[train], penalty = "lasso", family = "binomial", nfolds = 10)
unlink(xtrain$bk); rm(xtrain)
unlink(c(x$bk,x$rds)); rm(x)

[privefl]

I think you can directly use big_copy() when x is a big.matrix.