std::bad_alloc

[000Justin000]

std::bad_alloc

[lettis]

Dear Junteng,
can you provide me please with a bit more information: is 10^5 the number of observables (i.e. the number of columns) or the number of samples (== rows)?
Can you please give me the shape of your input (# rows x # cols)?

Thanks,
Florian

[000Justin000]

Dear Lettis,

The 100000 number is the number of observables. The shape of data is 500000 (rows)‎ and 100000(columns).

On top of that, is there a way to find out the largest k eigenvalue and their corresponding eigenvectors? Diagonalize the whole covariance matrix is too costly.

And I found that the code can not be compiled with intel compilers, or am I doing it wrong?

Thanks!

Best,
Junteng

Sent from my BlackBerry 10 smartphone.
From: Florian Sittel
Sent: Friday, February 5, 2016 1:16 AM
To: lettis/FastPCA
Reply To: lettis/FastPCA
Cc: Junteng Jia
Subject: Re: [FastPCA] std::bad_alloc (#2)

Dear Junteng,
can you provide me please with a bit more information: is 10^5 the number of observables (i.e. the number of columns) or the number of samples (== rows)?
Can you please give me the shape of your input (# rows x # cols)?

Thanks,
Florian

---

Reply to this email directly or view it on GitHub:
#2 (comment)

[lettis]

Dear Junteng,
indeed, 10^5 observables means > 8*10^10 Bytes = 80 Gb space, just for the covariance matrix. Not included are the space requirements for temporary memory needed for LAPACK and memory for the construction of the covariance matrix.
I guess, this is where the bad_alloc exception comes from ...

To compute such a big data set, you probably need another approach.
I would suggest something like:

1. compute the pairwise covariances of the observables
2. define a cutoff and store only the covariances above that cutoff
3. hope that your cov-matrix will be sparse
4. use an appropriate algorithm for symmetric sparse matrices to get the eigendecomposition

Unfortunately, the FastPCA code is not written for these sizes or sparse cov-matrices. Of course, you can try to adapt it, but I guess it will be easier to write your own code for that ...

On a side node:
Of course, I do not know what your application is.
But be reminded that for a large number of observables that may lie in very different spaces, it may be prudent to first normalize the input variables, effectively computing the correlations instead of covariances.

To answer your second question: I never used the Intel compiler, since the latest version did not fully support the C++11 standard (at least when I developed the code, I do not know it's status today).
The code was developed with the GCC.

Cheers,
Florian

[000Justin000]

Dear Florian,

Thanks for your reply! I noticed in the package ARPACK, which is used in Matlab, it provide the function to find k eigenvector with the largest eigenvalues. It is very useful since its a O(N'2) algorithm. I will try to figure to link against that routine if possible.

Have a good week!

Best Regards,
Junteng

Sent from my BlackBerry 10 smartphone.
From: Florian Sittel
Sent: Friday, February 5, 2016 2:06 AM
To: lettis/FastPCA
Reply To: lettis/FastPCA
Cc: Junteng Jia
Subject: Re: [FastPCA] std::bad_alloc (#2)

Dear Junteng,
indeed, 10^5 observables means > 8*10^10 Bytes = 80 Gb space, just for the covariance matrix. Not included are the space requirements for temporary memory needed for LAPACK and memory for the construction of the covariance matrix.
I guess, this is where the bad_alloc exception comes from ...

To compute such a big data set, you probably need another approach.
I would suggest something like:

1. compute the pairwise covariances of the observables
2. define a cutoff and store only the covariances above that cutoff
3. hope that your cov-matrix will be sparse
4. use an appropriate algorithm for symmetric sparse matrices to get the eigendecomposition

Unfortunately, the FastPCA code is not written for these sizes or sparse cov-matrices. Of course, you can try to adapt it, but I guess it will be easier to write your own code for that ...

On a side node:
Of course, I do not know what your application is.
But be reminded that for a large number of observables that may lie in very different spaces, it may be prudent to first normalize the input variables, effectively computing the correlations instead of covariances.

To answer your second question: I never used the Intel compiler, since the latest version did not fully support the C++11 standard (at least when I developed the code, I do not know it's status today).
The code was developed with the GCC.

Cheers,
Florian

---

Reply to this email directly or view it on GitHub:
#2 (comment)