Comments (8)
You can try using rowwise
(https://dcor.readthedocs.io/en/latest/functions/dcor.rowwise.html#dcor.rowwise), probably with "parallel" compile mode, as in the last section of https://dcor.readthedocs.io/en/latest/performance.html. Tell me if that works for you.
from dcor.
Hi @vnmabus ,
I am afraid this doesn't work for me - I tried doing pairwise distance corr from columns on a np.array as follows:
d_cor = np.apply_along_axis(lambda col1: np.apply_along_axis(lambda col2: dcor.rowwise(dcor.distance_correlation, [col1], [col2], compile_mode=dcor.CompileMode.COMPILE_PARALLEL), axis = 0, arr=df), axis =0, arr=df)
Running all night and still not completed in the morning - can you see anything wrong in the code that might be causing it not to work optimally (or another implementation code wise that may make it run faster)? thank you
from dcor.
rowwise
can only truly parallelize if you give it several columns. You want to pass to it all column combinations. If you for example had 3 columns, you would have to pass: [col1, col1, col1, col2, col2, col2, col3, col3, col3], [col1, col2, col3, col1, col2, col3, col1, col2, col3]
.
As the correlation is symmetric you can also avoid computing almost half of the combinations. There is still a memory problem, as the same column is copied several times. As dcor
uses the CPUs for parallelization, you can always pass chunks with the same size as the total number of CPUs in your system, in order to keep memory usage low.
But at the end, computing a measure pairwise between 10000 columns is going to take time. We are talking about 50.005.000 operations here. In a machine with 32 cores and perfect parallelization that would be 1.562.657 parallel runs. Even if each parallel run takes 0.1 seconds you will have to run that for almost two days (please repeat the math yourself as I did that quickly).
from dcor.
thank you for the quick response and explanation!
from dcor.
Did that work for you?
from dcor.
Hi @vnmabus ,
thank you for following up,
I am afraid not - I will run it as per my code above on a HPC to see if this can be left alone and run for a few days!
from dcor.
I recommend you to try to estimate the time that it will take, based on the number of operations and the time spend in one operation. Then if the estimated time is too high, try implementing the improvements I mentioned before. Otherwise you could potentially wait for weeks without knowing it.
from dcor.
yes, i think that time you calculated is accurate - I will try especially the running in chunks as you suggested this is sensible
from dcor.
Related Issues (20)
- Question: is there a fast method for `dcor.independence.distance_covariance_test` HOT 2
- OSError: [Errno 36] File name too long when importing dcor HOT 5
- __version__ returns 0.0. Version number is on a separate file HOT 6
- AttributeError: 'float' object has no attribute 'dtype' HOT 1
- Process killed due to very large array HOT 2
- FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\domin\\PycharmProjects\\Trading_Backtesting_ML\\venv\\lib\\site-packages\\dcor\\__pycache__\\_fast_dcov_mergesort._generate_distance_covariance_sqr_mergesort_generic_impl.locals._distance_covariance_sqr_mergesort_generic_impl-163.py38.nbi.tmp.4ae6be2f415b45ff' HOT 2
- Improve performance of pairwise distances computation
- Add goodness-of-fit tests
- Add distance skewness and symmetry test
- Implement distance components (DISCO)
- Study and implement energy-based clustering
- Implement energy distance in terms of distance covariance
- Adding support for python 3.7 HOT 1
- Question about the shape of the input array HOT 3
- Can dcor with method 'AVL' or 'megresort' is applicable between two data types float and integer, respectively or it always has to be float? HOT 13
- Can distance correlation-based t test is theoretically correct to implement for "uni"-dimensional data? HOT 2
- Seemingly incorrect results with `int` datatype HOT 3
- Incorrect documentation about arbitrary dimensions HOT 2
- Range of distance correlation HOT 11
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dcor.