Implementing Distributed Rank 1 Dictionary Learning in Dask
quinngroup / dr1dl-dask Goto Github PK
View Code? Open in Web Editor NEWDictionary Learning in Dask
Dictionary Learning in Dask
In the Pyspark implementation, there were two arguments that I really had to do anything with.
How to make this function work with or without a GPU available. What is the best way? Checking if this https://github.com/anderskm/gputil will return anything (None type if no GPUs available) i.e.
GPUtil.showUtilization()
OR:
try:
import cupy
make cupy array
else:
Make dask array
I want this thread to be used for coming up with names for for my thesis and JOSS submission.
Thesis:
Rank 1 Dictionary Learning: Comparing Distributed Frameworks
Comparing Dask and Spark: Scalable Rank 1 Dictionary Learning
JOSS Paper:
dR1DL-Dask: Scalable Rank 1 Dictionary Learning in Dask
Currently implemented is pydata's sparse.COO
but it isn't compatable with cupy yet! It is part of their future plans, but currently not.
Other option but not really. Cupy has scipy coo_matrix implemented but it doesn't support 1D vectors.
The S Matrix is a cupy array, so the 1D array must be cupy or compatible with cupy and numpy aka sparse.COO
A = da.random.random(200)
B = da.random.random(200)
B[B<.90] = 0
B = B.map_blocks(sparse.COO)
da.outer(A,B.compute().todense())
This test works great. A is a dask array and B is a Dask Sparse array. By calling B.compute()
it converts it back to a sparse.COO
array. In order for the outer function to work, we have to have the dense the sparse array by calling todense()
.
In the code, by the time the sparse array goes through calculation da.outer
will no longer accept the type.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.