Outer Product with sparse vector and dask array

[crumleyc]

Outer Product with sparse vector and dask array

[crumleyc]

After many different test, I finally got it.

[magsol]

Have you tested how well this scales? I don't see a .todense() call on a large matrix going over very well!

[crumleyc]

My exact thought but I couldnt get it to work regardless. It said that it couldn't automatically do it, so I had to call it. scipy.sparse wont work. This is the only implementation that I have been able to work.

[magsol]

True, but it also directly contravenes any notion of scalability, which arguably makes the use of dask obsolete in the first place.

Working in this area, you sometimes need to step back from "typical" linear algebra operations involved, and instead break them down into fundamental computations--e.g., row/column multiplications, list summations, etc--and see how you can reorganize and restructure these fundamental operations in order to scale them better.

Also, this seems to be almost exactly what you're looking for https://docs.dask.org/en/latest/array-api.html#dask.array.blockwise

[magsol]

Also, why wouldn't the dask array outer function work on other dask arrays? https://docs.dask.org/en/latest/array-api.html#dask.array.outer

[crumleyc]

Thats true. Thanks for sending that!

Also, It did, but once the dask array wrapped the COO it needed it to be dense. It also needed some flatten function, but I got the sparse team to implement flatten. now it says
RuntimeError: Cannot convert a sparse array to dense automatically. To manually densify, use the todense method.

[crumleyc]

I implemented the blockwise function pretty much the same way that it was on the dask's website, and it didnt throw an error at all without using the todense() method. I then had one array where I used the blockwise outer product and then the da.outer product. I subtracted them from each other to see what the difference would be. There are a hand full of values that aren't 0, but the rest are. Not sure why those few aren't but I feel pretty good about using the blockwise. Thanks again.

[magsol]

without using the todense() method

That means you implemented blockwise outer without using todense() at all? Sorry if this sounds repetitive, the wording is just strange to me for some reason.

Also, double-check those non-zero values to see if they're something really close to zero, like 1e-18 or something. Those sorts of values you can essentially consider zero, as they're only non-zero by virtue of floating point precision errors.

[crumleyc]

I was just able to throw in the broadcasted sparse vector without having to change it from a Dask Array to sparse array to a dense array before doing the outer product. I'll run it again to see what those values are.