More with nested Arrays

[nomeata]

More with nested Arrays

[lehins]

I don't really see anything wrong with your approach of folding inner dimensions.

Probably there should be a better way to do that that does not involve makeArray right?

Can you elaborate on this. You can't create an array without a function like makeArray, be it an unsafe or mutable variant of thereof. Why do you need a nested array if you are folding the inner array anyways?

Array P Ix2 (Array P Ix2 Double)

Such an array is impossible in general, because an element of a primitive array must have a Prim instance and no array can have such an instance, therefore with nested arrays an outer array must be either delayed, or boxed.

Here is an expanded version of your example that does the same thing:

densityDelayed :: Array P Ix4 Double -> Array D Ix2 (Array D Ix2 Double)
densityDelayed a = A.makeArray Seq (A.Sz2 100 100) $ \(A.Ix2 x y) -> a A.!> x A.!> y

density :: Array P Ix4 Double -> Array P Ix2 Double
density a = A.compute (A.sum <$> densityDelayed a)

Delayed array (D) is not directly represented by an in memory buffer, instead it is just a function that does the slicing into the original array. So, in both versions of the density function (the version above and the one in your question) there will be only two primitive arrays allocated in memory: the one that was supplied as an argument and the one for the result. Shape shifting/slicing is handled in massiv with delayed arrays. Maybe that's what you meant by shape shifting?

One note on parallelization in this example. If the outer size is 100x100 and the inner arrays are very small then it could be more efficient to compute the inner sum sequentially, eg. size a == 100 :> 100 :> 5 :. 5:

density :: Array P Ix4 Double -> Array P Ix2 Double
density a = A.makeArray Par (A.Sz2 100 100) $ \(A.Ix2 x y) -> A.sum (A.setComp A.Seq (a A.!> x A.!> y))

Another note on the size of the resulting array, it is better to make it a bit safer by getting the size of the source array:

density :: Array P Ix4 Double -> Array P Ix2 Double
density a = A.makeArray Par (A.Sz2 m n) $ \(A.Ix2 x y) -> A.sum (A.setComp A.Seq (a A.!> x A.!> y))
  where
    A.Sz4 m n _ _ = A.size a

[nomeata]

Thanks! For some reason I assumed that makeArray explicit will be less efficient than composing array functions where some magic fusion techniques kick in, but if that’s not the case, even better!

Also thanks for the explanation about the delayed arrays.

And yes, I was hard-coding the 100 just while playing around. I’ll fix that once my simulation no longer behaves completely weird (https://mastodon.online/@nomeata/109942791776084644 …) :-)