Shape is not preserved when deserializing blaze objects

[blaze]

Shape is not preserved when deserializing blaze objects

[sdiehl]

Github kind of muddled the code formatting, I tried to format it. Did I get the test case right?

[FrancescAlted]

No, I have updated it correctly now. BTW, I like this single liner:

if os.path.exists(path): shutil.rmtree(path)

but probably you don't :)

[sdiehl]

I can reproduce this issue. It's definitely in Blaze, seems carray does the correct thing.

[ovillellas]

I've started playing with blaze, as I will start making extra tests for the persistence layer. There seems to be related problems that do not involve persistence at all. The following code may serve as a hint of what's going wrong:

>>> ds = dshape('2,2,int')
>>> a = Array(range(4), ds)

>>> a
Array
  datashape := 2, 2, int 
  values    := [CArray(ptr=4328095024)] 
  metadata  := [manifest, arraylike] 
  layout    := Chunked(dim=0) 
[0 1 2 3]    <---- looks dodgy... shouldn't it be "[[0 1] [2 3]]" ???


>>> a[2]
array(2)  <--- dodgy... bad indexing for a (2,2) array, right?

>>> a[0][0]
IndexError: 0-d arrays can't be indexed

[ovillellas]

Another update. It seems it is related to the Array constructor.
zeros and ones do result in proper arrays. However, Array looks very dodgy right now:

In [101]: Array(np.eye(6), dshape('2, 2, float64'))
Out[101]: 
Array
  datashape := 2, 2, float64 
  values    := [CArray(ptr=4328164448)] 
  metadata  := [manifest, arraylike] 
  layout    := Chunked(dim=0) 
[ 1.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.
  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  1.]

It seems that it just takes whatever values you pass in the first parameter and makes it a 1-d array out of it. The datashape seems ignored other than to set the datashape attribute. Note that in the example even the the size of the initial data is wrong and is preserved (it enumerates all 36 elements, even as from the datashape point of view only 4 should be present...)

[sdiehl]

Yeah there is something weird going on in the constructor, especially if you try interpret a numpy array in terms of a datashape that doesn't make sense in terms of the input object. Most of the array objects feed into either numpy or carray and read off the shapes and dtype and then upconvert into datashape as stop-gap measure... seems like its time to remove the stop-gaps and write a proper constructor function.

[ovillellas]

Ok, I've traced back the problem to carray.

It seems that when the carray is built providing a dtype, it loses the shape of the original array. It seems this is causing this bug.

I am going to add tests in carray to check this code, and try to fix it.

[ovillellas]

Fixed on commit
f896163 FIX: github blaze issue #14. carray now preserves dimensionality when an732a378 added test_queries from carray

In persist branch

[sdiehl]

Want to just merge this to trunk whenever you feel the branch is in a good place?

[ovillellas]

That would be fine. I may just push the whole branch as well... as it only adds the fix and some extra unit-tests for carray.

Right now I am adding a test case for this issue, So I will wait till that one is ready.

[ovillellas]

The fix needs extra polish

[ovillellas]

This was fixed in a definitive way. It is in the persist branch