index error when fit_transform a large dataset

[scikit-learn-contrib]

index error when fit_transform a large dataset

[fmfn]

It seems like a problem resulting from using a sparse matrix. To be honest, sparse matrix were completely neglected when writing this. Which kinda makes sense given when they are usually needed and the fact that most methods here (definitely SMOTE) rely on local density estimations to generate new samples.

I'll take a look over the weekend.

[eromoe]

I found above problem cause by y_train be converted to pandas.Series in some place at my code. After I fix that, sparse become pain, you are right.

concatenate((overx,self.x[self.y == key],self.x[self.y == key][indx]), axis=0)
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\IPython\core\interactiveshell.py", line 3066, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-40-b5ea4e093941>", line 1, in <module>
    concatenate((overx,self.x[self.y == key],self.x[self.y == key][indx]), axis=0)
ValueError: zero-dimensional arrays cannot be concatenated

test code:

a = sparse.csr_matrix(np.zeros((100,200)))
b = sparse.csr_matrix(np.zeros((200,200)))
c = sparse.csr_matrix(np.zeros((300,200)))

np.concatenate((a,b,c), axis=0)

>>>  ValueError: zero-dimensional arrays cannot be concatenated .....

sulotion:

import scipy.sparse as sp
sparse.vstack((a, b, c))

>>> <600x200 sparse matrix of type '<type 'numpy.float64'>'
        with 0 stored elements in Compressed Sparse Row format>

Add a custom function to detect the type then choose concatenate or vstack?
Maybe there are some similar problem like this, I would search for is there any exist package would play this situation well.