[Question] Numpy derived memory error in Pysheds

[mdbartos]

[Question] Numpy derived memory error in Pysheds

[mdbartos]

Hi @Rockhound-NC

I use a 2015 Macbook Pro (8 GB ram) for most of my development and day-to-day testing.

A 36M-length numpy array should be 144 MB (if int32) and 288 MB (if int64). You could save some space by restricting it to unsigned int, but that shouldn't be the problem here.

I'll have to check and see if the behavior of some of the dependencies has changed. Otherwise, I can't see a reason why you'd be getting a memory error 🤔

Have you tried creating a 36M length numpy array outside of pysheds? (e.g. a = np.arange(36000000))

Thanks,
MDB

[scheip]

Thanks for the reply Matt. Some very interesting behavior on my end. I am a bit of a newb, so forgive me if this isn't the right forum for this question. Here is some commented code to help explain what I am seeing:

# import required modules
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.colors as colors
from pysheds.grid import Grid
import seaborn as sns
#import warnings
#warnings.filterwarnings('ignore')

# Set env variables and directories
workingDir='D:/_projects/pysheds'

# Assign variables
grid = Grid.from_raster(workingDir + '/_data/n30w100_con', data_name='dem')

# following lines runs
aTest=np.arange(36000000)

# following line runs
bTest = np.arange(grid.size)

# following line fails due to memory error.
grid.resolve_flats('dem', out_name='inflated_dem')

# Now, the command from a few lines ago also fails due to memory error
aTest = np.arange(36000000)

It seems that maybe something in the execution of any function from grid.py that employs the _inside_indices subroutine is bogging up my memory. Any help appreciated!

[mdbartos]

Are you using the 32-bit version of python? I would recommend using the 64-bit version (not just for this library, but in general). See: https://www.quora.com/What-are-the-advantages-of-64-bit-Python-over-32-bit-Python

Aside from that, I can't think of any issues that would be causing you to run out of memory. You can also try calling grid._inside_indices directly and see what happens.

[scheip]

Alright, fantastic advice there on the 64 bit version. That cleared up the memory issue. Thanks!

I did have to play a bit with skimage to get things to work and figured I'd pass this along. If it jives on your end, may consider an update to the code.

In grid.py you attempt to import measure and transform from skimage. I believe skimage has been updated to where measure and tranform are now submodules and you must import "from", not dot import.

I replaced your two lines (20 and 21, import skimage.measure and import skimage.transform) with the following block of code, recompiled the pyc, and voila.

    import skimage
    from skimage import measure
    from skimage import transform

Very excited about this package, appreciate your time!

[mdbartos]

Are you using the development version of skimage? I was able to import using the latest stable version from pypi:

In [1]: import skimage

In [2]: import skimage.transform

In [3]: import skimage.measure

In [4]: skimage.__version__
Out[4]: '0.14.0'