Test suite stalls

[pytroll]

Test suite stalls

[avalentino]

If i skip the test that stalls then 3 other tests fail:

======================================================================
ERROR: test_gauss_multi_mp (pyresample.test.test_kd_tree.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 546, in _query_resample_kdtree
    dt = resample_kdtree.data_pts.dtype
AttributeError: 'cKDTree_MP' object has no attribute 'data_pts'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/test/test_kd_tree.py", line 453, in test_gauss_multi_mp
    nprocs=2, segments=1)
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 179, in resample_gauss
    reduce_data=reduce_data, nprocs=nprocs, segments=segments, with_uncert=with_uncert)
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 259, in _resample
    segments=segments)
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 376, in get_neighbour_info
    nprocs=nprocs)
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 548, in _query_resample_kdtree
    dt = resample_kdtree.data.dtype
AttributeError: 'cKDTree_MP' object has no attribute 'data'

======================================================================
ERROR: test_gauss_multi_mp_segments (pyresample.test.test_kd_tree.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 546, in _query_resample_kdtree
    dt = resample_kdtree.data_pts.dtype
AttributeError: 'cKDTree_MP' object has no attribute 'data_pts'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/test/test_kd_tree.py", line 483, in test_gauss_multi_mp_segments
    nprocs=2, segments=1)
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 179, in resample_gauss
    reduce_data=reduce_data, nprocs=nprocs, segments=segments, with_uncert=with_uncert)
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 259, in _resample
    segments=segments)
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 376, in get_neighbour_info
    nprocs=nprocs)
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 548, in _query_resample_kdtree
    dt = resample_kdtree.data.dtype
AttributeError: 'cKDTree_MP' object has no attribute 'data'

======================================================================
ERROR: test_nearest_mp (pyresample.test.test_kd_tree.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 546, in _query_resample_kdtree
    dt = resample_kdtree.data_pts.dtype
AttributeError: 'cKDTree_MP' object has no attribute 'data_pts'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/test/test_kd_tree.py", line 266, in test_nearest_mp
    self.area_def, 50000, nprocs=2, segments=1)
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 101, in resample_nearest
    reduce_data=reduce_data, nprocs=nprocs, segments=segments)
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 259, in _resample
    segments=segments)
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 376, in get_neighbour_info
    nprocs=nprocs)
  File "/home/antonio/debian/git/build-area/pyresample-1.1.6/pyresample/kd_tree.py", line 548, in _query_resample_kdtree
    dt = resample_kdtree.data.dtype
AttributeError: 'cKDTree_MP' object has no attribute 'data'

----------------------------------------------------------------------
Ran 105 tests in 130.320s

FAILED (errors=3, skipped=1)

[mraspaud]

Seems to work on travis ? https://travis-ci.org/pytroll/pyresample/builds/112546653

[avalentino]

If my understanding is correct you install pykdtree while the error reported seems to happen when pykdtree is not installed and scipy is used instead.

P.S.: does pykdtree support python3?

[mraspaud]

Sorry, I had missed that

[mraspaud]

@avalentino Yes pykdtree runs on python 3.5: https://travis-ci.org/storpipfugl/pykdtree

[mraspaud]

@avalentino I fixed the last 3 errors in the pre-master branch, could see if it works for you also ?
I wasn't able to see the stalling though (python 3.3)...

[avalentino]

I wasn't able to see the stalling though (python 3.3)...

In my case it was python3.5
Sorry I'm travelling now so I can't provide more details about the configuration or check you fixes right now
I will do it ASAP

[avalentino]

OK, sorry for the long delay.
I still experiment the issue with test_nearest_resize on pyresample 1.2.4 (and the current master):

• Ubuntu 16.04 LTS (amd64)
• Python 3.5.2
• no pykdtree installed
• scipy 0.17.0

[djhoese]

@avalentino I saw this too when running on Windows with python 2.7, 3.4, and 3.5. It didn't actually stall necessarily, but it took ~25 minutes to finish test_nearest_resize. Once I fixed some things to get pykdtree to work on Windows it took 5-7 seconds to finish.

When you said it works as expected on python 2.7 is that using pykdtree or not?

[djhoese]

FYI I just tested this on my macbook with python 2.7. I ran python setup.py test -s pyresample.test.test_image.Test.test_nearest_resize with pykdtree and it took 5.592s. Without pykdtree 23.35 minutes. So the tests seem to require pykdtree to perform reasonably at all.

[avalentino]

@davidh-ssec I can confirm that using pykdtree all works fine

[djhoese]

@avalentino is installing pykdtree a "good enough" solution for you or would you like us to look in to why scipy is so much slower? We could also reduce the size of the test so that it passes in a reasonable amount of time even with scipy?

@mraspaud, thoughts?

[avalentino]

@davidh-ssec it is fine to set pykdtree as mandatory dependency (now it is also available a debian package for python3).

Thanks