Test failures on mips*, hppa & hurd-i386

[pytroll]

Test failures on mips*, hppa & hurd-i386

[djhoese]

Have these tests ever passed on these operating systems and architectures or is this only happening with the newest PyPI version of pyresample?

I added a lot of changes to pyresample recently to add a new resampling method, but it looks like all of your failures are with nearest neighbor. However, I noticed in the logs:

/«PKGBUILDDIR»/docs/source/API.rst:41: WARNING: autodoc: failed to import module u'ewa'; the following exception was raised:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/sphinx/ext/autodoc.py", line 518, in import_object
    __import__(self.modname)
  File "/«PKGBUILDDIR»/pyresample/ewa/__init__.py", line 72, in <module>
    from pyresample.ewa import _ll2cr, _fornav
  File "/«PKGBUILDDIR»/pyresample/ewa/__init__.py", line 72, in <module>
    from pyresample.ewa import _ll2cr, _fornav
ImportError: cannot import name _ll2cr

I'm also debugging some build/test issues on Windows (see #39) may fix these issues too. I'll have to talk to @mraspaud about how debian packages are built. It may be as simple as needing certain numpy headers available.

[avalentino]

Have these tests ever passed on these operating systems and architectures or is this only happening with the newest PyPI version of pyresample?

Previous versions of the debian package worked on all architectures.
You can find old logs in [1]

[1] https://buildd.debian.org/status/package.php?p=pyresample

The error you quoted happens during the sphinx doc generation,
IMO it could be non related to the test failure.

[mraspaud]

Just a thought: Could it be that the pykdtree package isn't installed ?

[storpipfugl]

TL;DR: Change warning handling semantics to allow additional warnings

The missing warning should be generated by https://github.com/pytroll/pyresample/blob/master/pyresample/kd_tree.py#L312 so basically just a simple size check and raising a user warning

It works for most of the test cases using this code path except for the ones adding uncertainty estimates but the uncertainty calculation does not involve any warning handling so it doesn't make much sense.

There are 2 ways this error could happen:

1. Incorrect warning suppression. As it works on other platforms this would most likely be in a pyresample dependency
2. More warnings than expected

Both scenarios could be created by some dependency having different warning behaviour on these uncommon platforms.

The warning test currently asserts that only the expected warning is raised. It would be more appropriate to just assert the expected warning is raised: len(w) != 1 -> len(w) > 0 and loop through the warning list to search for the expected warning

[avalentino]

Just a thought: Could it be that the pykdtree package isn't installed ?

From the log it seems pykdtree is installed

[djhoese]

@storpipfugl what do you think about this: http://jakevdp.github.io/astropy/_modules/astropy/tests/helper.html#catch_warnings

The astropy project found that warnings sometimes depended on the order the tests were run. I'm also having similar warning issues in some of the Windows tests I'm doing, but I'm not sure if these are completely related yet.

[djhoese]

As mentioned on #40 I'm also seeing some weird warning handling when running on Windows, but only under python 3.4 and only on appveyor. On my local Windows python 3.4 environment it seems to work fine.

[djhoese]

@avalentino could you try testing the current pre-master branch. I've made some important changes with warning checking on tests. This should fix the problem you've been seeing.

[avalentino]

I have the following error both with python2 and python3 on my laptop:

======================================================================
FAIL: test_gauss_multi_uncert (pyresample.test.test_kd_tree.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antonio/projects/pyresample/pyresample/test/test_kd_tree.py", line 408, in test_gauss_multi_uncert
    len(w) != 1, 'Failed to create neighbour radius warning')
AssertionError: True is not false : Failed to create neighbour radius warning

----------------------------------------------------------------------
Ran 103 tests in 143.026s

FAILED (failures=1)

[djhoese]

Are you running from the latest pre-master branch? That test doesn't even have that exact check anymore.

https://github.com/pytroll/pyresample/blob/pre-master/pyresample/test/test_kd_tree.py#L343

[avalentino]

@davidh-ssec sorry, wrong branch.
On ac6132c I have still a build error:

In file included from /usr/include/python2.7/pyport.h:325:0,
                 from /usr/include/python2.7/Python.h:58,
                 from pyresample/ewa/_fornav_templates.cpp:1:
/usr/include/python2.7/numpy/npy_math.h:169:34: error: non-floating-point argument in call to function ‘__builtin_isnan’
             #define npy_isnan(x) isnan(x)
                                  ^
pyresample/ewa/_fornav_templates.cpp:282:52: note: in expansion of macro ‘npy_isnan’
                     if ((this_val == img_fill) || (npy_isnan(this_val))) {
                                                    ^
/usr/include/python2.7/numpy/npy_math.h:169:34: error: non-floating-point argument in call to function ‘__builtin_isnan’
             #define npy_isnan(x) isnan(x)
                                  ^
pyresample/ewa/_fornav_templates.cpp:289:51: note: in expansion of macro ‘npy_isnan’
                   if ((this_val != img_fill) && !(npy_isnan(this_val))) {
                                                   ^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Build flags are something like the following:

x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -c pyresample/ewa/_fornav_templates.cpp -o build/temp.linux-x86_64-2.7/pyresample/ewa/_fornav_templates.o -O3 -Wno-unused-function

[djhoese]

Ok thanks, I'll take a look. I'm 99% sure I know what is going on. I'll need to see if I can reproduce this though...

[djhoese]

@avalentino What version of gcc is that?

[avalentino]

$ gcc --version
gcc (Debian 5.4.0-6) 5.4.0 20160609

[djhoese]

And ldd --version?

[avalentino]

$ ldd --version
ldd (Debian GLIBC 2.23-1) 2.23

[djhoese]

Thanks. I just made a commit to pre-master. Could you try that? It should at least compile, but I'm writing a test now to make sure things integers are handled correctly.

[avalentino]

OK, all works now on my laptop but to test the fix on mips*, hppa & hurd-i386 I need to prepare a new version of the package.

I see some relevant changes so maybe it is batter to wait for a new stable release.
Which are your plans?

[djhoese]

If it is difficult for you to create a debian package then waiting for a new stable release is probably best. I've only made one commit since I asked you to test it, but it shouldn't make a difference in the compilation. I was the one who originally introduced the EWA resampling in to the package and I think I've fixed all the problems that that has caused.

I leave it to @mraspaud whether to produce a new release or not.

[djhoese]

A new release has been made available (version 1.2.5)

[avalentino]

Unfortunately the problem still persists on mips:

https://buildd.debian.org/status/fetch.php?pkg=pyresample&arch=mips&ver=1.2.5-1&stamp=1470599278

Please re-open this issue.

[Deni90]

On the latest version (1.2.5-1) two tests are failing:

=====================================================================
FAIL: test_custom_uncert (pyresample.test.test_kd_tree.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/«PKGBUILDDIR»/pyresample/test/test_kd_tree.py", line 102, in test_custom_uncert
    len(w) != 1, 'Failed to create neighbour warning')
AssertionError: Failed to create neighbour warning

======================================================================
FAIL: test_gauss_uncert (pyresample.test.test_kd_tree.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/«PKGBUILDDIR»/pyresample/test/test_kd_tree.py", line 79, in test_gauss_uncert
    len(w) != 1, 'Failed to create neighbour warning')
AssertionError: Failed to create neighbour warning

I've added debug prints into failing tests. There are more warnings than expected on mips*:

DBG: test_custom_uncert w[0] = {message : UserWarning('Searching for 8 neighbours in 3 data points',), category : 'UserWarning', filename : '/build/pyresample-X1AqqR/pyresample-1.2.5/pyresample/kd_tree.py', lineno : 314, line : None}
DBG: test_custom_uncert w[1] = {message : RuntimeWarning('invalid value encountered in multiply',), category : 'RuntimeWarning', filename : '/build/pyresample-X1AqqR/pyresample-1.2.5/pyresample/kd_tree.py', lineno : 834, line : None}

DBG: test_gauss_uncert w[0] = {message : UserWarning('Searching for 8 neighbours in 3 data points',), category : 'UserWarning', filename : '/build/pyresample-X1AqqR/pyresample-1.2.5/pyresample/kd_tree.py', lineno : 314, line : None}
DBG: test_gauss_uncert w[1] = {message : RuntimeWarning('invalid value encountered in multiply',), category : 'RuntimeWarning', filename : '/build/pyresample-X1AqqR/pyresample-1.2.5/pyresample/kd_tree.py', lineno : 834, line : None}

By changing the asserts to handle more warnings than expected, as @storpipfugl mentioned, I was able to build pyresample successfully for mips*.

Could you please take a look at Deni90@0188c7f
If this looks ok for you I can make a pull request.

[mraspaud]

Find by me, would be great with a PR!

[djhoese]

I just got home after a long trip. I'll try to look at this later tonight. Looks like I still need to fix pre-master on travis. I'd like to apply the PR on top of pre-master, but right now it is on master.

[djhoese]

@avalentino Could you test pre-master again? Thanks.

[avalentino]

Thanks @davidh-ssec, it works fine now [1]

There is still an issue with hurd-i386 but it is not an issue in pyresample, it is probably a configuration issue of the build tool.

[1] https://buildd.debian.org/status/package.php?p=pyresample

[djhoese]

@avalentino Thanks, glad to know we fixed the original issue. I was curious so I did a little digging in to the hurd-i386 issue and it seems like it is the platform you're running it on that is the problem. I found this mailing list entry that discusses the error message:

https://lists.debian.org/debian-hurd/2015/02/msg00000.html

Which points to this python bug report:

http://bugs.python.org/issue3770

I haven't read through the whole bug report but there may be something in there to help you solve the problem. If you need help with anything in pyresample let us know by creating a new issue. I'll close this issue since it seems like we fixed the original problem.

[avalentino]

I haven't read through the whole bug report but there may be something in there to help you solve the problem. If you need help with anything in pyresample let us know by creating a new issue. I'll close this issue since it seems like we fixed the original problem.

@davidh-ssec thank you very much for your support.