Python 3

[cdeil]

Python 3

[esheldon]

I do plan it, but it will take some work. The integers in the underlying C code have changed, as well as the strings. If you have experience with that I would be interested to hear.

[mfouesneau]

I think I made a first crack at porting to python 3. I am running some tests.

(porting is such a pain, however thanks for making this package super clean and simple!!)

[esheldon]

Morgan, thanks that sounds very interesting. Are you working against master? I've made a lot of changes recently.

[mfouesneau]

Yes just downloaded a few hours ago. Should I use another branch??

I forked the master to https://github.com/mfouesneau/fitsio.
I added the current major changes and tried to make it as clean as possible. I also found a few unused variables that I commented in the python code.

So far, it compiles, and can read fits files. But none of the tests are sucessful. Apparently module's method definition is not as trivial.

(Note: it should work as before in python 2)

[mfouesneau]

Erin, I am going to need some help to finish. I think I am quite close but it could help to have some insight and understanding the python class.

For instance, I currently have the following error and get quite a hard time to understand the iterator on hdus

TypeError: iter() returned non-iterator of type 'TableHDU'

This error is because you do not use the generator class (or yield) but make your own iterator using a method .next() which does not pass python 3 restrictions anymore.

Below is the full output from the test units.

testAsciiTableWriteRead (__main__.TestReadWrite) ... FAIL
testChecksum (__main__.TestReadWrite) ... ok
testExtVer (__main__.TestReadWrite) ... ok
testGZIPTileCompressedWriteRead (__main__.TestReadWrite) ... ok
testGZWriteRead (__main__.TestReadWrite) ... ok
testHCompressTileCompressedWriteRead (__main__.TestReadWrite) ... ok
testImageSlice (__main__.TestReadWrite) ... ok
testImageWriteRead (__main__.TestReadWrite) ... ok
testLowerUpper (__main__.TestReadWrite) ... ok
testMoveByName (__main__.TestReadWrite) ... ok
testPLIOTileCompressedWriteRead (__main__.TestReadWrite) ... ok
testRiceTileCompressedWriteRead (__main__.TestReadWrite) ... ok
testSlice (__main__.TestReadWrite) ... ok
testTableAppend (__main__.TestReadWrite) ... ok
testTableInsertColumn (__main__.TestReadWrite) ... ok
testTableIter (__main__.TestReadWrite) ... ERROR
testTableSubsets (__main__.TestReadWrite) ... ok
testTableWriteDictOfArrays (__main__.TestReadWrite) ... ok
testTableWriteDictOfArraysScratch (__main__.TestReadWrite) ... ok
testTableWriteDictOfArraysVar (__main__.TestReadWrite) ... FAIL
testTableWriteListOfArrays (__main__.TestReadWrite) ... ok
testTableWriteListOfArraysScratch (__main__.TestReadWrite) ... ok
testTableWriteListOfArraysVar (__main__.TestReadWrite) ... FAIL
testTableWriteRead (__main__.TestReadWrite) ... ok
testVariableLengthColumns (__main__.TestReadWrite) ... ERROR

======================================================================
ERROR: testTableIter (__main__.TestReadWrite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 937, in testTableIter
    for row_data in hdu:
TypeError: iter() returned non-iterator of type 'TableHDU'

======================================================================
ERROR: testVariableLengthColumns (__main__.TestReadWrite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 546, in testVariableLengthColumns
    fits.write(self.vardata)
  File "/usr/lib/python3.3/site-packages/fitsio/fitslib.py", line 466, in write
    table_type=table_type)
  File "/usr/lib/python3.3/site-packages/fitsio/fitslib.py", line 676, in write_table
    table_type=table_type)
  File "/usr/lib/python3.3/site-packages/fitsio/fitslib.py", line 762, in create_table_hdu
    names, formats, dims = array2tabledef(data, table_type=table_type)
  File "/usr/lib/python3.3/site-packages/fitsio/fitslib.py", line 3139, in array2tabledef
    form, dim = npy_obj2fits(data,name)
  File "/usr/lib/python3.3/site-packages/fitsio/fitslib.py", line 3273, in npy_obj2fits
    "not allowed in variable length columns" % name)
ValueError: Field 'Sobj' is an arrays of strings, this is not allowed in variable length columns

======================================================================
FAIL: testAsciiTableWriteRead (__main__.TestReadWrite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 1006, in testAsciiTableWriteRead
    f], d, "table subcol, '%s'" % f)
  File "test.py", line 1396, in compare_array
    w.size, 0, "testing array '%s' dim %d are equal" % (name, i))
AssertionError: 4 != 0 : testing array 'table subcol, 'i2scalar'' dim 0 are equal

======================================================================
FAIL: testTableWriteDictOfArraysVar (__main__.TestReadWrite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 800, in testTableWriteDictOfArraysVar
    write_success, "write should not raise an error")
AssertionError: False is not true : write should not raise an error

======================================================================
FAIL: testTableWriteListOfArraysVar (__main__.TestReadWrite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 900, in testTableWriteListOfArraysVar
    write_success, "write should not raise an error")
AssertionError: False is not true : write should not raise an error

----------------------------------------------------------------------
Ran 25 tests in 0.866s

FAILED (failures=3, errors=2)

[esheldon]

I will make a python3 branch that we can work on

[esheldon]

I think the iterator problem is easily fixed. In python2 the next(self) method is used, in python3 I think it is

__next__(self)

Maybe you can just define both temporarily so it will work with both python2 and 3

[esheldon]

argh, the markup messed that up. In python 3 it would be

__next__(self)

[esheldon]

The testVariableLengthColumns failure is because a numpy string is apparenly no longer a subclass of str,but bytes instead

[esheldon]

That check is done in npy_obj2fits

[mfouesneau]

Some progress. (I aliased __next__ to next and corrected the string tests)

testAsciiTableWriteRead (__main__.TestReadWrite) ... FAIL
testChecksum (__main__.TestReadWrite) ... ok
testExtVer (__main__.TestReadWrite) ... ok
testGZIPTileCompressedWriteRead (__main__.TestReadWrite) ... ok
testGZWriteRead (__main__.TestReadWrite) ... ok
testHCompressTileCompressedWriteRead (__main__.TestReadWrite) ... ok
testImageSlice (__main__.TestReadWrite) ... ok
testImageWriteRead (__main__.TestReadWrite) ... ok
testLowerUpper (__main__.TestReadWrite) ... ok
testMoveByName (__main__.TestReadWrite) ... ok
testPLIOTileCompressedWriteRead (__main__.TestReadWrite) ... ok
testRiceTileCompressedWriteRead (__main__.TestReadWrite) ... ok
testSlice (__main__.TestReadWrite) ... ok
testTableAppend (__main__.TestReadWrite) ... ok
testTableInsertColumn (__main__.TestReadWrite) ... ok
testTableIter (__main__.TestReadWrite) ... ok
testTableSubsets (__main__.TestReadWrite) ... ok
testTableWriteDictOfArrays (__main__.TestReadWrite) ... ok
testTableWriteDictOfArraysScratch (__main__.TestReadWrite) ... ok
testTableWriteDictOfArraysVar (__main__.TestReadWrite) ... FAIL
testTableWriteListOfArrays (__main__.TestReadWrite) ... ok
testTableWriteListOfArraysScratch (__main__.TestReadWrite) ... ok
testTableWriteListOfArraysVar (__main__.TestReadWrite) ... FAIL
testTableWriteRead (__main__.TestReadWrite) ... ok
testVariableLengthColumns (__main__.TestReadWrite) ... FAIL

======================================================================
FAIL: testAsciiTableWriteRead (__main__.TestReadWrite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "fitsio/test.py", line 1006, in testAsciiTableWriteRead
    f], d, "table subcol, '%s'" % f)
  File "fitsio/test.py", line 1396, in compare_array
    w.size, 0, "testing array '%s' dim %d are equal" % (name, i))
AssertionError: 4 != 0 : testing array 'table subcol, 'i2scalar'' dim 0 are equal

======================================================================
FAIL: testTableWriteDictOfArraysVar (__main__.TestReadWrite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "fitsio/test.py", line 805, in testTableWriteDictOfArraysVar
    self.compare_rec_with_var(self.vardata, d, "dict of arrays, var")
  File "fitsio/test.py", line 1447, in compare_rec_with_var
    "testing '%s' num field '%s' equal" % (name, f))
  File "fitsio/test.py", line 1396, in compare_array
    w.size, 0, "testing array '%s' dim %d are equal" % (name, i))
AssertionError: 8 != 0 : testing array 'testing 'dict of arrays, var' num field 'Svec' equal' dim 0 are equal

======================================================================
FAIL: testTableWriteListOfArraysVar (__main__.TestReadWrite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "fitsio/test.py", line 905, in testTableWriteListOfArraysVar
    self.compare_rec_with_var(self.vardata, d, "list of arrays, var")
  File "fitsio/test.py", line 1447, in compare_rec_with_var
    "testing '%s' num field '%s' equal" % (name, f))
  File "fitsio/test.py", line 1396, in compare_array
    w.size, 0, "testing array '%s' dim %d are equal" % (name, i))
AssertionError: 4 != 0 : testing array 'testing 'list of arrays, var' num field 'u1scalar' equal' dim 0 are equal

======================================================================
FAIL: testVariableLengthColumns (__main__.TestReadWrite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "fitsio/test.py", line 551, in testVariableLengthColumns
    self.vardata, d, "read all test '%s'" % vstorage)
  File "fitsio/test.py", line 1447, in compare_rec_with_var
    "testing '%s' num field '%s' equal" % (name, f))
  File "fitsio/test.py", line 1396, in compare_array
    w.size, 0, "testing array '%s' dim %d are equal" % (name, i))
AssertionError: 4 != 0 : testing array 'testing 'read all test 'fixed'' num field 'u1scalar' equal' dim 0 are equal

----------------------------------------------------------------------
Ran 25 tests in 1.720s

FAILED (failures=4)

[mfouesneau]

Hi Erin, what's the status of the python 3 version? (I saw a branch)

[esheldon]

I now have some real experience porting a python package to python 3 (biggles). I think I know the way forward. Hopefully I'll have progress soon.

[esheldon]

I've created a new branch py3. It passes all my tests on my linux box. Please give it a try.

I don't think the test suite has complete coverage of all possible issues, in particular regarding strings/bytes. Also I have not tested on OS X (and windows support is still minimal for both python 2 and 3).

[esheldon]

the test suite is fitsio.test.test()

[mfouesneau]

Hi Erin,

All tests work fine on my computer.

My preliminary other tests are also working fine.

I found that reading the header does not include TUNIT keywords in the value from :func:TableHDU.get_info but I could access through the :func:TableHDU.read_header method

I could not find how to keep the comments on keywords in the header definition. I suppose I have to pass a list of dicts, one per keywords?

That's a great job! Thanks!

[esheldon]

Is the header behavior different from python2?

Yes, to specify a comment for a keyword you need to use a list of dicts or a FITSHDR object.

[mfouesneau]

I think it's the same behavior.

[esheldon]

OK, can you open a separate issue for that? thanks.

[mfouesneau]

Done: #31