Failure on (invalid) FITS tables from SDSS CasJobs

[esheldon]

Failure on (invalid) FITS tables from SDSS CasJobs

[esheldon]

Thanks for the pointer; I should probably at least give a better error message. I'll try to give this a look later in the week.

[esheldon]

I added a check for tdim None. tdim None happens when a call to
fits_read_tdimll
fails.

I'm tentatively going to continue and read these as a 1-d array, although that
will be surprising for high dimension columns that were simply declared wrong.

I am printing a warning message
'Warning:',name,'has malformed TDIM, reading as 1-d array'

[esheldon]

Actually that would not work; I'm raising an error

[esheldon]

Dustin, Can you please post one of these bad files somewhere?

[dstndstn]

Sure -- sorry for not including one in my original report

http://skyservice.pha.jhu.edu/CasJobsOutput/FITS/MyTable_dstn.fit

[esheldon]

Thanks; checking that file also prompted me to update the repr so it shows the column has a bad TDIM.

Can you check if the current behavior in master makes sense to you?

[dstndstn]

uh, I get,

python -c "import fitsio; F=fitsio.FITS('MyTable_dstn.fit'); print F[1]['survey']"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/fitsio/lib/python/fitsio/fitslib.py", line 2857, in __repr__
    if len(tdim) > 1:
TypeError: object of type 'NoneType' has no len()

[esheldon]

Ah, yes, I had not fixed the repr in the column subset. I factored out this stuff to a function so I only need to fix this in one place now.

Can you try again when you get time?

[esheldon]

by "try again" I mean git pull and try the new version

[dstndstn]

I get:

python -c "import fitsio; F=fitsio.FITS('MyTable_dstn.fit.1'); print 'Survey is:'; print F[1]['survey']; print F[1].read()['survey']")
Survey is:
  file: MyTable_dstn.fit.1
  extension: 1
  type: BINARY_TBL
  rows: 10
  column subset:
    survey              S1  array[bad TDIM]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/fitsio/lib/python/fitsio/fitslib.py", line 1624, in read
    data = self.read_all(**keys)
  File "/usr/local/fitsio/lib/python/fitsio/fitslib.py", line 1756, in read_all
    dtype, offsets, isvar = self.get_rec_dtype(**keys)
  File "/usr/local/fitsio/lib/python/fitsio/fitslib.py", line 2454, in get_rec_dtype
    dt,isvar = self.get_rec_column_descr(colnum, vstorage)
  File "/usr/local/fitsio/lib/python/fitsio/fitslib.py", line 2506, in get_rec_column_descr
    shape = tdim2shape(tdim, name, is_string=(npy_type[0] == 'S'))
  File "/usr/local/fitsio/lib/python/fitsio/fitslib.py", line 2916, in tdim2shape
    raise ValueError("field '%s' has malformed TDIM" % name)
ValueError: field 'survey' has malformed TDIM

[esheldon]

Yes, that is the intended error. Do you think the behavior should be different? One possibility is to fall back to reading it into a 1-d array column.

[dstndstn]

Ok, that's totally reasonable. The fallback would be handy in this case, but since the work-around is so easy (just remove the TDIM from the input to make it valid FITS), I wouldn't spend any time on it if I were you.

Thanks again!