fitsio write too slow on some file systems

[menanteau]

fitsio write too slow on some file systems

[esheldon]

The bottleneck seems to be writing the header key words, which I currently do one at a time.

Doing these in bulk would be better, but will take some work

[menanteau]

Can you tell if it is a fitsio or cfitsio issue?

[esheldon]

On 6/25/15, Felipe Menanteau [email protected] wrote:

Can you tell if it is a fitsio or cfitsio issue?

it seems to be cfitsio, or at least the way I'm using cfitsio

Erin Scott Sheldon
Brookhaven National Laboratory erin dot sheldon at gmail dot com

[esheldon]

I have fixed this partially. Writing the header is now much faster.
This is now in master. The overall writing is probably fast
enough to be usable, but see below

Also, the overall write speed on a normal disk is now faster than pyfits

On ext3

Using fitsio version 0.9.7
# Reading in DECam_00158995_09.fits
# Done: 0m 0.11s
# Writing file test.fits
# Done: 0m 0.33s
# ALL done: 0m 0.44s


# Reading in DECam_00158995_09.fits
Filename: DECam_00158995_09.fits
No.    Name         Type      Cards   Dimensions   Format
0    SCI         PrimaryHDU     360   (2048, 4096)   float32   
1    MSK         ImageHDU       108   (2048, 4096)   int16   
2    WGT         ImageHDU        72   (2048, 4096)   float32   
None
# Done: 0m 0.02s
# Writing file test.fits
# Done: 0m 1.44s
# ALL done: 0m 1.45s

On GPFS
However, the actual image write is still slower on GPFS, with pyfits
taking 2s and fitsio 30s. 30s is tolerable but we can do better. I'll
be looking into it.

[esheldon]

I have also fixed the image writing. The writing on GPFS is now also faster than pyfits

fitsio GPFS

Using fitsio version 0.9.7
# Reading in DECam_00158995_09.fits
# Done: 0m 0.15s
# Writing file test.fits
# Done: 0m 1.27s
# ALL done: 0m 1.42s

pyfits GPFS

# Reading in DECam_00158995_09.fits
Filename: DECam_00158995_09.fits
No.    Name         Type      Cards   Dimensions   Format
0    SCI         PrimaryHDU     360   (2048, 4096)   float32   
1    MSK         ImageHDU       108   (2048, 4096)   int16   
2    WGT         ImageHDU        72   (2048, 4096)   float32   
None
# Done: 0m 0.04s
# Writing file test.fits
# Done: 0m 2.45s
# ALL done: 0m 2.49s

[esheldon]

please pull from master and give it a try