[fitsbits] Large FITS files

On Mon 2007-07-30T10:08:07 -0400, Robert J. Harris hath writ:
I'm looking to process large ( ~ GB) FITS datacubes. They are large
enough not to load into memory. What I would like to do is input the
cube, to perform a filtering operation on each plane of the cube, and then
output another cube made up of the planes of the original cube after they
have been processed. If it weren't for the last part of it, I could use
the extended filename syntax of cfitsio, but I do need to put the planes
back together, and it's not obvious how I might do that. Any help is
appreciated. Thanks in advance!

Our 64 Mpix images from the DEIMOS 8k*8k imager aren't that big, but
they seemed a bit more of a challenge years ago when we were first
preparing to write them. We use the CFITSIO fits_create_memfile()
call after using the Unix shared memory routines to get a chunk of
shared memory. Then we use the other routines to create the FITS
elements and store the offsets in data structures which are used
thereafter by code which is unaware that it is actually addressing
memory which conforms to FITS.

In early tests we experimented with using Unix memory mapped files
instead of shared memory. It worked pretty much the same as far as
the use of fits_create_memfile() was concerned, but we found that the
operating system (with a little tuning of the kernel params) was
better at deciding how to do the swapping than our code was at trying
the same thing.

--
Steve Allen WGS-84 (GPS)
UCO/Lick Observatory Natural Sciences II, Room 165 Lat +36.99855
University of California Voice: +1 831 459 3046 Lng -122.06015
Santa Cruz, CA 95064 http://www.ucolick.org/~sla/ Hgt +250 m