|
|
Thread Tools | Display Modes |
#1
|
|||
|
|||
[fitsbits] Justifications for 64-bit integer FITS images
This list contains various examples, collected with the help of some of
my colleagues, to justify adding support in FITS for 64-bit integer images. These examples specifically deal with cases involving 64-bit integer FITS images with BITPIX = 64, not FITS table columns. A. Scientific justifications ---------------------------- These cases illustrate scientific problems that require images with more than 32-bits of precision. 1. Catalogs of astronomical sources or observations will soon (if not already) contain more objects than can be represented with 32-bit integers. If one computes histograms of the parameters in these large databases then 32-bit integers will not be sufficient in general to store the histogram values in a FITS primary array. 2. The precise measurement of time and time intervals can result in more clock 'ticks' than can be represented by 32-bit integers. This has been used to justify the need for 64-bit integer 'TIME' columns in tables, but it can just as well be used to justify the need for 64-bit integer arrays. For example, one might construct a 1-D FITS primary array that contains a sequence of times for some observed repetitive phenomena, such as the observed times of the flux maximum in a rotating object. Or, one might have a N-satellite x N-interval 2-D array where each entry is the time received from a given GPS satellite as seen by some other satellite. Why should researchers be forced to only represent these arrays of time values in table columns and not also as a 1-dimensional FITS primary arrays or image extensions? 3. When binning or summing pixels in a CCD array the sums may exceed 2 GB. For example, if one has a 10x10 array of 1024x1024 CCDs with an average image pixel value greater than 2000, then one cannot create the 10x10 summed image using integer*4 pixels.. 4. Use of FITS arrays for cumulative statistics (e.g., the summed counts to this point) will be strongly limited by restrictions to I*4. 5. Users may wish to create arrays containing the indices to various pixels in detectors that will soon exceed 2 GB pixels in size. Note that this may involve non-rectangular pixelizations like HTM or HEALPix so that the total number of pixels matters (they have a 1D index, instead of having separate X and Y coordinate indices). 6. The new field of gravitational wave astronomy involves very precise measurements of time and position. While it is difficult now to cite a specific case where arrays of 64-bit integers will be required, it would be prudent to begin supporting them now in FITS in case they are needed in this, or in other emerging fields of astronomical research. 7. Availability of 64-bit integer data may make FITS more compatible with storage of data from numerical simulations 8. Arrays of encryption keys and such may be stored within FITS images and could need the extra range provided by 64-bit integers. B. Practical or sociological justifications ------------------------------------------- The following cases illustrate various practical reasons for supporting 64-bit integer arrays in FITS files. 1. 64-bit integers are now almost universally available in standard programming languages. Applications programmers will naturally compute arrays of values using of this data type and will expect to be able to store the values in a FITS file in their native form. Even in cases where the scientific observations themselves do not require more than 32-bits of precision, the extreme largest and smallest possible values supported by that data type (often given by a predefined constant like LONG_MAX or LONG_MIN) are often used to flag undefined pixels and other conditions. It would be costly and a potential source of error or confusion if these extreme values had to be converted into the range of a less precise data type for storage in a FITS file. 2. FITS is the working data analysis format of choice for many current researchers. These researchers often write their own personal programs to do some specific type of analysis and use FITS files to store intermediate results. If a researcher has calculated a 64-bit integer array of numbers, and wants to temporarily store the values on disk, the simplest way to do this is just to write the values to a 1-dimensional FITS primary array. This "quick and dirty" type of programming is still widely practiced and is often the most cost effective way to solve small problems. 3. In order to "future proof" applications, it is prudent to use arrays of 64-bit integers even if 32-bit integers seem adequate for for current needs. Software applications can end up being used for decades, and it is difficult to foresee future needs. It costs very little to build in 64-bit integer support when an application is being designed and written, but it may be difficult and costly to retrofit 64-bit support into the application later on. 4. Nearly all other scientific data formats support the 64-bit integer data type, so FITS suffers by comparison. We are being encouraged to become more interdisciplinary, so it is important to be able to import data files from other fields (space physics, planetary astronomy, earth observations) into FITS to be able to process such images using astronomical image processing algorithms. This is made more difficult if FITS does not support the same basic data types as the other sciences. 5. From an abstract "data model" point of view, a primary array or image extension is identical to a table with 1 row and 1 column that contains the image as a vector. Conceptually, one should be able to convert any table vector into a FITS image array, and vice-versa. Because of this similarity, support for 64-bit images comes at little extra cost if that data type is already supported in tables. 6. In some of these cases, real*8 arrays could be used instead of integer*8, but the standard data compression algorithms do not perform as well on real data compared to integer data. This can be a real advantage when sending compressed data over slow networks. 7. "Build it, and they will come": One should not underestimate the ingenuity of users and programmers in finding uses for new features. Once 64-bit integers are available in FITS, users will surely discover uses that we have not even considered. |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
FITS long integer support (was [fitsbits] ADASS FITS BoFon Sunday) | William Pence | FITS | 6 | October 22nd 04 08:23 PM |
[fitsbits] FITS long integer support | Steve Allen | FITS | 0 | October 21st 04 06:22 PM |
[fitsbits] Start of the FITS MIME type Public Comment Period | William Pence | FITS | 8 | June 17th 04 06:08 AM |
[fitsbits] Happy Birthday, FITS! | Don Wells | FITS | 0 | March 28th 04 01:58 PM |
[fitsbits] BLANK keyword misinterpretation | Steve Allen | FITS | 4 | November 21st 03 04:42 PM |