FITS long integer support (was [fitsbits] ADASS FITS BoFon Sunday)

We should perhaps consider adding a short section to the FITS Standard to
clarify the status of long (64-bit) integer support in FITS for the benefit
of users. Here is a first cut at the main points we might want to make.
Any comments?


1. There is in principle no limit on the magnitude of integer keyword values
in FITS headers and on the values in integer columns in ASCII tables,
however many existing software packages only support values within the range
of a signed 32-bit integer (-2147483648 to +2147483648). This effectively
limits the sizes of FITS images and tables (as given by the NAXISn keywords).

2. FITS does not officially support 64-bit twos-complement signed binary
integers in either primary array images, image extensions, or columns in
binary tables.

3. Informal proposals have been made to support 64-bit FITS images with
BITPIX = 64 and 64-bit integer columns in binary tables with TFORMn = 'rK'
(where 'r' here represents an integer repeat value). It has also been
proposed to support a 64-bit variable length array descriptor format in
binary tables using a TFORMn keyword with data type = 'Q' (analogous to the
existing 32-bit 'P' variable length array descriptor format).

4. Some FITS software packages support these informal proposals on a trial
basis, however FITS file designers are urged to use these formats with
caution. These formats should only be used in cases where a) there is no
reasonable alternative way to represent the information using standard FITS
formats, and b) where it is reasonably certain that the software packages
that are required to read the FITS files have been specifically designed to
support the long integer FITS formats.

5. It is possible that the FITS Standard might be modified in the future to
officially support images with BITPIX = 64 and/or binary table columns with
TFORMn = 'K' or 'Q'. Developers of new FITS software may wish to anticipate
this possibility and add compatibility with these possible future formats.


--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)

On Thu, 2004-10-21 at 10:19, William Pence wrote:
(-2147483648 to +2147483648).

Incredibly minor point: that last value should be 2^32-1 (+2147483647).

--
Stephen Walton
Dept. of Physics & Astronomy, Cal State Northridge

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQBBd/vnURWByv7S9xcRAtTCAKCsVA6K40lBNf9kvnI11mHs6lOhrQCf cFgT
Per5JcQXH1Wxg6kbTkD6UJA=
=pVB/
-----END PGP SIGNATURE-----

To clarify a couple of the previous comments in this thread which may have
given the impression that providing FITS support for long integers is
dependent on upgrading 32-bit hardware to 64-bit CPUs, this is generally not
the case. Nearly all current C/C++ compilers provide native support for
64-bit integers even on 32-bit machines. Java also has native support for
64-bit integers on all platforms.

I'm less clear about long integer support in Fortran. Fortran 90/95 I
believe does support this, but I don't recall seeing support for integer*8
in Fortran 77 (it is certainly not part of ANSI standard Fortran-77). So
this may boil down to a language divide: C/C++, Java, Fortran-90, and
probably most other new languages naturally support long integers, but
Fortran-77 doesn't.

Bill
--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)

Preben Grosbol wrote:
On Wednesday 20 October 2004 16:29, Eric Greisen wrote:

Since many operating systems and compilers do not support
64-bit integers (other than in sneaky hidden ways to read large
files), we should move extremely slowly to explicitly allow them in
FITS.

I can fully support this view. The only good argument for 64-bit
integers is pointers as uncertainties in physical quantities hardly
can justify such accuracy. So the issue if pointers to the HEAP
or reference columns to rows in tables with more than 2G rows
are important currently. I would prefer to wait until 64-bit machines
are the default in our community.

a case for #ifdefs by architecture?

Aloha,

Maren Purves, UKIRT

Recalling history, the 64-bit issue has come up before.

There was a significant discussion in 2001-09/10 which was led by
Bill Pence who produced the document seen here
http://heasarc.gsfc.nasa.gov/docs/he...bit/64bit.html

Bill took a poll and published these results on 2001-10-15:

YES 20 69%
NO 4 14%
UNDECIDED 3 10%
NO OPINION 2 7%
Total 29 100%

Prior to that, the topic arose in 1998-07 as part of the editing of
the NOST standard for FITS.

The proposals already posted are not far from being couched in formal
terms, the characters on either side of the issue have not changed
much.

How do we determine at what point the balance should tip?

Perhaps if there were an "official FITS feature registry" we could
create a new keyword to be inserted into the PHDU of a FITS file.
Repeated instances of that registry keyword could be used to describe
the features of FITS which must be supported if the reader is to
understand the FITS file fully.

By default a FITS file would conform to Wells et al. (1981)
Other features could have names and meaning like
GROUPS Greisen et al. (1981)
XTENSION Grosbol et al. (1988)
TABLE Harten et al. (1988)
IEEEFP IEEE floating point agreement (1990)
BINTABLE Cotton et al. (1995)
WCS1 Greisen & Calabretta (2002)
WCS2 Calabretta & Greisen (2002)
INT64 the current issue

And the registry should also contain significant unofficial things like

NOAOMOSAIC Valdes (1997)
GROUPING Jennings et al.
various Space Telescope conventions for FITS
lots of the content from
http://fits.gsfc.nasa.gov/fits_conventions.html
and anything else deemed relevant.

It would be relatively straightforward for a FITS reader to test its
capabilties against a set of such keywords and offer the option of
gracefully declining to interpret the full meaning of the file.

--
Steve Allen UCO/Lick Observatory Santa Cruz, CA 95064
Voice: +1 831 459 3046 http://www.ucolick.org/~sla
PGP: 1024/E46978C5 F6 78 D1 10 62 94 8F 2E 49 89 0E FE 26 B4 14 93

Peter Teuben wrote:

I'm less clear about long integer support in Fortran. Fortran 90/95 I
believe does support this, but I don't recall seeing support for integer*8
in Fortran 77 (it is certainly not part of ANSI standard Fortran-77). So
this may boil down to a language divide: C/C++, Java, Fortran-90, and
probably most other new languages naturally support long integers, but
Fortran-77 doesn't.


indeed, there is no official support, since integer*8 isn't in the standard.
However, both the intel and gnu compiler support it, and I abuse this feature
(with caution). I also recall the Cray compiler used to have a flag to the
compiler that made floats become double's essentially, so something in this
direction may be implemented by compiler writers.

- peter
_______________________________________________
fitsbits mailing list

http://listmgr.cv.nrao.edu/mailman/listinfo/fitsbits



[I addressed this more briefly in a note I sent to sci.astro.fits, but
I gather that will take a while before it comes out...]

Since we're talking about standards....

The notations integer*2, integer*4 and integer*8 are all non-standard
Fortran and are not included in any of the Fortran standards F66, F77, F90, F95 or
the impending F2003. Integer*2 has never been standard Fortran.
The default (and in standard Fortran till F90 the only) integer type must occupy
the same amount of space as a single precision real.

Fortran (i.e., the standard) has no mechanism to specify the length in
bytes of the desired variable. The standard way to get different
kinds of integers is something like

integer (kind=n) i,j,k

where n is an integer. For many (but not all) compilers the kind
in used exactly the same way as in the integer*n extension,
integer(kind=4) are normal 4 byte integers. However,
it is perfectly standard to have have integer(kind=4) to be 1 byte
variable and integer(kind=303) to be 4 byte integers. There are
functions that allow one to select the kind index based upon
the desired range (and precision for reals). However the value
of these 'kinds' is implementation dependent.


So be careful trying to use 'Fortran' standards here... Fortran is
a higher level language than C or Java and has much less connection
(in the standard!) to the byte representations.
So most Fortran's do support eight byte (and two byte) integers
but the standard mechanism is rather more complex than many suppose.


Regards,
Tom McGlynn

I do stand corrected - I just had AIPS declare and compute with an
INTEGER*8 and it worked on Linux (gnu compilers incl 2.95.3), Macs
(IBM compiler) and Solaris (SUN Fortran). Only the last even issued a
warning message. Therefore, I would support movement toward a
standard to allow 64-bit integers. Nonetheless, since it will take
re-coding of existing packages at a time when we are all stressed
financially, I would require that there be warnings attached to the
standard to discourage its use except where it is actually needed.
Over time, of course, that discouragement becomes less needed.

All too often I have seen people get excited about "new" things and
use them for the fun of it. FITS may be very useful as an internal
and archival format but it got those uses by adhering to the
requirements for transporting data. The transport in internal and
archive uses is across local platforms and across time rather than
between observatories, but it is nonetheless transport.

Eric Greisen