[fitsbits] TDIMn

I noticed that the WCS Paper III mentions that there's a pending motion before
the FITS committees to make the TDIMn keyword official. I was wondering what
the status of that motion was.

Actually, I would have thought that TDIMn became an official keyword when the
IAU-FWG approved WCS Papers I and II, but I can see the value of making it
official independent of the WCS formalism.

William Thompson

On Tue 2004/08/24 18:15:28 -0400, William Thompson wrote
in a message to:

Actually, I would have thought that TDIMn became an official keyword when the
IAU-FWG approved WCS Papers I and II, but I can see the value of making it
official independent of the WCS formalism.

Paper IV needs this (Appendix B of Cotton, Tody, & Pence, 1995) and also
the "substring array" convention (Appendix C). It would be good to have
them, and Appendix A, properly formalized now.

Mark Calabretta
ATNF

William Thompson wrote:
I noticed that the WCS Paper III mentions that there's a pending motion
before the FITS committees to make the TDIMn keyword official. I was
wondering what the status of that motion was.

This motion was deferred until the FITS committees were reorganized earlier
this year and until after the recent consideration of the FITS MIME type and
the Australia/New Zealand FITS committee proposals. Now that that work is
out of the way, one of the next orders of business for the FITS committees
should be to consider this motion.

This motion proposes to incorporate Appendix B.1 and B.2 into the official
FITS Standard (the appendices are currently unofficial). Both appendices
apply to FITS binary tables: B.1 describes the Variable Length Array
facility, and B.2 describes the TDIMn keyword. (See
http://archive.stsci.edu/fits/fits_standard/node74.html). There is also a
third appendix, B.3, dealing with substring arrays, but only a portion of
this proposed convention has ever been used or implemented (to my
knowledge), so only part of this appendix, if any, should be considered for
incorporation into the FITS Standard.

If anyone has any issues or concerns about making these appendices part of
the official FITS format Standard, now would be a good time to raise them
here on this newsgroup. The FITS committees will likely be considering this
issue soon, later this year.

Bill Pence
--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)

On Wednesday 25 August 2004 19:04, William Pence wrote:
This motion proposes to incorporate Appendix B.1 and B.2 into the official
FITS Standard *(the appendices are currently unofficial). *Both appendices
apply to FITS binary tables: B.1 describes the Variable Length Array
facility, and B.2 describes the TDIMn keyword. (See
http://archive.stsci.edu/fits/fits_standard/node74.html). *There is also a
third appendix, B.3, dealing with substring arrays, but only a portion of
this proposed convention has ever been used or implemented (to my
knowledge), so only part of this appendix, if any, should be considered for
incorporation into the FITS Standard.

If anyone has any issues or concerns about making these appendices part of
the official FITS format Standard, now would be a good time to raise them
here on this newsgroup. *The FITS committees will likely be considering
this issue soon, later this year.

I see no problems with the B.2 (TDIMn) proposal as it only deals with the
interpretation of a one dimensional array. Certainly it would be good to
have the appendix B.2 included officially into the standard soon.

With respect to B.1 (variable length arrays), I have some reservations.
This proposal requires a full implementation of the HEAP and the P column
format. Technically, the B.1 proposal is good and I would endorse it as such.
The issue is that (as least to my taste) FITS files which conform to the
standard should be readable by most readers. The success of FITS is
to some extend based on this, namely that one having a standard FITS
file does not need to worry - it is, by default, readable by almost all
systems.

If B.1 is included into the standard I am concerned that a significant number
of systems will be unable to read and access such data. It is not just a
matter of if HEAPs can be accessed by software libraries but more if full
data processing systems can access such data as if they had been in a
'normal' table column. Beside leaving B.1 as it is, there may the following
options to include it into the standard:
1) Adding an clear, explicit preample which states that users cannot assume
the HEAP concept to be implemented by all readers and therefore should
be cautious in using it.
2) Verify that most readers actually already implements the HEAP which
would satisfy my concern.

On B.3, it would be good to know who is actually using it? One should
only include, into the standard, features which are used by a broad community.

Preben Grosbol

I share much of Preben's uncertainties about adopting the appendices.

The TDIMn (B.2) is absolutely needed and is or soon will be used. It
is also a convention that can be ignored by general readers that do
not need to know the internal details of a table column's data. It
uses a previously non-standard keyword and so has been ignored by
general readers.

The substring convention (B.3) is also relatively harmless in that a
general routine may ignore all the substring notation. However, that
notation appears as part of the data value of a standard keyword and
so may (should??) confuse a general reader that does not support the
extension. Has this been used by anyone?

The heap convention however does worry me. It changes the data
structure in the FITS file and I expect that general FITS readers will
not normally support the convention internally. In fact, I have
always regarded this convention with the worry that it encourages a
lack of discipline in designing the data tables in the first place.
Thus an over-worked designer may decide to dump all sorts of data into
a single table piling most of the "table" data into an unstructured
heap, rather than building two or more cleanly structured tables
separating unlike data into separate columns/tables.

I do not doubt that the heap may be an excellent choice for an
internal format in a data system geared to handle it. But as a
communication-of-data choice I expect that it will not be implemented
generally and hence stand in the way of communication.

Eric Greisen

The NASA/ESA SOHO project has been using TDIMn since 1995. I strongly support
adopting TDIMn officially.

The IDL software I wrote many years ago for binary tables does support variable
length arrays, but I'm not aware of any projects within the solar community
actually using it. I can ask around.

I'm not aware of any projects using the substring array convention, and am not
sure how well it's supported. My own software does not explicitely support it,
but wouldn't be confused by its presence.

Bill Thompson


Eric Greisen wrote:
I share much of Preben's uncertainties about adopting the appendices.

The TDIMn (B.2) is absolutely needed and is or soon will be used. It
is also a convention that can be ignored by general readers that do
not need to know the internal details of a table column's data. It
uses a previously non-standard keyword and so has been ignored by
general readers.

The substring convention (B.3) is also relatively harmless in that a
general routine may ignore all the substring notation. However, that
notation appears as part of the data value of a standard keyword and
so may (should??) confuse a general reader that does not support the
extension. Has this been used by anyone?

The heap convention however does worry me. It changes the data
structure in the FITS file and I expect that general FITS readers will
not normally support the convention internally. In fact, I have
always regarded this convention with the worry that it encourages a
lack of discipline in designing the data tables in the first place.
Thus an over-worked designer may decide to dump all sorts of data into
a single table piling most of the "table" data into an unstructured
heap, rather than building two or more cleanly structured tables
separating unlike data into separate columns/tables.

I do not doubt that the heap may be an excellent choice for an
internal format in a data system geared to handle it. But as a
communication-of-data choice I expect that it will not be implemented
generally and hence stand in the way of communication.

Eric Greisen

Although I can't name a list of sites or packages from memory, my impression
has been that the variable length array facility is used quite a lot. One
can't always allocate a fixed size array for data arrays, in which case there
is a real need for the variable length array facility. In a table containing
such columns a reader which does not support variable length arrays should
still be able to read the main table, which has fixed length records, although
it would not be able to read the variable length arrays. Doug


From Preben:
With respect to B.1 (variable length arrays), I have some reservations.
This proposal requires a full implementation of the HEAP and the P column
format. Technically, the B.1 proposal is good and I would endorse it as such.
The issue is that (as least to my taste) FITS files which conform to the
standard should be readable by most readers. The success of FITS is
to some extend based on this, namely that one having a standard FITS
file does not need to worry - it is, by default, readable by almost all
systems.

If B.1 is included into the standard I am concerned that a significant number
of systems will be unable to read and access such data. It is not just a
matter of if HEAPs can be accessed by software libraries but more if full
data processing systems can access such data as if they had been in a
'normal' table column. Beside leaving B.1 as it is, there may the following
options to include it into the standard:
1) Adding an clear, explicit preample which states that users cannot assume
the HEAP concept to be implemented by all readers and therefore should
be cautious in using it.
2) Verify that most readers actually already implements the HEAP which
would satisfy my concern.

On B.3, it would be good to know who is actually using it? One should
only include, into the standard, features which are used by a broad community.

Preben Grosbol wrote:

....

With respect to B.1 (variable length arrays), I have some reservations.
This proposal requires a full implementation of the HEAP and the P column
format. Technically, the B.1 proposal is good and I would endorse it as such.
The issue is that (as least to my taste) FITS files which conform to the
standard should be readable by most readers. The success of FITS is
to some extend based on this, namely that one having a standard FITS
file does not need to worry - it is, by default, readable by almost all
systems.

If B.1 is included into the standard I am concerned that a significant number
of systems will be unable to read and access such data. It is not just a
matter of if HEAPs can be accessed by software libraries but more if full
data processing systems can access such data as if they had been in a
'normal' table column. Beside leaving B.1 as it is, there may the following
options to include it into the standard:
1) Adding an clear, explicit preample which states that users cannot assume
the HEAP concept to be implemented by all readers and therefore should
be cautious in using it.
2) Verify that most readers actually already implements the HEAP which
would satisfy my concern.


The table at http://fits.gsfc.nasa.gov/fits_libraries.html suggests that
most FITS libraries that support binary tables do support variable length
arrays. Preben's comments could have been made about binary tables
(or even more strongly ASCII tables) which are not supported by substantial
numbers of FITS readers. Indeed, if one were to look at the distinct software
implementations of FITS readers I suspect that the vast majority are simple
readers that handle only primary FITS images. (I've written many such readers
over the past couple of decades for a variety of special projects).
However when we look at the libraries that are being used in the development
of new major software, then I think that it is clear that both binary
tables and variable length arrays are widely supported and used.

That said I'm not sure I agree with Preben's premise that putting something
in the standard implies a mandate that FITS readers should be able to
handle it. Certainly the many FITS readers that only handle images will continue
to ignore binary tables regardless of whether they have variable length records,
and FITS readers that wish to ignore heap data in binary tables
should feel free to do so.

In a similar vein, the new WCS papers suggest some quite sophisticated and complex
ways that coordinate information can be encoded. I expect that the vast
majority of readers will support only elements of the WCS that are necessary
to the kinds of science they are intended to support rather than providing
explicit support for all of the WCS features.

Regards,
Tom McGlynn



On B.3, it would be good to know who is actually using it? One should
only include, into the standard, features which are used by a broad community.

Preben Grosbol

In response to questions about the extent to which the FITS BINTABLE
conventions (defined in Appendix B of the NOST FITS Standard) have been
used, here are the cases that I'm aware of. Please let me know of any other
examples where these conventions have been used.

B.1 "Variable Length Array" facility
------------------------------------

This convention is supported by many FITS libraries (see
http://fits.gsfc.nasa.gov/fits_libraries.html) and has been used for the
past decade in a number of publicly distributed FITS data sets, including:

- The Copernicus satellite (OAO-3) spectral data, distributed
by the Multimission Archive at Space Telescope (MAST).

- The standard Response Matrix calibration files (RMF) that
are used by almost all X-ray satellite missions since the
mid 1990s.

- Arecibo Observatory radio telescope data. The format of
these data files is based on the SDFITS convention for
interchange of single dish data in radio astronomy which
explicitly supports the variable length array convention.

- The IRAF data analysis software uses the variable length
array convention when storing 'Pixel Lists' in FITS format.

In addition, the 'Tiled' image compression convention (see
http://heasarc.gsfc.nasa.gov/docs/he...ess_image.html)
is layered on top of the variable length array convention and is currently
used by several groups, including:

- Mosaic CCD camera data taken at the Canada-France-Hawaii
Telescope (CFHT) and publicly distributed by the CADC.

- The CCD image archive taken for the Optical Gravitational
Lensing Experiment (OGLE) using the 1.3m Warsaw telescope
at Las Campanas Observatory, Chile.


B.2 The TDIMn keyword convention
---------------------------------

This convention is widely supported by many software packages and is used in
many FITS data sets, too numerous to mention individually.


B.3 The "Substing Array" convention
------------------------------------

I'm not aware of any software packages or any FITS data files that support
or use the full form of this proposed convention, as in these examples:

TFORMn = '30A:SSTR10' / fixed length 10-char substrings
TFORMn = '30A:SSTR10/032' / variable-length substrings

The CFITSIO library does support the simpler convention for specifying
fixed-length substrings, e.g.,

TFORMn = '30A10'

however I don't know of any FITS files that have actually used this convention.


-Bill Pence
--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)

William Pence writes:
- Mosaic CCD camera data taken at the Canada-France-Hawaii
Telescope (CFHT) and publicly distributed by the CADC.

??? As far as I know, mosaic images taken at CFHT are distributed
as multiple IMAGE extensions (one extension per CCD). It is of
course possible that the CADC repackages them in a new format, but
that would be new to me. Do you have a pointer to where you got this
information from?