[fitsbits] FITS Binary Table Proposals

In response to the comments received so far during the official Public
Comment Period on the proposals to incorporate the TDIMn and variable length
array conventions for FITS binary tables into the official FITS Standard, I
have created new detailed drafts of the proposals which are available at

http://fits.gsfc.nasa.gov/bintable_proposals.html

In particular, these new drafts contain the following changes:

1. As recommended by Tom McGlynn, the drafts now specify precisely the
wording that should be inserted into the FITS Standard.

2. As recommended by Bob Garwood, the TDIMn convention has been extended
slightly to allow the size of the array as given by the product of the
dimensions to be "less than or equal" to the allocated array size in the
FITS file.

3. As recommended by Francois Ochsenbein, a cautionary phrase has been added
to warn users that the variable length array convention may present
difficulties for applications that ingest FITS files via a sequential input
stream.

The wording of these proposals may still not be optimal, so any further
suggestions are welcome.


Bill Pence
--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)

They look pretty good to me, but here are a few comments.

Some tiny editorial points in the TDIM convention.
I'd just make it
8.3.6 Multidimensional Arrays
This is now the standard way to define multidimensional arrays in FITS binary tables.

For the second sentence
Any column with a dimensionality ... has ...
not "will have" since we use present tense everywhere else in the discussion.

Typo in "see Section[m] 8.3.5"


Similary for the Variable length arrays...

8.3.5 Variable Length Arrays

I'm not sure that the last sentence to the third paragraph means
anything, i.e., "Other established..." so I'd get rid
of it.

There is an upper limit to the size of the stored array: the maximum
value of a 4 byte integer

A few things that should probably be tied down specifically that are
currently a bit loose:

1. Are the sizes and offsets signed integers or unsigned?
If there were defined to be unsigned integers they could handle
objects twice as large and we wouldn't need to worry about
negative offsets.

2. Can THEAP be negative? The possibility of a negative 'gap'
is not explicitly forbidden, but probably should be. Or
do we want to allow reuse of data from the table as variable
length data. One could make interesting structures... Even if
we permit negative THEAP's (which I suspect will not be the case)
presumably it needs to be restricted to staying within the
current Data unit.


Tom McGlynn

William Pence wrote:
In response to the comments received so far during the official Public
Comment Period on the proposals to incorporate the TDIMn and variable
length array conventions for FITS binary tables into the official FITS
Standard, I have created new detailed drafts of the proposals which are
available at

http://fits.gsfc.nasa.gov/bintable_proposals.html

In particular, these new drafts contain the following changes:

1. As recommended by Tom McGlynn, the drafts now specify precisely the
wording that should be inserted into the FITS Standard.

2. As recommended by Bob Garwood, the TDIMn convention has been extended
slightly to allow the size of the array as given by the product of the
dimensions to be "less than or equal" to the allocated array size in the
FITS file.

3. As recommended by Francois Ochsenbein, a cautionary phrase has been
added to warn users that the variable length array convention may
present difficulties for applications that ingest FITS files via a
sequential input stream.

The wording of these proposals may still not be optimal, so any further
suggestions are welcome.


Bill Pence

I've made most of the changes suggested by Tom (you may need to clear your
browser's cache to force it to download the new version dated 12 November):

Thomas McGlynn wrote:
They look pretty good to me, but here are a few comments.

Some tiny editorial points in the TDIM convention.
I'd just make it
8.3.6 Multidimensional Arrays
This is now the standard way to define multidimensional arrays in FITS
binary tables.

Done.

For the second sentence
Any column with a dimensionality ... has ...
not "will have" since we use present tense everywhere else in the
discussion.

Not done. The presence of the 'will' adds some desireable emphasis. If the
column contains a multidimentional array, then it 'will' (meaning 'must')
have the TDIM keyword to indicate this fact.

Typo in "see Section[m] 8.3.5"

Corrected.

Similary for the Variable length arrays...

8.3.5 Variable Length Arrays

Done.

I'm not sure that the last sentence to the third paragraph means
anything, i.e., "Other established..." so I'd get rid
of it.

It means that other FITS conventions, such as how to define world
coordinates, that have been defined to apply to fixed length arrays can also
be applied to variable length arrays.

There is an upper limit to the size of the stored array: the maximum
value of a 4 byte integer

A comment to this effect has been added

A few things that should probably be tied down specifically that are
currently a bit loose:

1. Are the sizes and offsets signed integers or unsigned?
If there were defined to be unsigned integers they could handle
objects twice as large and we wouldn't need to worry about
negative offsets.

I don't think there is any question that these are intended to be signed
integers, so I added the adjective 'signed'

2. Can THEAP be negative? The possibility of a negative 'gap'
is not explicitly forbidden, but probably should be. Or
do we want to allow reuse of data from the table as variable
length data. One could make interesting structures... Even if
we permit negative THEAP's (which I suspect will not be the case)
presumably it needs to be restricted to staying within the
current Data unit.

If THEAP is negative, it would be pointing a location before the start of
the main data table, i.e., to somewhere in the Binary table header, or even
to a previous extension in the FITS file. It would be sensible to require
that THEAP be positive, but that will require a change to a different
section of the FITS Standard than what is being considered in this proposal.

Tom McGlynn

William Pence wrote:

In response to the comments received so far during the official Public
Comment Period on the proposals to incorporate the TDIMn and variable
length array conventions for FITS binary tables into the official FITS
Standard, I have created new detailed drafts of the proposals which
are available at

http://fits.gsfc.nasa.gov/bintable_proposals.html

In particular, these new drafts contain the following changes:

1. As recommended by Tom McGlynn, the drafts now specify precisely the
wording that should be inserted into the FITS Standard.

2. As recommended by Bob Garwood, the TDIMn convention has been
extended slightly to allow the size of the array as given by the
product of the dimensions to be "less than or equal" to the allocated
array size in the FITS file.

3. As recommended by Francois Ochsenbein, a cautionary phrase has been
added to warn users that the variable length array convention may
present difficulties for applications that ingest FITS files via a
sequential input stream.

The wording of these proposals may still not be optimal, so any
further suggestions are welcome.


Bill Pence


_______________________________________________
fitsbits mailing list

http://listmgr.cv.nrao.edu/mailman/listinfo/fitsbits


Bill Pence
--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)

By golly, now it starts getting interesting!
If a negative THEAP points to the start of the data in a previous HDU,
you can include an entire table or image in another table's cell.

You could try to prevent this kind of behavior by requiring THEAP to
be positive, but that would not prevent anyone from making THEAP
point beyond the current extension, causing essentially the same
problem. Non-discrimination requires that rather stricter and
preciser limits be put on THEAP in both directions (as Tom indicated),
if any limits are applied at all.

- Arnold

William Pence wrote:
...
2. Can THEAP be negative? The possibility of a negative 'gap'
is not explicitly forbidden, but probably should be. Or
do we want to allow reuse of data from the table as variable
length data. One could make interesting structures... Even if
we permit negative THEAP's (which I suspect will not be the case)
presumably it needs to be restricted to staying within the
current Data unit.

If THEAP is negative, it would be pointing a location before the start of
the main data table, i.e., to somewhere in the Binary table header, or even
to a previous extension in the FITS file. It would be sensible to require
that THEAP be positive, but that will require a change to a different
section of the FITS Standard than what is being considered in this proposal.
...

--------------------------------------------------------------------------
Arnold H. Rots Chandra X-ray Science Center
Smithsonian Astrophysical Observatory tel: +1 617 496 7701
60 Garden Street, MS 67 fax: +1 617 495 7356
Cambridge, MA 02138
USA http://hea-www.harvard.edu/~arots/
--------------------------------------------------------------------------

Back when this was designed the intention was certainly that, for a single
BINTABLE, the heap must be in the same extension as the main table, and heap
offsets must point to data segments which are contained entirely within
the heap region. If any part of a referenced heap segment is outside the
heap area that is an error. Other than that anything is possible, e.g.,
arrays can be aliased (multiple table records can point to the same area
of the heap), or there can be gaps in the heap which do not contain valid
data, e.g., as a result of an update. - Doug



On Fri, 12 Nov 2004, Arnold Rots wrote:

By golly, now it starts getting interesting!
If a negative THEAP points to the start of the data in a previous HDU,
you can include an entire table or image in another table's cell.

You could try to prevent this kind of behavior by requiring THEAP to
be positive, but that would not prevent anyone from making THEAP
point beyond the current extension, causing essentially the same
problem. Non-discrimination requires that rather stricter and
preciser limits be put on THEAP in both directions (as Tom indicated),
if any limits are applied at all.

- Arnold

William Pence wrote:
...
2. Can THEAP be negative? The possibility of a negative 'gap'
is not explicitly forbidden, but probably should be. Or
do we want to allow reuse of data from the table as variable
length data. One could make interesting structures... Even if
we permit negative THEAP's (which I suspect will not be the case)
presumably it needs to be restricted to staying within the
current Data unit.

If THEAP is negative, it would be pointing a location before the start of
the main data table, i.e., to somewhere in the Binary table header, or even
to a previous extension in the FITS file. It would be sensible to require
that THEAP be positive, but that will require a change to a different
section of the FITS Standard than what is being considered in this proposal.
...

--------------------------------------------------------------------------
Arnold H. Rots Chandra X-ray Science Center
Smithsonian Astrophysical Observatory tel: +1 617 496 7701
60 Garden Street, MS 67 fax: +1 617 495 7356
Cambridge, MA 02138
USA http://hea-www.harvard.edu/~arots/
--------------------------------------------------------------------------
_______________________________________________
fitsbits mailing list

http://listmgr.cv.nrao.edu/mailman/listinfo/fitsbits

In anticipation that the FITS committees will soon be asked to vote on the
proposals to incorporate the TDIMn and variable length array conventions
into the FITS Standard, I have updated the draft proposal documents
available at

http://fits.gsfc.nasa.gov/bintable_proposals.html

to explicitly list every single editorial change that will need to be made
to the text of the FITS Standard document. The main change will be to move
the text that is currently in the Appendices B.1 and B.2 into section 8.3
that defines the binary table format, but in addition, a number of other
small editorial changes will be needed, for example to update all the
existing references to Appendix B.

The only new modification of any technical substance that has been made to
this latest draft is to clarify that the minimum allowed value for the THEAP
keyword (which gives the byte offset to the start of the heap data area)
shall be the product of the NAXIS1 and NAXIS2 keyword values. This forces
the heap to follow the main table data and prevents any overlap of the two.
As Tom McGlynn previously pointed out, this requirement is not clearly
stated by the current wording.

Without this requirement someone might design a FITS table in which some of
the variable length array descriptors point to a location in the fixed
length part of the table. Even more extremely, if negative THEAP values are
allowed, then the descriptor could point to a previous header keyword
record, or conceivably to data in a previous HDU in the file. While it is
possible that this feature might be put to good use by a clever FITS file
designer, I think it is clear that this usage was never intended, and that
allowing this could cause complications for software that supports variable
length arrays.

Bill Pence
--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)