[fitsbits] Proposed Changes to the FITS Standard

On Tue 2007-08-21T00:07:29 -1000, Maren Purves hath writ:
It being "once FITS always FITS" I object to making the
EXTEND keyword optional.

A question for those who have been around longer than I:

Does "once FITS always FITS" mean

Anything ever explicitly allowed can never be disallowed
or
Anything ever explicitly allowed must always remain required
or
Anything not explicitly disallowed must always remain allowed
or
Anything ever explicitly disallowed can never be allowed
or some other rearrangment of those notions, or what?

In RFC 4047 we have
no change may be made to FITS that invalidates existing files
but I'm not sure that's what everyone thinks.

--
Steve Allen WGS-84 (GPS)
UCO/Lick Observatory Natural Sciences II, Room 165 Lat +36.99855
University of California Voice: +1 831 459 3046 Lng -122.06015
Santa Cruz, CA 95064 http://www.ucolick.org/~sla/ Hgt +250 m

William Thompson wrote:
The SECCHI instrument on the STEREO mission produces some files with table
extensions, and other files which don't. The value of the EXTEND keyword is set
to either T or F to account for this.

Strictly speaking, this does not conform to the definition of the EXTEND
keyword in the FITS Standard, which states that the value field *shall*
contain a logical constant with the value T, and thus must never be set
to F. All the more reason I think to demote this keyword by making it
optional instead of required if the file contains extensions.

Bill Pence
--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)

It's not clear to me that the FITS standard forbids setting EXTEND=F.

EXTEND Keyword The use of extensions necessitates a single additional keyword in the
primary header of the FITS file. If the FITS file may contain extensions, a card image
with the keyword EXTEND and the value field containing the logical value T must appear
in the primary header immediately after the last NAXISn card image, or, if NAXIS=0,
the NAXIS card image. The presence of this keyword with the value T in the primary
header does not require that extensions be present.

To me, having EXTEND=F when one knows that no extensions are present is one of
those "bedrock of logic" that Rob Seaman refers to.

Bill Thompson


William Pence wrote:
William Thompson wrote:

The SECCHI instrument on the STEREO mission produces some files with table
extensions, and other files which don't. The value of the EXTEND keyword is set
to either T or F to account for this.


Strictly speaking, this does not conform to the definition of the EXTEND
keyword in the FITS Standard, which states that the value field *shall*
contain a logical constant with the value T, and thus must never be set
to F. All the more reason I think to demote this keyword by making it
optional instead of required if the file contains extensions.

Bill Pence

--
William Thompson
NASA Goddard Space Flight Center
Code 671
Greenbelt, MD 20771
USA

301-286-2040

The difference here is that I was quoting the revised definition in the
new draft standard, which doesn't allow for the possibility that EXTEND
= F. Given that there are existing FITS files with EXTEND = F, the
definition should probably be revised to at least not prohibit this usage.

One problem with relying on the veracity of EXTEND = F is that even if
the FITS file doesn't currently have extensions, they might be added at
a later time, and there is a good chance the EXTEND keyword value would
not get updated to T in the process. So just as software should not
assume that a file has extensions if EXTEND = T, it could be unwise to
assume there are no extensions just because EXTEND = F.

The purpose of this proposed change to the standard is to allow a FITS
file to containing extensions even if the primary array neglects to
contain the EXTEND = T keyword. Similarly, a FITS file should be
allowed to have extensions if EXTEND = F, even though this is
contradictory, simply because the actual existence of the extension
should take precedence, regardless of what the EXTEND keyword implies.
Software can easily determine for itself whether there are extensions by
examining the file, so there is no need to depend on the EXTEND keyword,
whose presence or value may be unreliable.

Bill Pence

William Thompson wrote:
It's not clear to me that the FITS standard forbids setting EXTEND=F.

EXTEND Keyword The use of extensions necessitates a single additional keyword in the
primary header of the FITS file. If the FITS file may contain extensions, a card image
with the keyword EXTEND and the value field containing the logical value T must appear
in the primary header immediately after the last NAXISn card image, or, if NAXIS=0,
the NAXIS card image. The presence of this keyword with the value T in the primary
header does not require that extensions be present.

To me, having EXTEND=F when one knows that no extensions are present is one of
those "bedrock of logic" that Rob Seaman refers to.

Bill Thompson


William Pence wrote:
William Thompson wrote:

The SECCHI instrument on the STEREO mission produces some files with table
extensions, and other files which don't. The value of the EXTEND keyword is set
to either T or F to account for this.

Strictly speaking, this does not conform to the definition of the EXTEND
keyword in the FITS Standard, which states that the value field *shall*
contain a logical constant with the value T, and thus must never be set
to F. All the more reason I think to demote this keyword by making it
optional instead of required if the file contains extensions.

Bill Pence



--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)

Steve Allen wrote:
On Tue 2007-08-21T00:07:29 -1000, Maren Purves hath writ:
It being "once FITS always FITS" I object to making the
EXTEND keyword optional.

A question for those who have been around longer than I:

I'm not sure I've been around FITS for longer than you have,
but none of these:

Does "once FITS always FITS" mean

Anything ever explicitly allowed can never be disallowed
or
Anything ever explicitly allowed must always remain required
or
Anything not explicitly disallowed must always remain allowed
or
Anything ever explicitly disallowed can never be allowed
or some other rearrangment of those notions, or what?

sound like what I'd be after.

Anything ever required cannot be disallowed
and anything ever disallowed cannot be required

is about as far as I would go.

In RFC 4047 we have
no change may be made to FITS that invalidates existing files
but I'm not sure that's what everyone thinks.

By the same token: anything that writes FITS cannot write things
that are explicitly disallowed and anything that reads FITS cannot
throw errors/exit when reading valid FITS files?

(We don't write FITS files as such, we write HDS files (NDFs)
with FITS headers)

??

Maren

Steve Allen a écrit :
A question for those who have been around longer than I:

Not sure I qualify as a true old timer (my first contact with
FITS was circa 1985), but for what it's worth my reading of "once FITS
always FITS" is

Anything ever explicitly allowed can never be disallowed

plus

Anything not explicitly disallowed must always remain allowed
to within reason (i.e. closing contorted loopholes that pretty
much nobody ever used is fine, forbiding commonly used patterns
is not; borderline cases of course exist and then need
detailed discussion).

In RFC 4047 we have
no change may be made to FITS that invalidates existing files
but I'm not sure that's what everyone thinks.

To me that's really what the rule is about: we want data archives
to remain valid+readable, and forcing a format conversion on
archive maintainers is an absolute no-no.

Maren Purves wrote:

Anything ever required cannot be disallowed
and anything ever disallowed cannot be required

is about as far as I would go.

I would add:

Nothing that has not been required in the past can be required.

Otherwise, new readers will not be able to read old data at all.

-Doug Mink

Here are a few comments on several related recent FITS issues:

1. The EXTEND keyword: Since the STEREO mission has started producing
FITS files with EXTEND = F, it is too late to prohibit this usage in the
FITS Standard (because of the "once FITS, always FITS" rule). The
technical panel was not aware of this usage when it drafted the change
to the EXTEND keyword definition (which was modeled after the definition
of the BLOCKED keyword which can only have a T value, never F). The
definition of the EXTEND keyword should probably be revised to explain
what is meant by EXTEND = F, e.g., something perhaps like this:

"If the EXTEND keyword value is the logical constant F, then this
suggests that there are no extensions following the primary array.
This keyword is only advisory, however, so extensions may
follow the primary array even if EXTEND = F, but this is not
recommended."

2. Repeated keywords: Since the recent discussions have made it clear
that it is not a rare occurrence for existing FITS files to contain the
same keyword multiple times (the technical panel had hoped this was not
the case), it appears not feasible to simply declare that this practice
is prohibited in a conforming FITS file (again because of the "Once FITS
always FITS" rule). It seems that the most we can do at this point is
to strongly discourage this practice, because it is indeterminate which
of the keywords will read by software systems when retrieving the value.

3. On the "Once FITS, always FITS" rule: I'm not a true FITS 'old
timer' since I only seriously got into the FITS business in 1990, but I
believe the origin of this basic FITS rule can be traced back to the
following statement in the 1988 "Generalized Extensions" paper by
Grosbol, Harten, Greisen, and Wells:

"The most important rule for designing new extensions to FITS is that
existing FITS tapes must remain valid. We are not permitted to alter
the basic format in such a way as to make existing FITS tapes invalid or
unreadable by standard FITS tape reading programs".

When the FITS Standard was drafted in circa 1990, this rule was then
restated as "Any structure that is a valid FITS structure shall remain a
valid FITS structure at all future times." (see section 3.7 of the
proposed new Standard).

In practice, I believe this rule has generally been interpreted to mean
that if there are existing FITS files that implement a certain FITS
feature, then that feature must remain legal for all time in the FITS
Standard (although features can be deprecated, or strongly discouraged
from use). If, however, a *hypothetically* valid FITS feature has never
been used, then it is possible to revise the FITS rules to declare that
feature is not considered valid in a FITS file. (I listed a number of
these types of changes that have been made in a previous email).

Note also that this "Once FITS, Always FITS" rule does not provide
blanket protection to existing software packages from changes to FITS
that might break the software. I think we are in fact sensitive to any
changes that might disrupt existing software, but there is no absolute
ban on making such changes to FITS.

Bill Pence
--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)

1. The EXTEND keyword: Since the STEREO mission has started
producing
FITS files with EXTEND = F, it is too late to prohibit this usage
in the
FITS Standard (because of the "once FITS, always FITS" rule). The
technical panel was not aware of this usage when it drafted the change
to the EXTEND keyword definition (which was modeled after the
definition
of the BLOCKED keyword which can only have a T value, never F). The
definition of the EXTEND keyword should probably be revised to explain
what is meant by EXTEND = F, e.g., something perhaps like this:

"If the EXTEND keyword value is the logical constant F, then this
suggests that there are no extensions following the primary array.
This keyword is only advisory, however, so extensions may
follow the primary array even if EXTEND = F, but this is not
recommended."

Perhaps I am misunderstanding or misremembering something here, but I
think EXTEND=T does actually have a purpose, and to suggest that both
EXTEND=T and EXTEND=F are merely advisory is crazy. We should not add
the words "nor does the absence of this keyword necessarily imply
that the file does not contain extensions" to the FITS standard.

The Grosbøl et al. paper (1988 A&A) said:

"Note that the presence of EXTEND=T in a primary FITS header ...
indicates that the file _may_ have extensions records and that any
special records will conform to the rules below." The important part
of this statement is the last bit, "any special records will conform
to the rules below" - meaning that the special records follow the
rules laid down for standard extensions, so that in particular the
size of each extension and hence the starting point of the next
extension can be determined.

Note that in this paper "special records" are any 2880-byte records
that follow the last record of the primary HDU. It is (or was) legal
FITS to put anything you like in these records, so long as EXTEND=T
is not present in the primary header.

If EXTEND is not present in the primary header, or if EXTEND is
present with any value other than T, then the reader should not
attempt to interpret any special records as standard extensions.

My guess (it's a long time ago) is that there were FITS files out
there that had special records that did not conform to the rules for
standard extensions, so the EXTEND keyword was needed. Or perhaps
Grosbøl et al wanted to leave open the possibility of other uses for
the special records, which would be indicated by different header
keywords.

So if we change the standard to make EXTEND optional in all cases, we
are effectively prohibiting use of the "special records" for anything
except standard extensions. This sounds like an incompatible change
to me.

The proposed FITS 3.0 standard talks about "special records"
following the last extension, although it "deprecates" them. I don't
think this is or ever has been legal FITS usage: if you have one
standard extension (and EXTEND=T), all the special records must be
standard extensions. If EXTEND is not T, then you can have special
records that are not standard extensions. The only legal options a

Primary HDU with EXTEND=T, followed by zero or more standard
extensions (HDUs)
Primary HDU without EXTEND=T, followed by zero or more "special records"

If standard extensions are followed by additional special records,
there is no way to indicate where the last extension ends and any
"special records" begin, except to say that any records which don't
conform to the rules for standard extensions must be additional
special records. I don't think Grosbøl et al. intended that, and I
think their paper prohibits such additional special records.

- Tim Pearson

Tim Pearson wrote:

My guess (it's a long time ago) is that there were FITS files out
there that had special records that did not conform to the rules for
standard extensions, so the EXTEND keyword was needed. Or perhaps
Grosbøl et al wanted to leave open the possibility of other uses for
the special records, which would be indicated by different header
keywords.

binary tables?
(I remember converting JCMT data to NRAO style binary tables FITS
some time around 1990)

Aloha,
Maren