A Space & astronomy forum. SpaceBanter.com

Go Back   Home » SpaceBanter.com forum » Astronomy and Astrophysics » FITS
Site Map Home Authors List Search Today's Posts Mark Forums Read Web Partners

[fitsbits] Proposed Changes to the FITS Standard



 
 
Thread Tools Display Modes
  #41  
Old August 23rd 07, 03:12 AM posted to sci.astro.fits
Doug Tody
external usenet poster
 
Posts: 19
Default [fitsbits] once FITS always FITS

On Wed, 22 Aug 2007, William Pence wrote:

Here are a few comments on several related recent FITS issues:

1. The EXTEND keyword: Since the STEREO mission has started producing
FITS files with EXTEND = F, it is too late to prohibit this usage in the
FITS Standard (because of the "once FITS, always FITS" rule). The
technical panel was not aware of this usage when it drafted the change
to the EXTEND keyword definition (which was modeled after the definition
of the BLOCKED keyword which can only have a T value, never F). The
definition of the EXTEND keyword should probably be revised to explain
what is meant by EXTEND = F, e.g., something perhaps like this:

"If the EXTEND keyword value is the logical constant F, then this
suggests that there are no extensions following the primary array.
This keyword is only advisory, however, so extensions may
follow the primary array even if EXTEND = F, but this is not
recommended."


Ok; but logically, if EXTEND = T is defined, then EXTEND = F should be
as well, and what the STEREO mission did was correct. If EXTEND=F is
explict this should probably disallow any extensions (they should be
ignored or at least suspect if present). If EXTEND=T or the EXTEND
keyword is missing, there *may* be extensions, and the reader should
check for extensions.

2. Repeated keywords: Since the recent discussions have made it clear
that it is not a rare occurrence for existing FITS files to contain the
same keyword multiple times (the technical panel had hoped this was not
the case), it appears not feasible to simply declare that this practice
is prohibited in a conforming FITS file (again because of the "Once FITS
always FITS" rule). It seems that the most we can do at this point is
to strongly discourage this practice, because it is indeterminate which
of the keywords will read by software systems when retrieving the value.


Right; repeated keywords are inherently ambiguous and should be discouraged
(theoretically this need not be the case, and they could have meaning, but
this is *not* the case with FITS, and they should be discouraged).

- Doug
  #42  
Old August 23rd 07, 02:05 PM posted to sci.astro.fits
LC's NoSpam Newsreading account[_2_]
external usenet poster
 
Posts: 23
Default [fitsbits] once FITS always FITS

On Wed, 22 Aug 2007, William Pence wrote:

Here are a few comments on several related recent FITS issues:

1. The EXTEND keyword: Since the STEREO mission has started producing FITS


I posted on this in another thread.

2. Repeated keywords: Since the recent discussions have made it clear
that it is not a rare occurrence [...] It seems that the most we
can do at this point is to strongly discourage this practice,
because it is indeterminate which of the keywords will read by
software systems when retrieving the value.


Yes.
Perhaps we might also add some provisions by which duplication
of (all or a specific list of) reserved keywords is forbidden. For
mandatory keywords this is implicitly already specified by the
prescribed order (a double BITPIX, XTENSION or NAXIS1 or PCOUNT are
already illegal ... a double TFIELDS, TFORMn and alike should be !)

Lucio Chiappetti

--
----------------------------------------------------------------------
is a newsreading account used by more persons to
avoid unwanted spam. Any mail returning to this address will be rejected.
Users can disclose their e-mail address in the article if they wish so.
  #43  
Old August 23rd 07, 08:18 PM posted to sci.astro.fits
William Pence
external usenet poster
 
Posts: 66
Default [fitsbits] Abuse of EXTEND keyword

Tim Pearson wrote:
Perhaps I am misunderstanding or misremembering something here, but I
think EXTEND=T does actually have a purpose, and to suggest that both
EXTEND=T and EXTEND=F are merely advisory is crazy. We should not add
the words "nor does the absence of this keyword necessarily imply
that the file does not contain extensions" to the FITS standard.


Can you explain why this should not be done? Also, the EXTEND = T
keyword has always been advisory, so it seems consistent to declare that
EXTEND = F should also be advisory.

Whether or not a FITS file has conforming extensions following the
primary array is a simple observable fact; the presence or value of the
EXTEND keyword will not change that fact, so the EXTEND keyword seems to
serve little useful purpose, and even worse, it can create confusion if
EXTEND is not present, or has the value = F in a file that actually has
extensions. So the purpose of the proposed change to the EXTEND keyword
is to make it clear to software developers that they should not depend
on the existence, or value, of the EXTEND keyword when deciding whether
or not to read or write an extension in a FITS file.

The proposed FITS 3.0 standard talks about "special records"
following the last extension, although it "deprecates" them. I don't
think this is or ever has been legal FITS usage: if you have one
standard extension (and EXTEND=T), all the special records must be
standard extensions. If EXTEND is not T, then you can have special
records that are not standard extensions. The only legal options a

Primary HDU with EXTEND=T, followed by zero or more standard
extensions (HDUs)
Primary HDU without EXTEND=T, followed by zero or more "special records"

If standard extensions are followed by additional special records,
there is no way to indicate where the last extension ends and any
"special records" begin, except to say that any records which don't
conform to the rules for standard extensions must be additional
special records. I don't think Grosbøl et al. intended that, and I
think their paper prohibits such additional special records.


The FITS Standard clarified the meaning of "special records", such that
a FITS file may contain the following structures (ignoring random
groups, which are a special case): The primary array, followed by
optional conforming extensions, followed by optional special records.
Special records are defined as a sequence of 1 or more 2880 byte records
that do not obey the rules for the primary array or a conforming
extension (in particular, special records cannot begin with "XTENSION"
or "SIMPLE". It is now proposed to deprecate any further use of special
records, without the explicit approval of the IAU FITS Working Group, on
the grounds that a) conforming extensions should be sufficient for most
future needs, and b) allowing anyone to tack on arbitrary data records
to the end of their FITS files is not conducive to the interoperability
of data.

Bill Pence
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)


  #44  
Old August 23rd 07, 10:15 PM posted to sci.astro.fits
Randall Thompson
external usenet poster
 
Posts: 3
Default [fitsbits] Abuse of EXTEND keyword

I agree with Tim, defining EXTEND=F to mean the same thing as
EXTEND=T makes it worthless, and at that point maybe it should
just be deprecated. You are basically saying that EXTEND=F,
EXTEND=T, or leaving it out completely all mean the same thing;
namely, extensions may or may not exist.

By the way, we get many files delivered for archiving within MAST
that use both the EXTEND and the NEXTEND keyword. Although not a
reserved keyword, NEXTEND is commonly used to describe the
number of included extensions. I guess we are in the minority on this,
but when these keywords disagree with the file contents we either
correct them or ask the contributors to do it. I don't know if either
is really useful for a FITS reader, but they can be useful for getting
an idea of the file structure when just reading the primary header.

Randy

William Pence wrote:

Tim Pearson wrote:


Perhaps I am misunderstanding or misremembering something here, but I
think EXTEND=T does actually have a purpose, and to suggest that both
EXTEND=T and EXTEND=F are merely advisory is crazy. We should not add
the words "nor does the absence of this keyword necessarily imply
that the file does not contain extensions" to the FITS standard.



Can you explain why this should not be done? Also, the EXTEND = T
keyword has always been advisory, so it seems consistent to declare that
EXTEND = F should also be advisory.

Whether or not a FITS file has conforming extensions following the
primary array is a simple observable fact; the presence or value of the
EXTEND keyword will not change that fact, so the EXTEND keyword seems to
serve little useful purpose, and even worse, it can create confusion if
EXTEND is not present, or has the value = F in a file that actually has
extensions. So the purpose of the proposed change to the EXTEND keyword
is to make it clear to software developers that they should not depend
on the existence, or value, of the EXTEND keyword when deciding whether
or not to read or write an extension in a FITS file.




  #45  
Old August 24th 07, 01:41 AM posted to sci.astro.fits
Tim Pearson
external usenet poster
 
Posts: 3
Default [fitsbits] Abuse of EXTEND keyword


On Aug 23, 2007, at 12:18 PM, William Pence wrote:

Tim Pearson wrote:
Perhaps I am misunderstanding or misremembering something here, but I
think EXTEND=T does actually have a purpose, and to suggest that both
EXTEND=T and EXTEND=F are merely advisory is crazy. We should not add
the words "nor does the absence of this keyword necessarily imply
that the file does not contain extensions" to the FITS standard.


Can you explain why this should not be done? Also, the EXTEND = T
keyword has always been advisory, so it seems consistent to declare
that
EXTEND = F should also be advisory.


EXTEND=T when introduced was not advisory: it was an explicit
indication that all the "special records" (if any were present) after
the primary HDU contained only standard extensions. It would not have
been introduced if it was not necessary to distinguish such well-
formed files from those that contain arbitrary (unspecified) data in
the special records.

Sometime since the Grosbøl paper, it appears that two things crept
into the standard:

1. Special records that are not standard extensions can appear after
the last standard extension, even if EXTEND=T. I don't like this: it
is unnecessary and ugly and makes it harder to detect corrupted
files. Perhaps it is too late to undo it (although it is now
"deprecated").

2. Special records that are not standard extensions are not allowed
to start with 'XTENSION'. This is stated in section 3.5 of the
standard. It is this change that allows a reader to ignore the EXTEND
keyword and just look for standard extensions in the data stream.
This change would not have been necessary if change 1 had not been
made. I suppose that the chance that this change invalidated an
existing FITS file is negligible. When I wrote the above, I had not
noticed or had forgotten this statement in Section 3.5.

With these two changes, I agree that EXTEND is no longer needed. I
see no point in attaching "advisory" meaning to its presence,
absence, or value (what should a program do based on this advisory
information?); just say it conveys no information and should be ignored.

- Tim Pearson
  #46  
Old August 24th 07, 08:36 AM posted to sci.astro.fits
Peter Bunclark
external usenet poster
 
Posts: 6
Default [fitsbits] once FITS always FITS

LC's NoSpam Newsreading account wrote:

Yes.
Perhaps we might also add some provisions by which duplication
of (all or a specific list of) reserved keywords is forbidden. For
mandatory keywords this is implicitly already specified by the
prescribed order (a double BITPIX, XTENSION or NAXIS1 or PCOUNT are
already illegal ... a double TFIELDS, TFORMn and alike should be !)

Lucio Chiappetti



Peter Bunclark wrote (20 Aug):
Would it be too complex to make the non-duplication of reserved words
mandatory and of others just strongly recommended?


Looks like this discussion has hit that well-known threshold where
people who need to keep up just can't manage the time to do so;
certainly applies to me!

Pete.
  #47  
Old August 24th 07, 10:43 AM posted to sci.astro.fits
LC's NoSpam Newsreading account[_2_]
external usenet poster
 
Posts: 23
Default [fitsbits] Abuse of EXTEND keyword

On Thu, 23 Aug 2007, Randall Thompson wrote:

You are basically saying that EXTEND=F, EXTEND=T, or leaving it out
completely all mean the same thing; namely, extensions may or may not
exist.


Hence the reason for deprecation.

By the way, we get many files delivered for archiving within MAST
that use both the EXTEND and the NEXTEND keyword. Although not a
reserved keyword, NEXTEND is commonly used to describe the
number of included extensions. I guess we are in the minority on this,


Maybe you could register a "convention" about NEXTEND.

Providing the number of extensions in the primary header seems a
very reasonable thing for me, if the file is intended to be static.

What will be a file with no extension in your convention ?

- no keyword at all
- EXTEND=F and no NEXTEND kwd
- EXTEND=F and NEXTEND=0
- EXTEND=T and NEXTEND=0


--
----------------------------------------------------------------------
is a newsreading account used by more persons to
avoid unwanted spam. Any mail returning to this address will be rejected.
Users can disclose their e-mail address in the article if they wish so.
  #48  
Old August 24th 07, 07:03 PM posted to sci.astro.fits
Phil Hodge
external usenet poster
 
Posts: 5
Default [fitsbits] Abuse of EXTEND keyword

What will be a file with no extension in your convention ?

- no keyword at all
- EXTEND=F and no NEXTEND kwd
- EXTEND=F and NEXTEND=0
- EXTEND=T and NEXTEND=0



For HST data we write NEXTEND in files that are supposed to have
extensions, and in that case it's useful in that it tells a calibration
program how many exposures (for example) to expect. A utility program
might not even look for NEXTEND.

A program that expects data in extensions would typically use a default
value if the NEXTEND keyword were absent. If NEXTEND = 0, there's no
data to process. EXTEND is almost always ignored.

Phil

  #49  
Old August 24th 07, 07:39 PM posted to sci.astro.fits
William Pence
external usenet poster
 
Posts: 66
Default [fitsbits] Abuse of EXTEND keyword

This mention of the NEXTEND keyword caused me to rethink what is really
meant by EXTEND = F: EXTEND = T is defined to mean that the FITS file
*is permitted* to have extensions following the primary array (the
actually wording in the Standard is "may contain extensions" ); it does
not mean that the file actually has any extensions.

If EXTEND = F, then this logically means the opposite, i.e., that the
FITS file is *not permitted* to have any extensions. But this is not
the meaning that the STEREO mission folks really intended by setting
EXTEND = F since they surely don't care if users were to add extensions
to the files at a later date. In fact, I can't think of any case where
it would really be appropriate to set EXTEND = F. Instead, I think the
desired meaning can be expressed by this pair of keywords:

EXTEND = T
NEXTEND = 0

which means that the file currently has no extensions following the
primary array, but it is permitted to add extensions if desired. I'm
not really recommending this, however, because as previously mentioned
by Preben Grosbol, it could become a burden to keep the NEXTEND keyword
up to date as extensions get added or removed from the file. Instead,
if software needs to know if there are extensions in the file then it
can just do an inventory of the file in real time. This is not a time
consuming operation since it does not require reading the whole file;
the software can directly jump to the header of each subsequent
extension, at least on random access storage devices.

Bill Pence

Randall Thompson wrote:
I agree with Tim, defining EXTEND=F to mean the same thing as
EXTEND=T makes it worthless, and at that point maybe it should
just be deprecated. You are basically saying that EXTEND=F,
EXTEND=T, or leaving it out completely all mean the same thing;
namely, extensions may or may not exist.

By the way, we get many files delivered for archiving within MAST
that use both the EXTEND and the NEXTEND keyword. Although not a
reserved keyword, NEXTEND is commonly used to describe the
number of included extensions. I guess we are in the minority on this,
but when these keywords disagree with the file contents we either
correct them or ask the contributors to do it. I don't know if either
is really useful for a FITS reader, but they can be useful for getting
an idea of the file structure when just reading the primary header.

--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)


  #50  
Old August 24th 07, 07:48 PM posted to sci.astro.fits
Randall Thompson
external usenet poster
 
Posts: 3
Default [fitsbits] Abuse of EXTEND keyword

LC's NoSpam Newsreading account wrote:

On Thu, 23 Aug 2007, Randall Thompson wrote:



You are basically saying that EXTEND=F, EXTEND=T, or leaving it out
completely all mean the same thing; namely, extensions may or may not
exist.



Hence the reason for deprecation.


As I understand the proposed standard changes and previous discussion,
EXTEND
would not be deprecated. It would become an optional reserved keyword
with the definition of
EXTEND=T, and possibly EXTEND=F, both meaning extensions may or may not
exist. I don't think
it will serve any purpose if these changes are adopted, so I would
think deprecation is preferable.
The advantage of deprecation would be that its use would not be
recommended, and the definition
would not need to be clarified.

Note also that demoting it to a optional reserved keyword means it does
not have to appear
in any particular location in the FITS header. This will also make it
less useful in terms of
human readability. Last month we received FITS files with over 14,000
keywords in the
primary header!




By the way, we get many files delivered for archiving within MAST
that use both the EXTEND and the NEXTEND keyword. Although not a
reserved keyword, NEXTEND is commonly used to describe the
number of included extensions. I guess we are in the minority on this,



Maybe you could register a "convention" about NEXTEND.

Providing the number of extensions in the primary header seems a
very reasonable thing for me, if the file is intended to be static.

What will be a file with no extension in your convention ?

- no keyword at all
- EXTEND=F and no NEXTEND kwd
- EXTEND=F and NEXTEND=0
- EXTEND=T and NEXTEND=0



I wasn't necessarily recommending the use of NEXTEND, just stating that
its being used by others.
If I were to create FITS files after the proposed changes are adopted,
with or without extensions,
I would probably not include either keyword.

 




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[fitsbits] Proposed Changes to the FITS Standard Mark Calabretta FITS 0 August 2nd 07 09:39 AM
[fitsbits] Proposed Changes to the FITS Standard Steve Allen FITS 0 August 1st 07 06:08 PM
[fitsbits] Proposed Changes to the FITS Standard Thierry Forveille FITS 0 August 1st 07 04:51 PM
[fitsbits] Proposed Changes to the FITS Standard William Pence FITS 0 July 27th 07 07:38 PM
[fitsbits] Proposed Changes to the FITS Standard Rob Seaman FITS 0 July 24th 07 07:21 PM


All times are GMT +1. The time now is 03:18 AM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 SpaceBanter.com.
The comments are property of their posters.