|
|
Thread Tools | Display Modes |
#41
|
|||
|
|||
[fitsbits] once FITS always FITS
On Wed, 22 Aug 2007, William Pence wrote:
Here are a few comments on several related recent FITS issues: 1. The EXTEND keyword: Since the STEREO mission has started producing FITS files with EXTEND = F, it is too late to prohibit this usage in the FITS Standard (because of the "once FITS, always FITS" rule). The technical panel was not aware of this usage when it drafted the change to the EXTEND keyword definition (which was modeled after the definition of the BLOCKED keyword which can only have a T value, never F). The definition of the EXTEND keyword should probably be revised to explain what is meant by EXTEND = F, e.g., something perhaps like this: "If the EXTEND keyword value is the logical constant F, then this suggests that there are no extensions following the primary array. This keyword is only advisory, however, so extensions may follow the primary array even if EXTEND = F, but this is not recommended." Ok; but logically, if EXTEND = T is defined, then EXTEND = F should be as well, and what the STEREO mission did was correct. If EXTEND=F is explict this should probably disallow any extensions (they should be ignored or at least suspect if present). If EXTEND=T or the EXTEND keyword is missing, there *may* be extensions, and the reader should check for extensions. 2. Repeated keywords: Since the recent discussions have made it clear that it is not a rare occurrence for existing FITS files to contain the same keyword multiple times (the technical panel had hoped this was not the case), it appears not feasible to simply declare that this practice is prohibited in a conforming FITS file (again because of the "Once FITS always FITS" rule). It seems that the most we can do at this point is to strongly discourage this practice, because it is indeterminate which of the keywords will read by software systems when retrieving the value. Right; repeated keywords are inherently ambiguous and should be discouraged (theoretically this need not be the case, and they could have meaning, but this is *not* the case with FITS, and they should be discouraged). - Doug |
#42
|
|||
|
|||
[fitsbits] once FITS always FITS
On Wed, 22 Aug 2007, William Pence wrote:
Here are a few comments on several related recent FITS issues: 1. The EXTEND keyword: Since the STEREO mission has started producing FITS I posted on this in another thread. 2. Repeated keywords: Since the recent discussions have made it clear that it is not a rare occurrence [...] It seems that the most we can do at this point is to strongly discourage this practice, because it is indeterminate which of the keywords will read by software systems when retrieving the value. Yes. Perhaps we might also add some provisions by which duplication of (all or a specific list of) reserved keywords is forbidden. For mandatory keywords this is implicitly already specified by the prescribed order (a double BITPIX, XTENSION or NAXIS1 or PCOUNT are already illegal ... a double TFIELDS, TFORMn and alike should be !) Lucio Chiappetti -- ---------------------------------------------------------------------- is a newsreading account used by more persons to avoid unwanted spam. Any mail returning to this address will be rejected. Users can disclose their e-mail address in the article if they wish so. |
#43
|
|||
|
|||
[fitsbits] Abuse of EXTEND keyword
Tim Pearson wrote:
Perhaps I am misunderstanding or misremembering something here, but I think EXTEND=T does actually have a purpose, and to suggest that both EXTEND=T and EXTEND=F are merely advisory is crazy. We should not add the words "nor does the absence of this keyword necessarily imply that the file does not contain extensions" to the FITS standard. Can you explain why this should not be done? Also, the EXTEND = T keyword has always been advisory, so it seems consistent to declare that EXTEND = F should also be advisory. Whether or not a FITS file has conforming extensions following the primary array is a simple observable fact; the presence or value of the EXTEND keyword will not change that fact, so the EXTEND keyword seems to serve little useful purpose, and even worse, it can create confusion if EXTEND is not present, or has the value = F in a file that actually has extensions. So the purpose of the proposed change to the EXTEND keyword is to make it clear to software developers that they should not depend on the existence, or value, of the EXTEND keyword when deciding whether or not to read or write an extension in a FITS file. The proposed FITS 3.0 standard talks about "special records" following the last extension, although it "deprecates" them. I don't think this is or ever has been legal FITS usage: if you have one standard extension (and EXTEND=T), all the special records must be standard extensions. If EXTEND is not T, then you can have special records that are not standard extensions. The only legal options a Primary HDU with EXTEND=T, followed by zero or more standard extensions (HDUs) Primary HDU without EXTEND=T, followed by zero or more "special records" If standard extensions are followed by additional special records, there is no way to indicate where the last extension ends and any "special records" begin, except to say that any records which don't conform to the rules for standard extensions must be additional special records. I don't think Grosbøl et al. intended that, and I think their paper prohibits such additional special records. The FITS Standard clarified the meaning of "special records", such that a FITS file may contain the following structures (ignoring random groups, which are a special case): The primary array, followed by optional conforming extensions, followed by optional special records. Special records are defined as a sequence of 1 or more 2880 byte records that do not obey the rules for the primary array or a conforming extension (in particular, special records cannot begin with "XTENSION" or "SIMPLE". It is now proposed to deprecate any further use of special records, without the explicit approval of the IAU FITS Working Group, on the grounds that a) conforming extensions should be sufficient for most future needs, and b) allowing anyone to tack on arbitrary data records to the end of their FITS files is not conducive to the interoperability of data. Bill Pence __________________________________________________ __________________ Dr. William Pence NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice) Greenbelt MD 20771 +1-301-286-1684 (fax) |
#44
|
|||
|
|||
[fitsbits] Abuse of EXTEND keyword
I agree with Tim, defining EXTEND=F to mean the same thing as
EXTEND=T makes it worthless, and at that point maybe it should just be deprecated. You are basically saying that EXTEND=F, EXTEND=T, or leaving it out completely all mean the same thing; namely, extensions may or may not exist. By the way, we get many files delivered for archiving within MAST that use both the EXTEND and the NEXTEND keyword. Although not a reserved keyword, NEXTEND is commonly used to describe the number of included extensions. I guess we are in the minority on this, but when these keywords disagree with the file contents we either correct them or ask the contributors to do it. I don't know if either is really useful for a FITS reader, but they can be useful for getting an idea of the file structure when just reading the primary header. Randy William Pence wrote: Tim Pearson wrote: Perhaps I am misunderstanding or misremembering something here, but I think EXTEND=T does actually have a purpose, and to suggest that both EXTEND=T and EXTEND=F are merely advisory is crazy. We should not add the words "nor does the absence of this keyword necessarily imply that the file does not contain extensions" to the FITS standard. Can you explain why this should not be done? Also, the EXTEND = T keyword has always been advisory, so it seems consistent to declare that EXTEND = F should also be advisory. Whether or not a FITS file has conforming extensions following the primary array is a simple observable fact; the presence or value of the EXTEND keyword will not change that fact, so the EXTEND keyword seems to serve little useful purpose, and even worse, it can create confusion if EXTEND is not present, or has the value = F in a file that actually has extensions. So the purpose of the proposed change to the EXTEND keyword is to make it clear to software developers that they should not depend on the existence, or value, of the EXTEND keyword when deciding whether or not to read or write an extension in a FITS file. |
#45
|
|||
|
|||
[fitsbits] Abuse of EXTEND keyword
On Aug 23, 2007, at 12:18 PM, William Pence wrote: Tim Pearson wrote: Perhaps I am misunderstanding or misremembering something here, but I think EXTEND=T does actually have a purpose, and to suggest that both EXTEND=T and EXTEND=F are merely advisory is crazy. We should not add the words "nor does the absence of this keyword necessarily imply that the file does not contain extensions" to the FITS standard. Can you explain why this should not be done? Also, the EXTEND = T keyword has always been advisory, so it seems consistent to declare that EXTEND = F should also be advisory. EXTEND=T when introduced was not advisory: it was an explicit indication that all the "special records" (if any were present) after the primary HDU contained only standard extensions. It would not have been introduced if it was not necessary to distinguish such well- formed files from those that contain arbitrary (unspecified) data in the special records. Sometime since the Grosbøl paper, it appears that two things crept into the standard: 1. Special records that are not standard extensions can appear after the last standard extension, even if EXTEND=T. I don't like this: it is unnecessary and ugly and makes it harder to detect corrupted files. Perhaps it is too late to undo it (although it is now "deprecated"). 2. Special records that are not standard extensions are not allowed to start with 'XTENSION'. This is stated in section 3.5 of the standard. It is this change that allows a reader to ignore the EXTEND keyword and just look for standard extensions in the data stream. This change would not have been necessary if change 1 had not been made. I suppose that the chance that this change invalidated an existing FITS file is negligible. When I wrote the above, I had not noticed or had forgotten this statement in Section 3.5. With these two changes, I agree that EXTEND is no longer needed. I see no point in attaching "advisory" meaning to its presence, absence, or value (what should a program do based on this advisory information?); just say it conveys no information and should be ignored. - Tim Pearson |
#46
|
|||
|
|||
[fitsbits] once FITS always FITS
LC's NoSpam Newsreading account wrote:
Yes. Perhaps we might also add some provisions by which duplication of (all or a specific list of) reserved keywords is forbidden. For mandatory keywords this is implicitly already specified by the prescribed order (a double BITPIX, XTENSION or NAXIS1 or PCOUNT are already illegal ... a double TFIELDS, TFORMn and alike should be !) Lucio Chiappetti Peter Bunclark wrote (20 Aug): Would it be too complex to make the non-duplication of reserved words mandatory and of others just strongly recommended? Looks like this discussion has hit that well-known threshold where people who need to keep up just can't manage the time to do so; certainly applies to me! Pete. |
#47
|
|||
|
|||
[fitsbits] Abuse of EXTEND keyword
On Thu, 23 Aug 2007, Randall Thompson wrote:
You are basically saying that EXTEND=F, EXTEND=T, or leaving it out completely all mean the same thing; namely, extensions may or may not exist. Hence the reason for deprecation. By the way, we get many files delivered for archiving within MAST that use both the EXTEND and the NEXTEND keyword. Although not a reserved keyword, NEXTEND is commonly used to describe the number of included extensions. I guess we are in the minority on this, Maybe you could register a "convention" about NEXTEND. Providing the number of extensions in the primary header seems a very reasonable thing for me, if the file is intended to be static. What will be a file with no extension in your convention ? - no keyword at all - EXTEND=F and no NEXTEND kwd - EXTEND=F and NEXTEND=0 - EXTEND=T and NEXTEND=0 -- ---------------------------------------------------------------------- is a newsreading account used by more persons to avoid unwanted spam. Any mail returning to this address will be rejected. Users can disclose their e-mail address in the article if they wish so. |
#48
|
|||
|
|||
[fitsbits] Abuse of EXTEND keyword
What will be a file with no extension in your convention ?
- no keyword at all - EXTEND=F and no NEXTEND kwd - EXTEND=F and NEXTEND=0 - EXTEND=T and NEXTEND=0 For HST data we write NEXTEND in files that are supposed to have extensions, and in that case it's useful in that it tells a calibration program how many exposures (for example) to expect. A utility program might not even look for NEXTEND. A program that expects data in extensions would typically use a default value if the NEXTEND keyword were absent. If NEXTEND = 0, there's no data to process. EXTEND is almost always ignored. Phil |
#49
|
|||
|
|||
[fitsbits] Abuse of EXTEND keyword
This mention of the NEXTEND keyword caused me to rethink what is really
meant by EXTEND = F: EXTEND = T is defined to mean that the FITS file *is permitted* to have extensions following the primary array (the actually wording in the Standard is "may contain extensions" ); it does not mean that the file actually has any extensions. If EXTEND = F, then this logically means the opposite, i.e., that the FITS file is *not permitted* to have any extensions. But this is not the meaning that the STEREO mission folks really intended by setting EXTEND = F since they surely don't care if users were to add extensions to the files at a later date. In fact, I can't think of any case where it would really be appropriate to set EXTEND = F. Instead, I think the desired meaning can be expressed by this pair of keywords: EXTEND = T NEXTEND = 0 which means that the file currently has no extensions following the primary array, but it is permitted to add extensions if desired. I'm not really recommending this, however, because as previously mentioned by Preben Grosbol, it could become a burden to keep the NEXTEND keyword up to date as extensions get added or removed from the file. Instead, if software needs to know if there are extensions in the file then it can just do an inventory of the file in real time. This is not a time consuming operation since it does not require reading the whole file; the software can directly jump to the header of each subsequent extension, at least on random access storage devices. Bill Pence Randall Thompson wrote: I agree with Tim, defining EXTEND=F to mean the same thing as EXTEND=T makes it worthless, and at that point maybe it should just be deprecated. You are basically saying that EXTEND=F, EXTEND=T, or leaving it out completely all mean the same thing; namely, extensions may or may not exist. By the way, we get many files delivered for archiving within MAST that use both the EXTEND and the NEXTEND keyword. Although not a reserved keyword, NEXTEND is commonly used to describe the number of included extensions. I guess we are in the minority on this, but when these keywords disagree with the file contents we either correct them or ask the contributors to do it. I don't know if either is really useful for a FITS reader, but they can be useful for getting an idea of the file structure when just reading the primary header. -- __________________________________________________ __________________ Dr. William Pence NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice) Greenbelt MD 20771 +1-301-286-1684 (fax) |
#50
|
|||
|
|||
[fitsbits] Abuse of EXTEND keyword
LC's NoSpam Newsreading account wrote:
On Thu, 23 Aug 2007, Randall Thompson wrote: You are basically saying that EXTEND=F, EXTEND=T, or leaving it out completely all mean the same thing; namely, extensions may or may not exist. Hence the reason for deprecation. As I understand the proposed standard changes and previous discussion, EXTEND would not be deprecated. It would become an optional reserved keyword with the definition of EXTEND=T, and possibly EXTEND=F, both meaning extensions may or may not exist. I don't think it will serve any purpose if these changes are adopted, so I would think deprecation is preferable. The advantage of deprecation would be that its use would not be recommended, and the definition would not need to be clarified. Note also that demoting it to a optional reserved keyword means it does not have to appear in any particular location in the FITS header. This will also make it less useful in terms of human readability. Last month we received FITS files with over 14,000 keywords in the primary header! By the way, we get many files delivered for archiving within MAST that use both the EXTEND and the NEXTEND keyword. Although not a reserved keyword, NEXTEND is commonly used to describe the number of included extensions. I guess we are in the minority on this, Maybe you could register a "convention" about NEXTEND. Providing the number of extensions in the primary header seems a very reasonable thing for me, if the file is intended to be static. What will be a file with no extension in your convention ? - no keyword at all - EXTEND=F and no NEXTEND kwd - EXTEND=F and NEXTEND=0 - EXTEND=T and NEXTEND=0 I wasn't necessarily recommending the use of NEXTEND, just stating that its being used by others. If I were to create FITS files after the proposed changes are adopted, with or without extensions, I would probably not include either keyword. |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
[fitsbits] Proposed Changes to the FITS Standard | Mark Calabretta | FITS | 0 | August 2nd 07 09:39 AM |
[fitsbits] Proposed Changes to the FITS Standard | Steve Allen | FITS | 0 | August 1st 07 06:08 PM |
[fitsbits] Proposed Changes to the FITS Standard | Thierry Forveille | FITS | 0 | August 1st 07 04:51 PM |
[fitsbits] Proposed Changes to the FITS Standard | William Pence | FITS | 0 | July 27th 07 07:38 PM |
[fitsbits] Proposed Changes to the FITS Standard | Rob Seaman | FITS | 0 | July 24th 07 07:21 PM |