A Space & astronomy forum. SpaceBanter.com

Go Back   Home » SpaceBanter.com forum » Astronomy and Astrophysics » FITS
Site Map Home Authors List Search Today's Posts Mark Forums Read Web Partners

[fitsbits] Proposed Changes to the FITS Standard



 
 
Thread Tools Display Modes
  #1  
Old August 10th 07, 01:20 PM posted to sci.astro.fits
Preben Grosbol
external usenet poster
 
Posts: 11
Default [fitsbits] Proposed Changes to the FITS Standard

On Monday 23 July 2007 21:03, William Pence wrote:
Please post any comments, suggestions, or concerns regarding these
proposals (or any other suggestions for improving the Standard document)
here on the FITSBITS email list, or on the associated sci.astro.fits
newsgroup. * Comments may also be submitted to any member of the FITS
technical panel that produced these recommendations (listed below).


After some delays I finally had a chance to go through the changes to the
FITS standard as proposed by the technical panel.

First of all my congratulations to the panel for its excellent work which has
produced much more consistent and readable standards document. During
my reading I did not come across any major issues but there are several
points which are important enough to raise. Taking first section 1 of the
summary:

4. Section 3.5 Deprecate 'special records'

I think it's okay to deprecate 'special records' as the general
extensions should be able to handle our future needs. In an
age with computer virus, it may be safer to have some handle
on what can be in a FITS file.

5. Section 3.7 Restrictions on changes

The argument is understandable but the implications very wide.
It would allow dramatic changes such as redefinition of keywords.
Something like this is needed but we may need to consider the
best wording.

7. Section 4.1.2.3 Repeat of keywords

It may be too strong to forbid such repeats of keywords. It should
certainly be deprecated but it may place a significant load on many
applications which just want to add some keywords to a header.
They would be required to actually check if they would duplicate
or not. A deprecation and definition that the last value takes
precedence may be more appropriate.

11. Section 4.4.1.2 PCOUNT and GCOUNT

In principle okay but one should check if this would require
existing software to be changed. There may still be legacy
tasks which write e.g. Random groups format. The new
section 3.7 would not work in that case.

15. Section 4.4.3.1 Reference to specific extensions

As there are many recommendations in the document, it would
be good to retain a deprecation of explicit reference to other
extensions. The point is broken links if HDU's are moved.
This opens the old issue on how to create a unique reference
to any HDU. It would be an advantage for some applications,
such as in interferometry, where cross-references between binary
table sometimes are used.

17. Sections 7.2.2 and 7.3.2 TTYPEn

The reason for avoiding '-' was the mapping of the TTYPEn names
into variables is some languages. It is irritating to have '-' in
keywords for the same reason and most applications would just
substitute them with '_'. I would either retain the old wording or if
'-' should be allow at the same point deprecate the usage of it.

20. Section 7.2.5 Deprecate implicit decimal point

We should check it with the data centers. The reason for including
such cases was to accommodate direct encoding of legacy tables
which could have used such formats to save space (at the time
when a punch cards was real and only had 80 columns).

Going through the actual text I noticed the following minor points:

- It would be better to name Appendix A 'Formal Syntax of Keyword Records'
to be in line with section 4.1

- The font of COMMENT in section 4.1.2.2 seems wrong

- It would look nicer to start the table with BITPIX 8 in table 4.9 and 7.7

- Since the description of WCS keywords like CDELTn was moved to section
8, it would be good to have a reference to that section in section 4.

- It may be good to repeat a reference to fortran in the description of the
TDISPn keyword in 7.2

As said before - my compliments to the panel
for very good work,
Preben

  #2  
Old August 13th 07, 02:40 PM posted to sci.astro.fits
LC's NoSpam Newsreading account[_2_]
external usenet poster
 
Posts: 23
Default [fitsbits] Proposed Changes to the FITS Standard

On Fri, 10 Aug 2007, Preben Grosbol wrote:

11. Section 4.4.1.2 PCOUNT and GCOUNT

In principle okay but one should check if this would require
existing software to be changed. There may still be legacy
tasks which write e.g. Random groups format. The new
section 3.7 would not work in that case.


I am not sure what you mean here ? The fact that PCOUNT and GCOUNT are
inthe 5th and 6th place in table 4.8 ? But table 6.1 shows (correctly)
them in arbitrary position for random groups !

And random groups are NOT a "generalized conforming extension" but
something which pre-date them (maybe this could be stated more clearly).

Lucio Chiappetti

--
----------------------------------------------------------------------
is a newsreading account used by more persons to
avoid unwanted spam. Any mail returning to this address will be rejected.
Users can disclose their e-mail address in the article if they wish so.
  #3  
Old August 16th 07, 09:32 AM posted to sci.astro.fits
Preben Grosbol
external usenet poster
 
Posts: 11
Default [fitsbits] Proposed Changes to the FITS Standard

On Monday 13 August 2007 15:40, LC's NoSpam Newsreading account wrote:
I am not sure what you mean here ? The fact that PCOUNT and GCOUNT are
inthe 5th and 6th place in table 4.8 ? But table 6.1 shows (correctly)
them in arbitrary position for random groups !


It may be a matter of exact wording but the random groups header is
the prime header. A FITS reader which does not know about random
groups may flag such a header as an error if PCOUNT does not
follow directly after the last NAXISn keyword.

In principle the new Sec. 3.7 introduces a FITS version that is
in FITS v3+ certain things are not valid any more. We previously
were very reluctant to introduce the concept of versioning but maybe
it cannot be avoided. It is a rather principle matter which should
be discussed in some detail (maybe at the ADASS BoF).

Preben
  #4  
Old August 16th 07, 03:18 PM posted to sci.astro.fits
Eric Greisen
external usenet poster
 
Posts: 8
Default [fitsbits] Proposed Changes to the FITS Standard

Random groups is by no means a dead format. Following the lead of the
FITS community I added to AIPS the ability to write interferometer
visibility data using binary table format instead. I believe that
none of the other major interferometry packages (casa, difmap, miriad,
etc) can read data written this way. They all read data written in
the random groups format and future plans seem to be more of the
same.

If random groups are deprecated to the point of being illegal, then
the portion of astronomy which created FITS in the first place will be
left out of the FITS community.

Eric Greisen
  #5  
Old August 16th 07, 03:32 PM posted to sci.astro.fits
Peter Teuben
external usenet poster
 
Posts: 3
Default [fitsbits] Proposed Changes to the FITS Standard

On [Thu Aug 16 08:18], Eric Greisen wrote:
Random groups is by no means a dead format. Following the lead of the
FITS community I added to AIPS the ability to write interferometer
visibility data using binary table format instead. I believe that
none of the other major interferometry packages (casa, difmap, miriad,
etc) can read data written this way. They all read data written in
the random groups format and future plans seem to be more of the
same.


gosh, Miriad sure seems so old now, but we totally rely on the random
groups format for input/output to other packages. So, random groups may
be old, but it works and isn't broken.

- peter
  #6  
Old August 16th 07, 04:17 PM posted to sci.astro.fits
William Pence
external usenet poster
 
Posts: 66
Default [fitsbits] Proposed Changes to the FITS Standard

The new draft of the FITS Standard (available from
http://fits.gsfc.nasa.gov/) does not propose any changes at all to the
definition or use of random groups. One of the proposed changes,
however, is to require that the PCOUNT and GCOUNT keywords must
immediately follow the last NAXISn keyword in all conforming extensions,
but this does not apply to random groups. The Standard tries to make a
clear distinction between the random groups structure and conforming
extensions, so hopefully there should be no confusion between the 2.

Bill Pence

Peter Teuben wrote:
On [Thu Aug 16 08:18], Eric Greisen wrote:
Random groups is by no means a dead format. Following the lead of the
FITS community I added to AIPS the ability to write interferometer
visibility data using binary table format instead. I believe that
none of the other major interferometry packages (casa, difmap, miriad,
etc) can read data written this way. They all read data written in
the random groups format and future plans seem to be more of the
same.


gosh, Miriad sure seems so old now, but we totally rely on the random
groups format for input/output to other packages. So, random groups may
be old, but it works and isn't broken.

- peter

--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)


  #7  
Old August 16th 07, 05:33 PM posted to sci.astro.fits
Rob Seaman
external usenet poster
 
Posts: 49
Default [fitsbits] Proposed Changes to the FITS Standard

William Pence wrote:

One of the proposed changes, however, is to require that the
PCOUNT and GCOUNT keywords must immediately follow the last NAXISn
keyword in all conforming extensions,


It seems to me that if this or any future revision to the standard is
going to attempt to require newly defined usage that this MUST be
paired with a version mechanism. Otherwise "once FITS always FITS"
is violated big time. Requiring a feature (a demand) is not the same
as deprecating its converse (a suggestion). Rather, a feature that
isn't already required can only in the future be required against
some specially labeled subset of FITS files.

I agree with Preben that there are also problems with introducing
versioning, but otherwise conformance is unenforceable. The
suggested new wording of 3.7:

‘Existing FITS files that conformed to the latest
version of the standard at the time the files were
created are expressly exempt from any new
requirements imposed by subsequent versions
of the standard’.

can't serve as a substitute for an explicit version since A) not all
files have DATE keywords, so how would you know, and B) it will take
several years - perhaps never - for all prior software to adopt the
new standards. Rather, projects wanting to assert compliance would
include a recognizable FITSVER (for example) keyword. Files missing
this tag would by default only claim compliance with the original
standard.

In a real sense, the standard as revised is really a brand new
standard, subclassed out of FITS.

- Rob


  #8  
Old August 17th 07, 03:38 PM posted to sci.astro.fits
LC's NoSpam Newsreading account[_2_]
external usenet poster
 
Posts: 23
Default [fitsbits] Proposed Changes to the FITS Standard

On Thu, 16 Aug 2007, Preben Grosbol wrote:

It may be a matter of exact wording but the random groups header is
the prime header. A FITS reader which does not know about random
groups may flag such a header as an error if PCOUNT does not
follow directly after the last NAXISn keyword.


Since random groups were in the earliest FITS, I dare say that a generic
or non-radio FITS reader may decide not to support them, but cannot
afford to ignore them. They are clearly recognizable as flagged by
GROUPS=T, so the reader or verifier *must* recognize them (and ONLY then
*may* decide to ignore them).


On Thu, 16 Aug 2007, Eric Greisen wrote:

Random groups is by no means a dead format. [...]
If random groups are deprecated to the point of being illegal


Nobody is really proposing that. Actually the text of chapter 6 on
random group (and the introduction about deprecation) is UNCHANGED with
respect to earlier versions of the standard.

I am not sure whether the actual idea of "deprecation" arose in an
earlier version of chapter 6 or elsewhere. Anyhow also the general
definition of "deprecated" in chapter 2 did not change much and was NOT
made more restrictive.

It is (as it was originally) just a recommendation ("should") for NEW
application, and the text added in 3.0 is a guarantee clause for OLD
application (for them the deprecated structure SHALL remain valid).

On Thu, 16 Aug 2007, Rob Seaman wrote:

I agree with Preben that there are also problems with introducing
versioning, but otherwise conformance is unenforceable. The suggested
new wording of 3.7:


It might be that the text of 3.7 could be improved but what is the
actual impact of all this ? Does it apply to specific pieces of
software ? to existing one ? to new one ? Or just to (human) programmers
instead of software ?

The main point is that it does NOT force OLDER files to need to be
rewritten to conform to new standard "additions".

Just as e.g. the Y2K change to the DATE format did not force anybody to
rewrite old file with the new ISO timestamp.

So a generic reader might have to support older and newer variants
(which gives a burden on us to clearly document how and when they
changed, and from what into what). Or if it supports only the new
variant it shall warn in case of non-compliance "maybe this is an older
file" (or check if possible if this is the case), and be somewhat
lenient.

However it is always possible to add a COMMENT which claims conformance
with the latest (e.g. 3.0) standard.


Lucio Chiappetti

--
----------------------------------------------------------------------
is a newsreading account used by more persons to
avoid unwanted spam. Any mail returning to this address will be rejected.
Users can disclose their e-mail address in the article if they wish so.
  #9  
Old August 17th 07, 06:18 PM posted to sci.astro.fits
William Pence
external usenet poster
 
Posts: 66
Default [fitsbits] Proposed Changes to the FITS Standard

Rob Seaman wrote:
William Pence wrote:

One of the proposed changes, however, is to require that the
PCOUNT and GCOUNT keywords must immediately follow the last NAXISn
keyword in all conforming extensions,


It seems to me that if this or any future revision to the standard is
going to attempt to require newly defined usage that this MUST be
paired with a version mechanism. Otherwise "once FITS always FITS"
is violated big time. Requiring a feature (a demand) is not the same
as deprecating its converse (a suggestion). Rather, a feature that
isn't already required can only in the future be required against
some specially labeled subset of FITS files.


The "once FITS always FITS" philosophy captures the spirit of FITS, but
in practice each new version of the FITS Standard has imposed new
requirements that in principle could invalidate existing FITS files.
For example, version 2.0 of the FITS Standard introduced a new
requirement that the value and comment fields in a keyword MUST be
separated by a slash character. I think the FITS community in general
has taken a somewhat pragmatic view that it is OK to add a new
requirement to FITS that *might* invalidate older FITS files, as long as
the benefit of the new requirement is perceived to out weigh the
possible negative affects.

Of course, if the FITS community thinks a new requirement would cause
too much dislocation to existing data or software, then an alternative
would be to just "strongly recommend" instead of "require" the new
feature. It's also possible to specify that a new requirement will come
into effect at some point in the future to allow time for software
systems to adapt, as was done with the Y2000 change to the DATE keyword
format.

In the new draft of the FITS Standard, the technical panel listed 22
specific changes to the FITS requirements in the "Summary of Recommended
Changes" document (at http://fits.gsfc.nasa.gov/fits_draft.html), but
most of these either remove requirements, or only add a recommendation.
There are only 3 proposed new absolute requirements in this list:

1. Keywords that have a value shall not be repeated in a header.

2. PCOUNT and GCOUNT must immediately follow the last NAXISn
keyword in all conforming extensions (as is already required
in IMAGE, TABLE, and BINTABLE extensions).

3. Embedded space characters are now forbidden within numeric
values in an ASCII Table (e.g. "1 23 4.5" is no longer
allowed to represent the decimal value 1234.5)

The public comment period on these, as well as all the other recommended
changes, remains open here on this email list/newsgroup until at least
the end of September...

Bill Pence
--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)

  #10  
Old August 17th 07, 06:29 PM posted to sci.astro.fits
Steve Allen
external usenet poster
 
Posts: 37
Default [fitsbits] Proposed Changes to the FITS Standard

On Fri 2007-08-17T13:18:40 -0400, William Pence hath writ:
1. Keywords that have a value shall not be repeated in a header.


If this is to be implemented with exactly that wording then
on behalf of UCO/Lick/Keck I have to ask for a very clear
answer to this question:

Starting when?

We can do it, but in order to move the organization to get there we're
going to need a little warning beforehand.

--
Steve Allen WGS-84 (GPS)
UCO/Lick Observatory Natural Sciences II, Room 165 Lat +36.99855
University of California Voice: +1 831 459 3046 Lng -122.06015
Santa Cruz, CA 95064 http://www.ucolick.org/~sla/ Hgt +250 m
 




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[fitsbits] Proposed Changes to the FITS Standard Mark Calabretta FITS 0 August 2nd 07 09:39 AM
[fitsbits] Proposed Changes to the FITS Standard Steve Allen FITS 0 August 1st 07 06:08 PM
[fitsbits] Proposed Changes to the FITS Standard Thierry Forveille FITS 0 August 1st 07 04:51 PM
[fitsbits] Proposed Changes to the FITS Standard William Pence FITS 0 July 27th 07 07:38 PM
[fitsbits] Proposed Changes to the FITS Standard Rob Seaman FITS 0 July 24th 07 07:21 PM


All times are GMT +1. The time now is 07:23 PM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 SpaceBanter.com.
The comments are property of their posters.