A Space & astronomy forum. SpaceBanter.com

Go Back   Home » SpaceBanter.com forum » Astronomy and Astrophysics » FITS
Site Map Home Authors List Search Today's Posts Mark Forums Read Web Partners

[fitsbits] New DUMP FITS extension



 
 
Thread Tools Display Modes
  #1  
Old August 16th 07, 07:56 PM posted to sci.astro.fits
William Pence
external usenet poster
 
Posts: 66
Default [fitsbits] New DUMP FITS extension

One of the proposed changes to the FITS Standard is a complete rewrite
of the Appendix F (previously Appendix I) which lists the reserved FITS
extension type names (i.e., the value of the XTENSION keyword). One of
the registered extension types listed in that appendix is as follows:

'DUMP ' - Suggested extension name for storing a stream of
binary data (such as a telemetry stream) in a FITS file. This
extension type was never implemented, but the FOREIGN extension
type serves a similar purpose.

This definition may now need to be updated because I recently learned
that the FITS data from Solar Optical Telescope on the Hinode satellite
(http://solar-b.nao.ac.jp/index_e.shtml), launched in late 2006,
contains a DUMP extension with the following minimal set of header keywords:

XTENSION= 'DUMP '
BITPIX = 8
NAXIS = 1
NAXIS1 = 300
PCOUNT = 0
GCOUNT = 1
END

My understanding is that this extension contains the image header packet
from the satellite telemetry. The NAXIS1 value may vary, but otherwise
all the DUMP extensions look the same.

The IAU FITS Working Group is charged with maintaining the list of all
registered FITS extension types, and at least at some level, is
responsible for defining how each type is supposed to be used. A number
of questions regarding the DUMP extension would need to be resolved
before writing such a definition document:

1. Is this how the DUMP extension was originally envisioned to be
implemented? An alternate scheme, as used in the FOREIGN extension
type, would be:

XTENSION= 'DUMP '
BITPIX = 8
NAXIS = 0
PCOUNT = 300
GCOUNT = 1
END

Is one of these 2 forms preferable over the other?

2. How does the DUMP extension as used in the Hinode data differ from
an IMAGE or FOREIGN extension? What are the fundamental differences, if
any, between the DUMP and IMAGE (or FOREIGN) extensions? What criteria
should data providers use in deciding which extension type to use?

These questions are particularly relevant now because the new draft FITS
Standard in section 3.4.1 states: "New extension types should be
created only when the organization of the information is such that it
cannot be handled by one of the existing extension types." (This
requirement comes from the original "Generalized Extensions" FITS paper
published in 1987). Given that an IMAGE extension presumably could be
used to store the same data, does the DUMP extension satisfy this
requirement?

3. Can the DUMP extension have a BITPIX value other than 8, and a NAXIS
value other than 1? Must PCOUNT always = 0 and GCOUNT always = 1? If
any of these keywords can have different values, how does this affect
the interpretation of the data?

Presumably these questions, and more, should be answered in a short
definition document that specifies exactly how the use DUMP extension
should be used. At the moment however, it is not clear to me that
there is any consensus on the answer to these questions.

Bill Pence
--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)


  #2  
Old August 17th 07, 10:15 AM posted to sci.astro.fits
LC's NoSpam Newsreading account[_2_]
external usenet poster
 
Posts: 23
Default [fitsbits] New DUMP FITS extension

On Thu, 16 Aug 2007, William Pence wrote:

I recently learned that the FITS data from Solar Optical Telescope on
the Hinode satellite [...] contains a DUMP extension with

BITPIX = 8
NAXIS = 1
NAXIS1 = 300
PCOUNT = 0
GCOUNT = 1


.... which actually resembles a 1-d image ...

My understanding is that this extension contains the image header
packet from the satellite telemetry.


.... if one wishes to encapsulate raw telemetry in a FITS file, the idea
of using an ad-hoc extension is sound ...


1. Is this how the DUMP extension was originally envisioned to be
implemented?



WHO KNOWS ? Who did suggest the original name ?

I think it was a mistake on our part to accept the registration of a
name without an implementation proposal, without keeping track of who
might have drafted such proposal, and without setting an expiry date.

We should define a more formal procedure for extensions which either (or
both) :

- allows pre-registration of a name for a fixed time (6 months ?) after
which the name is retired if no implementation proposal has come

- allows direct registration of an existing implementation (probably
contextually with a registration of an associated "convention") like
for FOREIGN


An alternate scheme, as used in the FOREIGN extension type, would be:

BITPIX = 8
NAXIS = 0
PCOUNT = 300
GCOUNT = 1


Is one of these 2 forms preferable over the other?


The second one WOULD BE preferable for me for the packing of an
arbitrary blob of data into a FITS file.

However if the arbitrary data is organized as a fixed record length file
(anybody remembering IBM JCL ?) it might make sense using NAXIS1 and
NAXIS2

But it is now too late to ask this question (see further below) !


2. How does the DUMP extension as used in the Hinode data differ from an
IMAGE or FOREIGN extension?


For me the difference is that an IMAGE extension is an image (a 2-d or
n-d array, not necessarily a spatial image), a FOREIGN extension is the
encapsulation of some specific FILE structure (with some relation to
actual "live" operating systems e.g. for things like file naming, or
even directory trees, soft links etc (which may become obsolescent in
the future). A DUMP will be just a blob of data with no implications as
files at OS level (users will have freedom to use it in their OS).

A further possibility I thought of in the recent past would be a MIME
extension, similar to FOREIGN but with no "live" OS dependency, but just
encapsulating a mime file with its content type and other header (which
presumably is more general, less OS-related and more long lived).


In a later mail you wrote :

The Hinode satellite project is now publicly distributing FITS data
files with this DUMP extension. They apparently saw the DUMP
extension name in the list of proposed extensions in the appendix to
the FITS standard and decided to use that extension name for their
telemetry data. Even though there is no specification of what the
DUMP extension keywords should look like


Again, it was our fault that no such specification did exist, and still
DUMP was in the namespace !

The only thing we can do now is asking the Hinode people to provide a
documentation of their *convention*, enter it in the convention
registry, and ideally "assign" to them the registered DUMP extension
(like FOREIGN is "assigned" to NOAO).

This assumes nobody else is using another DUMP variant. If so one could
have different (hopefully not inconsistent) conventions : a Hinode DUMP
convention, an Erewhon DUMP convention ...


On Thu, 16 Aug 2007, Doug Tody wrote in two other messages:

Since the convention is in use for actual data, we should probably
follow our existing practice of merely documenting existing
conventions, and describe the convention/extension as used by Hinode.


I agree with Doug in the terms specified above (the Hinode people should
provide the documentation).

My personal view on the more general question is that Image should
only be used for image data, and it is a trick to stuff arbitrary
binary blobs in something called an Image.


I also agree with this. It is not "clean" to store a blob as an 1xn
image nor as an 1xn binary table.

But what is then an image ?

One answer would be that if it is array data of any dimension it is
reasonable to store it as an "image"


I also agree here. I see absolutely no difficulty in storing e.g. a
chi-square grid vs spectral index and NH as an image.

I have also seen people storing 1-d spectra or histograms as images, or
even collections of 1-d arrays (typically spectra) packed in an image
(e.g. gross, bkg, net and quality flag vs wavelength stored as 4 rows in
an image). Although in mostt cases my preference would be to use a
binary table, both usages are legal and probably nearly equally
efficient.


Lucio Chiappetti

--
----------------------------------------------------------------------
is a newsreading account used by more persons to
avoid unwanted spam. Any mail returning to this address will be rejected.
Users can disclose their e-mail address in the article if they wish so.
 




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[fitsbits] fitsbits--sci.astro.fits gateway fully operationalagain Don Wells FITS 0 September 29th 06 05:17 PM
[fitsbits] sci.astro.fits link to FITSBITS is still broken William Pence FITS 0 May 31st 06 07:21 PM
FITS long integer support (was [fitsbits] ADASS FITS BoF onSunday) Eric Greisen FITS 10 October 26th 04 08:14 AM
FITS long integer support (was [fitsbits] ADASS FITS BoFon Sunday) William Pence FITS 6 October 22nd 04 08:23 PM
FITS long integer support (was [fitsbits] ADASS FITS BoF onSunday) Thomas McGlynn FITS 0 October 20th 04 03:18 PM


All times are GMT +1. The time now is 05:58 AM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 SpaceBanter.com.
The comments are property of their posters.