A Space & astronomy forum. SpaceBanter.com

Go Back   Home » SpaceBanter.com forum » Astronomy and Astrophysics » FITS
Site Map Home Authors List Search Today's Posts Mark Forums Read Web Partners

[fitsbits] BLANK keyword misinterpretation



 
 
Thread Tools Display Modes
  #1  
Old November 21st 03, 01:22 AM
Steve Allen
external usenet poster
 
Posts: n/a
Default [fitsbits] BLANK keyword misinterpretation

We are currently testing of the new CCD mosaic which is being readied
for shipment to Keck to replace the old detector in the HIRES
spectrograph. The laboratory setup permits illumination patterns
which are hard to duplicate in the instrument after deployment.
In the course of examining the images we discovered that I had made a
misinterpretation of the FITS standard.

I believe that the FITS standard is unambiguous. Nevertheless, slight
changes to the wording of the standard could be very useful in
preventing other such misinterpretations. Until it becomes possible
to make such changes other implementors should be cautious when using
the BLANK keyword. It is also possible that changes to the APIs that
are used for writing FITS images would help.

Our data originate via a 16-bit AtoD conversion which produces bits
corresponding to C "unsigned short" on modern big-endian processors.
The pixel values are always offset such that all values are positive.
We store the bits directly in the image arrays as unsigned short
integers using the traditional trick of offsetting the physical data
values using the keywords

BITPIX = 16
BZERO = 32768
BSCALE = 1

The complete HDU structure of the mosaic FITS files is created at the
beginning of the CCD readout, and at this time the values in the data
arrays of all IMAGE extensions are initialized to be all zero bits.
During the readout the FITS file is stored in shared memory where it
can be accessed by FITS viewer to provide real-time display of the
incoming image. In our systems the pixels may not arrive in sequence,
so the images being constructed may have "holes" during the readout.

The "holes" in the image during the readout confound the automatic
contrast setting algorithm in the FITS viewer. However, the FITS
viewer that we use understands the usage of the BLANK keyword. In
order to instruct it ignore pixels that do not yet have data I
modified the FITS headers to include the keyword

BLANK = 0

Whereas this had the intended effect during real-time readout, this
was a mistake for the archival data.

FITS is defined by NOST 100-2.0 as printed in
http://adsabs.harvard.edu/cgi-bin/np...6A...376..359H

The interpretation of BITPIX is shown by Table 5.2 in section 5.4.1.1.
I believe it would be helpful to many implementors if the text where
slightly enhanced such that the entries
16-bit twos-complement binary integer
32-bit twos-complement binary integer
were changed to read
16-bit twos-complement (signed) binary integer
32-bit twos-complement (signed) binary integer

The definition of the data representation for BITPIX = 16 is given by
section 6.2.2. It is completely unambiguous.

The definition of the BLANK keyword is given in section 5.4.2.5.
Its relevant sentence reads:
The value field shall contain an integer that specifies the
representation of array values whose physical values are
undefined.
This is unambiguous, but I misinterpreted it. I would have got
the interpretation right if there had been one extra sentence
akin to this:
This number corresponds to the data array values before applying
any transformation indicated by BSCALE and BZERO.

The problem is that with these values of BZERO and BSCALE the signed
FITS array value of 0 corresponds to the unsigned CCD data value of
32768. The result of my misinterpretation is that images with a
gentle ramp of CCD data values in the vicinity of 32768 have sprinkles
of single pixels where a conforming FITS viewer flags the pixels as
NULL or undefined.

In our software the fix for this is relatively straightforward. The
image in shared memory during readout is currently used only for
real-time display. There is a final pass over each HDU where keyword
are modified before the image is written to disk for archiving. In
that pass the software will reset the keywords such that

BLANK = -32768

This change will have no significant impact on real-time performance.

We are using CFITSIO as the API for creating these FITS images.
This problem could also be alleviated if the API provided an
explicitly defined means of indicating BLANK along with other
parameters of the image.

--
Steve Allen UCO/Lick Observatory Santa Cruz, CA 95064
Voice: +1 831 459 3046 http://www.ucolick.org/~sla
PGP: 1024/E46978C5 F6 78 D1 10 62 94 8F 2E 49 89 0E FE 26 B4 14 93

  #2  
Old November 21st 03, 03:09 PM
Arnold Rots
external usenet poster
 
Posts: n/a
Default [fitsbits] BLANK keyword misinterpretation

Shouldn't you also flip the sign bit?
That's part of the traditional trick.

- Arnold

Steve Allen wrote:
...
Our data originate via a 16-bit AtoD conversion which produces bits
corresponding to C "unsigned short" on modern big-endian processors.
The pixel values are always offset such that all values are positive.
We store the bits directly in the image arrays as unsigned short
integers using the traditional trick of offsetting the physical data
values using the keywords

BITPIX = 16
BZERO = 32768
BSCALE = 1
...

--
Steve Allen UCO/Lick Observatory Santa Cruz, CA 95064
Voice: +1 831 459 3046 http://www.ucolick.org/~sla
PGP: 1024/E46978C5 F6 78 D1 10 62 94 8F 2E 49 89 0E FE 26 B4 14 93

_______________________________________________
fitsbits mailing list

http://listmgr.cv.nrao.edu/mailman/listinfo/fitsbits

--------------------------------------------------------------------------
Arnold H. Rots Chandra X-ray Science Center
Smithsonian Astrophysical Observatory tel: +1 617 496 7701
60 Garden Street, MS 67 fax: +1 617 495 7356
Cambridge, MA 02138
USA
http://hea-www.harvard.edu/~arots/
--------------------------------------------------------------------------

  #3  
Old November 21st 03, 03:09 PM
Arnold Rots
external usenet poster
 
Posts: n/a
Default [fitsbits] BLANK keyword misinterpretation

Shouldn't you also flip the sign bit?
That's part of the traditional trick.

- Arnold

Steve Allen wrote:
...
Our data originate via a 16-bit AtoD conversion which produces bits
corresponding to C "unsigned short" on modern big-endian processors.
The pixel values are always offset such that all values are positive.
We store the bits directly in the image arrays as unsigned short
integers using the traditional trick of offsetting the physical data
values using the keywords

BITPIX = 16
BZERO = 32768
BSCALE = 1
...

--
Steve Allen UCO/Lick Observatory Santa Cruz, CA 95064
Voice: +1 831 459 3046 http://www.ucolick.org/~sla
PGP: 1024/E46978C5 F6 78 D1 10 62 94 8F 2E 49 89 0E FE 26 B4 14 93

_______________________________________________
fitsbits mailing list

http://listmgr.cv.nrao.edu/mailman/listinfo/fitsbits

--------------------------------------------------------------------------
Arnold H. Rots Chandra X-ray Science Center
Smithsonian Astrophysical Observatory tel: +1 617 496 7701
60 Garden Street, MS 67 fax: +1 617 495 7356
Cambridge, MA 02138
USA
http://hea-www.harvard.edu/~arots/
--------------------------------------------------------------------------

  #4  
Old November 21st 03, 05:42 PM
Steve Allen
external usenet poster
 
Posts: n/a
Default [fitsbits] BLANK keyword misinterpretation

On Fri 2003-11-21T09:09:29 -0500, Arnold Rots hath writ:
Shouldn't you also flip the sign bit?
That's part of the traditional trick.


Yes.
Without that the data value in every pixel would be wrong.
With the BLANK value wrong it was only a few in 10000, and
randomly scattered, in most non-lab images.

--
Steve Allen UCO/Lick Observatory Santa Cruz, CA 95064
Voice: +1 831 459 3046 http://www.ucolick.org/~sla
PGP: 1024/E46978C5 F6 78 D1 10 62 94 8F 2E 49 89 0E FE 26 B4 14 93

  #5  
Old November 21st 03, 05:42 PM
Steve Allen
external usenet poster
 
Posts: n/a
Default [fitsbits] BLANK keyword misinterpretation

On Fri 2003-11-21T09:09:29 -0500, Arnold Rots hath writ:
Shouldn't you also flip the sign bit?
That's part of the traditional trick.


Yes.
Without that the data value in every pixel would be wrong.
With the BLANK value wrong it was only a few in 10000, and
randomly scattered, in most non-lab images.

--
Steve Allen UCO/Lick Observatory Santa Cruz, CA 95064
Voice: +1 831 459 3046 http://www.ucolick.org/~sla
PGP: 1024/E46978C5 F6 78 D1 10 62 94 8F 2E 49 89 0E FE 26 B4 14 93

 




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
S&T Bulletins .... the REAL numbers Anthony Ayiomamitis Amateur Astronomy 9 January 3rd 04 10:22 PM
Electric Gravity&Instantaneous Light ralph sansbury Astronomy Misc 8 August 31st 03 02:53 AM


All times are GMT +1. The time now is 07:36 PM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 SpaceBanter.com.
The comments are property of their posters.