Question(s) regarding development of proprietary FITS manipulation software. . .

Michael,

(replying to Steve's messages as I find it easier to piggyback in
this case)

Steve Allen wrote:
On Mon 2007-08-27T11:32:19 -0400, Michael Williams hath writ:
we're obviously speaking terribly different languages here.

We are also not managing to communicate that FITS has been in use
for nearly 30 years. It does work for exchanging pictures
of the sky, catalogs of the sky, and other related information.

and the "I" stands for interchange, not for image.

Over those 30 years there have probably been thousands of person-years
of effort developing the software which interprets the information in

(and writes)

the FITS files. That may be a major point of disconnect here.

I think there is, and it is major.

Are the images
literally live pictures that were taken, or some arbitrary graphical
representation of data?

Yes, but in optical astronomy the first notion is usually a safe one.

this is part of the disconnect.
Even in optical astronomy you have other data in a FITS file than
a "picture" (= e.g. CCD image?). Think spectra, even light curves.

The data array do not by and in themselves have any kind of graphical
information or representation attached to them. They are data points
with coordinates, where coordinates don't even have to be spacial
(e.g. wavelength or time come to mind).

A spectrum is not a 'picture'. You can make a picture out of the
spectrum, but the detail of the information in it will be lost
independent of how you turn it into a picture.

3) How do I determine what type of image manipulations are legitimate
for any type of image data? Should I allow sepia toning? Should I

why do you care? A FITS data array, even if it represents an array
of data with coordinates X and Y does not have colors associated
with it. Taking this with "beauty is in the eye of the beholder",
any color scheme will work (think false color imaging like e.g.
in infrared weather satellite imagery or NEXRAD), depending on what
the person looking at it prefers.

allow them to run photoshop filters on the pictures? I mean, what,

they aren't 'pictures', they're images, and that isn't the same.

really is the usefulness of any manipulation on the image and what
are the most common techniques?

In the most general sense the answers to these questions count as the
fulfillment of the requirements for PhD theses and refereed scientific
literature.
"If we knew what we were doing it wouldn't be research."

g

4) What is the goal of scientists when it comes to examining
another's FITS files? What kind of "information" is truly gathered
from the "data"?

Why do you put "data" in double quotes?

This is science.

Taking a Schmidt plate or CCD image as an example: I may be looking
for galaxies, the next person for dark clouds and a third one for
planetary nebulae. Taking spectra, there's temperature and other
physical condition that can be derived, as well as chemistry.


If your client gave you examples (in data format, rather than in
picture format) saying "this is what we want", are you
second-guessing them?

HTH,

Maren

Alright, at the risk of sounding like a broken record and drawing the
ire of the community, would someone be willing to look at a couple of
files that I have and please explain what is occurring? I'm using 3
programs to check the information: 1) SAOImage DS9 2) FV 3) Graphic
Converter. All 3 programs display clearly visible images. . .some
pictures of stars, others of the moon, and others of what I'm guessing
are "gas" clouds.

As per LCNoSpam's post, I'm pretty sure it is a matter of Case "a".
I've got my program parsing out the header data just fine. But the
raw image data is useless to me unless I know how to interpret it.
The required header values are present (e.g. BITPIX, NAXIS, etc.), but
I still don't fully understand the interpretation.

Again, my apologies for the repeated post, but I'm absolutely in need
of help. I've read more documentation than I care to recount, but my
head is still not wrapping around the handling of the actual images
aspect of this project.

Best Regards,
Michael


On Aug 27, 5:48 am, LC's NoSpam Newsreading account
wrote:
On Fri, 24 Aug 2007, Michael Williams wrote:
valid FITS files all formatted completely differently. [...] the most
common "formats". Can someone send me (offline or otherwise) some
sample data with accompanying comments/markup regarding what each
piece of data means?

You can do this better by yourself. On one hand grab a copy of the FITS
standard fromhttp://fits.gsfc.nasa.gov/fits_home.html(there is even a
draft of the most recent revision, which does not change really the
format, but has maybe clearer explanation), on the other way consult the
archives of the major ground observatories (ESO, NOAO, NRAO) and of the
major space missions (NASA and ESA).

FITS compliant header followed by. . .

All headers I know are "compliant" ... what you should get from the FITS
standard is the list of : mandatory keywords which appear in ANY header
of a given sub-type (*) ; reserved keywords which, when they appear,
must be used as prescribed in the standard ; other keywords which
essentially can be considered of commentary or specific nature (cfr.
e.g. the convention registryhttp://fits.gsfc.nasa.gov/fits_conventions.html)

I am not sure what you intend to do. I suppose some sort of new general
purpose tool. Probably you should concentrate on interpreting the
mandatory keywords, and some of the reserved keywords, and ignore all
the rest (even if not of commentary nature, might be useful only for
some project specific piece of software).

(*) by sub-type I mean here the various kinds of extensions :

a) primary HDU with image data and IMAGE extensions
b) binary table extensions
c) ascii table extensions
d) primary HDU with random groups
e) anything else

Let me first say that (e) is extremely unfrequent (that would mean
conforming extensions not part of the standard, which at present is only
the FOREIGN extension, which is a way of wrapping non-FITS stuff in a
FITS file used at some places), so you'd just ignore it.

I'll then add that (d) is an old format, now deprecated, and used
essentially only by radio astronomers. I guess it is unlikely you'd
encounter it unless you look in the archives of radio observatories.

Now for your classification

1) binary data
3) array's of numbers

I'm not really sure where you draw the difference here. I suppose one
case may be my "a" and another my "b", or perhaps "c". See further
below.

2) non-compliant header-like text

I am not sure what you mean by this too. Whether you mean "c" (ASCII
tables ... look printable text but not in header format), or whether you
mean some kind of headers (ESO HIERARCH convention perhaps ? looks odd
but is compliant !).

Essentially what I say is that it is my a/b/c classification which makes
sense (and unfortunately is not enough).

(a) the case of images is possibly the simplest. It may occur in the
primary HDU, or in XTENSION='IMAGE' extensions. The data which
follows is a binary n-dimensional array A(i,j,...). In most cases
a 2-d image A(i,j), in some cases a 1-d array A(i), in some other a
datacube etc.

In most cases of the 2-d images they are sky images (and data cubes
are stacks of sky images), but can be anything (also in some cases
they are actual detector images, they may be something different,
consider e.g. a multi-object spectrometer).

The default action here is to display a 2-d image with some false
colour mapping. On the X and Y axes just put "pixels".

If the header has WCS keywords, these would allow to map pixel
coordinates to sky coordinates OR OTHER "world" (physical)
coordinates. Rather complex to handle.

Example handling program is ds9.

(b) the case of binary tables is nowadays probably the next most diffuse
but comes in a large number of varieties. First of all it always
comes as an extension, so you have a dataless PHDU and an extension
header.

The data are a binary array with a number of rows, and a number of
columns of DIFFERENT data types. To further complicate things,
a column may have a depth (see the TFORMn keywords). To further
complicate things, this depth way be constant, or variable (in
which case the cell contains pointers to data in the HEAP, see
the P and Q TFORMn)

Here I'd say that the typical "generic" action (example handling
program is fv) is to produce a tabular printout of the content
(optionally a plot of one column vs another ... but this can be
chosen only interactively). With some special action for large
or variable depth cells.

Some reserved optional kwds can be used to annotate the column
with a title or plot label, with physical units, with a recommended
display format ...

This format is widely used for lots of different things : a
catalogue of sources, or any database ; a spectrum (but sometimes
spectra are formatted as 1-d images) ; a time series ; a photon
list or even more complex things like the RMF (Response Matrix
Function) of an high energy detector.

... and of course a file may contain many different extensions.
You should consider each one as independent.

(c) the case of ascii tables is nowadays encountered mainly in older
(or old-style) archives, typically of catalogues. The data is
written in fixed-length ASCII record (so it appears printable).

It is simpler than binary tables (columns have always depth one)
but for the rest it shall be displayed similarly.

--
----------------------------------------------------------------------
is a newsreading account used by more persons to
avoid unwanted spam. Any mail returning to this address will be rejected.
Users can disclose their e-mail address in the article if they wish so.

The silence is deafening. . .

Alright all. I suppose I was asking the right question(s) in the
wrong manner in the beginning. I have now actually figured out how to
interpret 2-D image data. The information literally has to be read in
8-byte increments and converted to a double. That will then allow me
to map RGB pixel data to a buffer.

*THAT* is what I needed to know. How to read in and interpret the
data. I had *NO* idea what that "garbage" was supposed to mean or how
many chickens I needed to sacrifice in order to get it to work.

Anyway, thanks to all that contributed. My apologies for any
confusion.

Regards,
Michael

On Aug 27, 10:38 am, Michael Williams wrote:
Ok, I actually asked a follow-on question in another post/e-mail.
Basically I asked if these astronomers images were literally live
images (perhaps of the stars, or nieces and nephews), or if they were
simply arbitrary graphical representations of data that had no
reference in reality. Also, given the fact that, as you said,
reverse transformation isn't always possible, why in the world does
it matter if the user can perform image manipulations of any type? I
mean, if there is no consistent way to produce an image, then why do
we care?

Thanks,
Michael

On Aug 27, 2007, at 5:15 AM, Thierry Forveille wrote:

gberz3 a écrit :
Why would *anyone* present FITS data as images if they are
not image data? Why not represent it as sound?
I guess that's what I'm getting at. What relevance does an image
have
to actual FITS data if there is no "attached" image, and what is the
proper means by which to display said image?
The issue is mostly your notion of an image, vs an astronomer's
notion of the same. To you an image is something that can be
univocally displayed on a screen or printed, while to
astronomers it is a set of values (ideally in physical units,
such as Watts per square meter per steradian) sampled on a
regular grid. There is some link between the two notions,
but not a unique one: an astronomer's image can be displayed
but not in one unique way, and the reverse transformation
if/when at all possible, requires additional information
(e.g. the physical values for a subset of the pixels) and
significant calibration work. Essentially, an astromer's
image is a richer dataset, and someone needs to decide how
to degrade that information to what a display can show.

In addition, images (in the astronomer's definition) is only
one of the data classes that can be stored in FITS.

you use cfitsio (well, I use cfitsio because I primarily
program in C these days when I have a chance, but I do a
few other things than write FITS files), and read in the header
which then tells you what length your data are ...

Again, the data have nothing to do with RGB pixels,
they are in physically meaningful units (and if it isn't
specified what they are you have pretty unreduceable (sp?)
data). You can map them to a color scale, but that's up
to you.

Maren
(trying to design and implement a cryostat control
system that needs to be converted from VAX FORTRAN
to EPICS, probably on vxWorks, but could end up being
Linux - on the side to dealing with operational issues)

On Tue, 18 Sep 2007, gberz3 wrote:

The silence is deafening. . .

Alright all. I suppose I was asking the right question(s) in the
wrong manner in the beginning. I have now actually figured out how to
interpret 2-D image data. The information literally has to be read in
8-byte increments and converted to a double. That will then allow me
to map RGB pixel data to a buffer.

*THAT* is what I needed to know. How to read in and interpret the
data. I had *NO* idea what that "garbage" was supposed to mean or how
many chickens I needed to sacrifice in order to get it to work.

Anyway, thanks to all that contributed. My apologies for any
confusion.

Regards,
Michael

On Aug 27, 10:38 am, Michael Williams wrote:
Ok, I actually asked a follow-on question in another post/e-mail.
Basically I asked if these astronomers images were literally live
images (perhaps of the stars, or nieces and nephews), or if they were
simply arbitrary graphical representations of data that had no
reference in reality. Also, given the fact that, as you said,
reverse transformation isn't always possible, why in the world does
it matter if the user can perform image manipulations of any type? I
mean, if there is no consistent way to produce an image, then why do
we care?

Thanks,
Michael

On Aug 27, 2007, at 5:15 AM, Thierry Forveille wrote:

gberz3 a écrit :
Why would *anyone* present FITS data as images if they are
not image data? Why not represent it as sound?
I guess that's what I'm getting at. What relevance does an image
have
to actual FITS data if there is no "attached" image, and what is the
proper means by which to display said image?
The issue is mostly your notion of an image, vs an astronomer's
notion of the same. To you an image is something that can be
univocally displayed on a screen or printed, while to
astronomers it is a set of values (ideally in physical units,
such as Watts per square meter per steradian) sampled on a
regular grid. There is some link between the two notions,
but not a unique one: an astronomer's image can be displayed
but not in one unique way, and the reverse transformation
if/when at all possible, requires additional information
(e.g. the physical values for a subset of the pixels) and
significant calibration work. Essentially, an astromer's
image is a richer dataset, and someone needs to decide how
to degrade that information to what a display can show.

In addition, images (in the astronomer's definition) is only
one of the data classes that can be stored in FITS.