A Space & astronomy forum. SpaceBanter.com

Go Back   Home » SpaceBanter.com forum » Astronomy and Astrophysics » FITS
Site Map Home Authors List Search Today's Posts Mark Forums Read Web Partners

Question(s) regarding development of proprietary FITS manipulation software. . .



 
 
Thread Tools Display Modes
  #21  
Old August 27th 07, 07:08 AM posted to sci.astro.fits
Steve Allen
external usenet poster
 
Posts: 37
Default [fitsbits] Question(s) regarding development of proprietaryFITS manipulation software. . .

On Sun 2007-08-26T22:26:09 -0700, Rob Seaman hath writ:
For the proper means, you might download a copy of Ximtool or DS9 to
reverse engineer the common techniques, the representational language
(perhaps start with "window and stretch") and the graphical tools
(histogram vectors are more expressive than scalar gammas, for
instance).


I'm surprised to see Rob not referring to the IRAF source code, for
there are FITS files which contain a 2-d array in which each row
is a spectrum. In that case the proper display is a series of x-y
plots showing spectral intensity as a function of some sort of
wavelength.

The X-ray satellites tend to put their image data into tables.
The fv tool from HEASARC can present these as images as well as
via other sorts of database-like queries.

The radio observations have many formats, some of the earlies of which
are 3-d -- a stack of 2-d images where each layer represents a
different polarization. In that case the code to look at is AIPS and
its kin.

I'm still not getting the underlying driver for the project.

Perhaps the most pertinent question is whether there is any particular
source of FITS files which is most relevant for this proprietary tool.

--
Steve Allen WGS-84 (GPS)
UCO/Lick Observatory Natural Sciences II, Room 165 Lat +36.99855
University of California Voice: +1 831 459 3046 Lng -122.06015
Santa Cruz, CA 95064 http://www.ucolick.org/~sla/ Hgt +250 m
  #22  
Old August 27th 07, 10:15 AM posted to sci.astro.fits
Thierry Forveille
external usenet poster
 
Posts: 14
Default [fitsbits] Question(s) regarding development ofproprietary FITS manipulation software. . .

gberz3 a écrit :
Why would *anyone* present FITS data as images if they are
not image data? Why not represent it as sound?

I guess that's what I'm getting at. What relevance does an image have
to actual FITS data if there is no "attached" image, and what is the
proper means by which to display said image?

The issue is mostly your notion of an image, vs an astronomer's
notion of the same. To you an image is something that can be
univocally displayed on a screen or printed, while to
astronomers it is a set of values (ideally in physical units,
such as Watts per square meter per steradian) sampled on a
regular grid. There is some link between the two notions,
but not a unique one: an astronomer's image can be displayed
but not in one unique way, and the reverse transformation
if/when at all possible, requires additional information
(e.g. the physical values for a subset of the pixels) and
significant calibration work. Essentially, an astromer's
image is a richer dataset, and someone needs to decide how
to degrade that information to what a display can show.

In addition, images (in the astronomer's definition) is only
one of the data classes that can be stored in FITS.
  #23  
Old August 27th 07, 10:48 AM posted to sci.astro.fits
LC's NoSpam Newsreading account[_2_]
external usenet poster
 
Posts: 23
Default [fitsbits] Question(s) regarding development of proprietary FITSmanipulation software. . .

On Fri, 24 Aug 2007, Michael Williams wrote:

valid FITS files all formatted completely differently. [...] the most
common "formats". Can someone send me (offline or otherwise) some
sample data with accompanying comments/markup regarding what each
piece of data means?


You can do this better by yourself. On one hand grab a copy of the FITS
standard from http://fits.gsfc.nasa.gov/fits_home.html (there is even a
draft of the most recent revision, which does not change really the
format, but has maybe clearer explanation), on the other way consult the
archives of the major ground observatories (ESO, NOAO, NRAO) and of the
major space missions (NASA and ESA).

FITS compliant header followed by. . .


All headers I know are "compliant" ... what you should get from the FITS
standard is the list of : mandatory keywords which appear in ANY header
of a given sub-type (*) ; reserved keywords which, when they appear,
must be used as prescribed in the standard ; other keywords which
essentially can be considered of commentary or specific nature (cfr.
e.g. the convention registry
http://fits.gsfc.nasa.gov/fits_conventions.html)

I am not sure what you intend to do. I suppose some sort of new general
purpose tool. Probably you should concentrate on interpreting the
mandatory keywords, and some of the reserved keywords, and ignore all
the rest (even if not of commentary nature, might be useful only for
some project specific piece of software).

(*) by sub-type I mean here the various kinds of extensions :

a) primary HDU with image data and IMAGE extensions
b) binary table extensions
c) ascii table extensions
d) primary HDU with random groups
e) anything else

Let me first say that (e) is extremely unfrequent (that would mean
conforming extensions not part of the standard, which at present is only
the FOREIGN extension, which is a way of wrapping non-FITS stuff in a
FITS file used at some places), so you'd just ignore it.

I'll then add that (d) is an old format, now deprecated, and used
essentially only by radio astronomers. I guess it is unlikely you'd
encounter it unless you look in the archives of radio observatories.

Now for your classification

1) binary data
3) array's of numbers


I'm not really sure where you draw the difference here. I suppose one
case may be my "a" and another my "b", or perhaps "c". See further
below.

2) non-compliant header-like text


I am not sure what you mean by this too. Whether you mean "c" (ASCII
tables ... look printable text but not in header format), or whether you
mean some kind of headers (ESO HIERARCH convention perhaps ? looks odd
but is compliant !).

Essentially what I say is that it is my a/b/c classification which makes
sense (and unfortunately is not enough).

(a) the case of images is possibly the simplest. It may occur in the
primary HDU, or in XTENSION='IMAGE' extensions. The data which
follows is a binary n-dimensional array A(i,j,...). In most cases
a 2-d image A(i,j), in some cases a 1-d array A(i), in some other a
datacube etc.

In most cases of the 2-d images they are sky images (and data cubes
are stacks of sky images), but can be anything (also in some cases
they are actual detector images, they may be something different,
consider e.g. a multi-object spectrometer).

The default action here is to display a 2-d image with some false
colour mapping. On the X and Y axes just put "pixels".

If the header has WCS keywords, these would allow to map pixel
coordinates to sky coordinates OR OTHER "world" (physical)
coordinates. Rather complex to handle.

Example handling program is ds9.

(b) the case of binary tables is nowadays probably the next most diffuse
but comes in a large number of varieties. First of all it always
comes as an extension, so you have a dataless PHDU and an extension
header.

The data are a binary array with a number of rows, and a number of
columns of DIFFERENT data types. To further complicate things,
a column may have a depth (see the TFORMn keywords). To further
complicate things, this depth way be constant, or variable (in
which case the cell contains pointers to data in the HEAP, see
the P and Q TFORMn)

Here I'd say that the typical "generic" action (example handling
program is fv) is to produce a tabular printout of the content
(optionally a plot of one column vs another ... but this can be
chosen only interactively). With some special action for large
or variable depth cells.

Some reserved optional kwds can be used to annotate the column
with a title or plot label, with physical units, with a recommended
display format ...

This format is widely used for lots of different things : a
catalogue of sources, or any database ; a spectrum (but sometimes
spectra are formatted as 1-d images) ; a time series ; a photon
list or even more complex things like the RMF (Response Matrix
Function) of an high energy detector.

... and of course a file may contain many different extensions.
You should consider each one as independent.

(c) the case of ascii tables is nowadays encountered mainly in older
(or old-style) archives, typically of catalogues. The data is
written in fixed-length ASCII record (so it appears printable).

It is simpler than binary tables (columns have always depth one)
but for the rest it shall be displayed similarly.

--
----------------------------------------------------------------------
is a newsreading account used by more persons to
avoid unwanted spam. Any mail returning to this address will be rejected.
Users can disclose their e-mail address in the article if they wish so.
  #24  
Old August 27th 07, 03:38 PM posted to sci.astro.fits
Michael Williams
external usenet poster
 
Posts: 6
Default [fitsbits] Question(s) regarding development ofproprietary FITS manipulation software. . .

Ok, I actually asked a follow-on question in another post/e-mail.
Basically I asked if these astronomers images were literally live
images (perhaps of the stars, or nieces and nephews), or if they were
simply arbitrary graphical representations of data that had no
reference in reality. Also, given the fact that, as you said,
reverse transformation isn't always possible, why in the world does
it matter if the user can perform image manipulations of any type? I
mean, if there is no consistent way to produce an image, then why do
we care?

Thanks,
Michael


On Aug 27, 2007, at 5:15 AM, Thierry Forveille wrote:

gberz3 a écrit :
Why would *anyone* present FITS data as images if they are
not image data? Why not represent it as sound?
I guess that's what I'm getting at. What relevance does an image
have
to actual FITS data if there is no "attached" image, and what is the
proper means by which to display said image?

The issue is mostly your notion of an image, vs an astronomer's
notion of the same. To you an image is something that can be
univocally displayed on a screen or printed, while to
astronomers it is a set of values (ideally in physical units,
such as Watts per square meter per steradian) sampled on a
regular grid. There is some link between the two notions,
but not a unique one: an astronomer's image can be displayed
but not in one unique way, and the reverse transformation
if/when at all possible, requires additional information
(e.g. the physical values for a subset of the pixels) and
significant calibration work. Essentially, an astromer's
image is a richer dataset, and someone needs to decide how
to degrade that information to what a display can show.

In addition, images (in the astronomer's definition) is only
one of the data classes that can be stored in FITS.



  #25  
Old August 27th 07, 04:12 PM posted to sci.astro.fits
Thierry Forveille
external usenet poster
 
Posts: 14
Default [fitsbits] Question(s) regarding developmentof proprietary FITS manipulation software. . .

Michael Williams a écrit :
Basically I asked if these astronomers images were literally live
images (perhaps of the stars, or nieces and nephews), or if they were
simply arbitrary graphical representations of data that had no
reference in reality.

Please read what has actually been written, by others as well as
by myself, rather than your preconceptions: the images are not
"graphical representations of data", they are the data themselves.
Whether those data have any reference in reality or not is a
philosophical issue, and irrelevant for all practical display
purposes that I can see.

Also, given the fact that, as you said,
reverse transformation isn't always possible, why in the world does
it matter if the user can perform image manipulations of any type? I
mean, if there is no consistent way to produce an image, then why do
we care?

Well, that's very much a question for you, which has been
repeatedly asked on the forum and which you never answered:
why do YOU care about FITS images, and what do YOU want to
do with them?

We've all tried to explain what they actually are, but without
any productive reaction on your side I've personally reached
a point where the only useful suggestion I can have is that
you attend/read an Astronomy 101/Philosophy of Science 101
class/book.
  #26  
Old August 27th 07, 04:32 PM posted to sci.astro.fits
Michael Williams
external usenet poster
 
Posts: 6
Default [fitsbits] Question(s) regarding developmentof proprietary FITS manipulation software. . .

Thierry,

I certainly appreciate your input, and apologize for the frustration,
but we're obviously speaking terribly different languages here. I am
a software developer; nothing more, nothing less. I have a client
that is interested in opening and editing FITS files. We've been
provided with nothing more than FITS example files, a FITS
specification, and a few line items like "the ability to view a
histogram". You're right, a class in astronomy might certainly be
helpful, but you have certainly *not* answered my questions. You've
answered my questions in the same manner you say I'm asking them;
according to your preconception of what you *think* I should be asking.

You obviously have a good grasp of FITS. I simply need that
knowledge translated. All of these techno-philosophical answers do
nothing for a software developer. Again, I am not a scientist. I
asked other questions in a different post. Perhaps you can answer
them. My apologies for re-posts. Feel free to only answer the ones
that don't try your patience:


1) Is a FITS file really similar to, say, a Microsoft Word
document. . .in that it can store *any* kind of data (images, mp3s,
pdfs, minivans) with the only thing that can be counted on being the
"standard" header? It seems that my problem is the fact that I'm
expecting FITS to actually *fit* (pun intended) into some specific
category with absolutely specific syntactical and semantic requirements.


2) What is the significance of *any* imagery in FITS? Are the images
literally live pictures that were taken, or some arbitrary graphical
representation of data?


3) How do I determine what type of image manipulations are legitimate
for any type of image data? Should I allow sepia toning? Should I
allow them to run photoshop filters on the pictures? I mean, what,
really is the usefulness of any manipulation on the image and what
are the most common techniques?


4) What is the goal of scientists when it comes to examining
another's FITS files? What kind of "information" is truly gathered
from the "data"?


Regards,
Michael



On Aug 27, 2007, at 11:12 AM, Thierry Forveille wrote:

Michael Williams a écrit :
Basically I asked if these astronomers images were literally live
images (perhaps of the stars, or nieces and nephews), or if they
were simply arbitrary graphical representations of data that had
no reference in reality.

Please read what has actually been written, by others as well as
by myself, rather than your preconceptions: the images are not
"graphical representations of data", they are the data themselves.
Whether those data have any reference in reality or not is a
philosophical issue, and irrelevant for all practical display
purposes that I can see.

Also, given the fact that, as you said, reverse transformation
isn't always possible, why in the world does it matter if the
user can perform image manipulations of any type? I mean, if
there is no consistent way to produce an image, then why do we care?

Well, that's very much a question for you, which has been
repeatedly asked on the forum and which you never answered:
why do YOU care about FITS images, and what do YOU want to
do with them?

We've all tried to explain what they actually are, but without
any productive reaction on your side I've personally reached
a point where the only useful suggestion I can have is that
you attend/read an Astronomy 101/Philosophy of Science 101
class/book.



  #27  
Old August 27th 07, 04:58 PM posted to sci.astro.fits
Thierry Forveille
external usenet poster
 
Posts: 14
Default [fitsbits] Question(s) regarding developmentof proprietary FITS manipulation software. . .

Michael Williams a écrit :
I certainly appreciate your input, and apologize for the frustration,
but we're obviously speaking terribly different languages here. I am a
software developer; nothing more, nothing less. I have a client that
is interested in opening and editing FITS files. We've been provided
with nothing more than FITS example files, a FITS specification, and a
few line items like "the ability to view a histogram".

Well, then you need to ask your customer what THEY want. FITSBITS is
a helpful forum, but you need to realize that we are all busy
astronomers or/and software engineers, and that our time is valuable:
we are all willing to steer you towards helpful documentation,
but I'd be surprised if anyone on the list has the time to either
write the specifications that your customer apparently doesn't
provide or to fill in your science education to the point where
you can write them.

  #28  
Old August 27th 07, 06:10 PM posted to sci.astro.fits
Steve Allen
external usenet poster
 
Posts: 37
Default [fitsbits] Question(s) regarding development of proprietaryFITS manipulation software. . .

On Mon 2007-08-27T11:32:19 -0400, Michael Williams hath writ:
we're obviously speaking terribly different languages here.


We are also not managing to communicate that FITS has been in use
for nearly 30 years. It does work for exchanging pictures
of the sky, catalogs of the sky, and other related information.

In most cases the data transmitted via FITS are still tightly coupled
with the peculiar characteristics of the instruments which were used
to acquire them. In astronomy those instruments literally cover a
very wide spectrum -- from the detection of the weak electric field
propagated with ensembles of radio photons to the almost audible ping
when individual gamma ray photons hit their detector.

Over those 30 years there have probably been thousands of person-years
of effort developing the software which interprets the information in
the FITS files. That may be a major point of disconnect here.

1) Is a FITS file really similar to, say, a Microsoft Word
document. . .in that it can store *any* kind of data (images, mp3s,
pdfs, minivans) with the only thing that can be counted on being the
"standard" header?


There is an IRAF package which does exactly that. It serves a purpose
akin to tar when feeding ensembles of files into a pipeline which is
designed to handle FITS.

Are the images
literally live pictures that were taken, or some arbitrary graphical
representation of data?


Yes, but in optical astronomy the first notion is usually a safe one.

3) How do I determine what type of image manipulations are legitimate
for any type of image data? Should I allow sepia toning? Should I
allow them to run photoshop filters on the pictures? I mean, what,
really is the usefulness of any manipulation on the image and what
are the most common techniques?


In the most general sense the answers to these questions count as the
fulfillment of the requirements for PhD theses and refereed scientific
literature.
"If we knew what we were doing it wouldn't be research."

4) What is the goal of scientists when it comes to examining
another's FITS files? What kind of "information" is truly gathered
from the "data"?


This is science.

--
Steve Allen WGS-84 (GPS)
UCO/Lick Observatory Natural Sciences II, Room 165 Lat +36.99855
University of California Voice: +1 831 459 3046 Lng -122.06015
Santa Cruz, CA 95064 http://www.ucolick.org/~sla/ Hgt +250 m
  #29  
Old August 27th 07, 06:28 PM posted to sci.astro.fits
LC's NoSpam Newsreading account[_2_]
external usenet poster
 
Posts: 23
Default [fitsbits] Question(s) regarding development of proprietary FITSmanipulation software. . .

On Mon, 27 Aug 2007, Michael Williams wrote:

1) Is a FITS file really similar to, say, a Microsoft Word document.
. .in that it can store *any* kind of data


Well, most dead astronomers will revolve in their grave hearing a
comparison of FITS with Microsoft Word (fortunately most FITS people
are kicking and alive, and they will complain too), but in a sense yes,
a FITS file can be a generic container. It is used as a container for

- sky images [i]
- detector images [i]
- any other sort of image data [i]
- astronomical catalogues [A,B]
- astronomical spectra [B,I]
- time series [b]
- photon lists [b]
- detector response matrices [b]
- any tabular data (typically an instrument calibration data) [b]
- or even an index to a collection of other FITS files (e.g. the
XMM calibration index file) [b]

These are all real examples I've encountered. [i],[A],[b] means they can
be images, ascii tables, binary tables. See my former posting.

2) What is the significance of *any* imagery in FITS? Are the images
literally live pictures that were taken, or some arbitrary graphical
representation of data?


Most images are derived somehow from raw images. When not, ok they may
fit in your category of "arbitrary graphical representation" ... if I
put in a FITS image the value of chi-square vs spectral index on the x
axis and hydrogen column density on the y axis.

Raw images are some form of intensity (roughly proportional to light
intensity or number of photons) collected on some detector (a CCD, a
photographic plate or whatsoever) as function of pixel xy position.

In most cases this corresponds to an actual picture of the sky in some
band but calibration of intensity of even xy can be tricky.

But in other cases what is on the detector comes from the sky but is
something else (e.g. a collection of spectra, see e.g.
http://www.oamp.fr/virmos/vvds.htm)

3) How do I determine what type of image manipulations are legitimate
for any type of image data? Should I allow sepia toning? Should I
allow them to run photoshop filters on the pictures?


I would say in general no. You can, perhaps do that to make a figure
(the GIMP reads FITS files), but in general the typical FITS image
display (ds9) will first scale the image data somehow (they are data in
physical units !!). And most programs do analysis without display.

Ask your client what he wants !!!

4) What is the goal of scientists when it comes to examining another's FITS
files? What kind of "information" is truly gathered from the "data"?


Too long to explain ... have to go home now


On Aug 27, 2007, at 11:12 AM, Thierry Forveille wrote:


you attend/read an Astronomy 101/Philosophy of Science 101


Whatt's this 101 ? Dalmatian dogs ? :-)


--
----------------------------------------------------------------------
is a newsreading account used by more persons to
avoid unwanted spam. Any mail returning to this address will be rejected.
Users can disclose their e-mail address in the article if they wish so.
  #30  
Old August 27th 07, 07:45 PM posted to sci.astro.fits
Rob Seaman
external usenet poster
 
Posts: 49
Default [fitsbits] Question(s) regarding developmentof proprietary FITS manipulation software. . .

On Aug 27, 2007, at 8:32 AM, Michael Williams wrote:

I have a client that is interested in opening and editing FITS
files. We've been provided with nothing more than FITS example
files, a FITS specification, and a few line items like "the ability
to view a histogram".


Perhaps your issue is more with your client's fuzzy description of
the problem? Since you mention a histogram I might guess that the
application is specific to pixel arrays rather than tables?

Again, I am not a scientist.


Perhaps solving your client's problem requires one?

It seems that my problem is the fact that I'm expecting FITS to
actually *fit* (pun intended) into some specific category with
absolutely specific syntactical and semantic requirements.


No, as you say, FITS won't do that. If your application is specific
to pixel arrays, however, the range of categories is much narrower.

2) What is the significance of *any* imagery in FITS? Are the
images literally live pictures that were taken, or some arbitrary
graphical representation of data?


All computer representations are arbitrary. A CCD exposure of the
sky represented as FITS is, however, as "living" a picture as
treasured family snapshots in an album or socially ruinous JPEGs on
Facebook.

To attempt to answer the question I think you think you're asking,
however, unlike a GIF or JPEG, a FITS pixel array contains no
explicit lookup tables conveying a false color mapping and no
explicit color model conveying "true" color.

Should I allow sepia toning?


I wouldn't forbid it, but no, don't spend much effort supporting such
a feature.

Should I allow them to run photoshop filters on the pictures?


Does your client's application have anything to do with publication
graphics?

I mean, what, really is the usefulness of any manipulation on the
image and


Each pixel represents a measurement. Each array represents
gazillions of such measurements. Measurements require calibration to
be scientifically useful. Calibrated data are compared with other
calibrated data to deduce assertions pertaining to empirical reality.

I'm not being facetious in asking whether your clients care about
empirical reality. If not, perhaps they care about FITS so they can
list FITS compliance as a feature on a shrink wrapped box?

what are the most common techniques?


Coaddition - by all means, coaddition. Oh - and centroiding! Ooh!
Principle components analysis - not common now, but worth a passel of
pork belly futures, mark my words.

My daughter's camera has "face recognition" to decide what part of
the picture to focus against. If your clients are involved in
planetary imaging this could aid in detecting ROIs resembling Elvis.
(Hmmm - another VO web service to explore...)

4) What is the goal of scientists when it comes to examining
another's FITS files?


The same thing we do every night, Pinky - try to take over the world!

Rob

 




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Custom Software Development VB Amateur Astronomy 1 June 9th 07 04:18 PM
case tool for guidance software development Ananth Space Shuttle 0 May 3rd 06 12:03 PM
Ann: Fits Previewer Software for Windows Eddie Trimarchi CCD Imaging 0 January 13th 04 04:07 AM
NASA Honors Agency Software Development Ron Baalke Space Shuttle 0 September 2nd 03 08:31 PM
NASA Honors Agency Software Development Ron Baalke Space Station 0 September 2nd 03 08:31 PM


All times are GMT +1. The time now is 07:01 AM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 SpaceBanter.com.
The comments are property of their posters.