|
|
|
Thread Tools | Display Modes |
#1
|
|||
|
|||
[fitsbits] TDIMn
I noticed that the WCS Paper III mentions that there's a pending motion before
the FITS committees to make the TDIMn keyword official. I was wondering what the status of that motion was. Actually, I would have thought that TDIMn became an official keyword when the IAU-FWG approved WCS Papers I and II, but I can see the value of making it official independent of the WCS formalism. William Thompson |
#2
|
|||
|
|||
On Tue 2004/08/24 18:15:28 -0400, William Thompson wrote in a message to: Actually, I would have thought that TDIMn became an official keyword when the IAU-FWG approved WCS Papers I and II, but I can see the value of making it official independent of the WCS formalism. Paper IV needs this (Appendix B of Cotton, Tody, & Pence, 1995) and also the "substring array" convention (Appendix C). It would be good to have them, and Appendix A, properly formalized now. Mark Calabretta ATNF |
#3
|
|||
|
|||
William Thompson wrote:
I noticed that the WCS Paper III mentions that there's a pending motion before the FITS committees to make the TDIMn keyword official. I was wondering what the status of that motion was. This motion was deferred until the FITS committees were reorganized earlier this year and until after the recent consideration of the FITS MIME type and the Australia/New Zealand FITS committee proposals. Now that that work is out of the way, one of the next orders of business for the FITS committees should be to consider this motion. This motion proposes to incorporate Appendix B.1 and B.2 into the official FITS Standard (the appendices are currently unofficial). Both appendices apply to FITS binary tables: B.1 describes the Variable Length Array facility, and B.2 describes the TDIMn keyword. (See http://archive.stsci.edu/fits/fits_standard/node74.html). There is also a third appendix, B.3, dealing with substring arrays, but only a portion of this proposed convention has ever been used or implemented (to my knowledge), so only part of this appendix, if any, should be considered for incorporation into the FITS Standard. If anyone has any issues or concerns about making these appendices part of the official FITS format Standard, now would be a good time to raise them here on this newsgroup. The FITS committees will likely be considering this issue soon, later this year. Bill Pence -- __________________________________________________ __________________ Dr. William Pence NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice) Greenbelt MD 20771 +1-301-286-1684 (fax) |
#4
|
|||
|
|||
On Wednesday 25 August 2004 19:04, William Pence wrote:
This motion proposes to incorporate Appendix B.1 and B.2 into the official FITS Standard *(the appendices are currently unofficial). *Both appendices apply to FITS binary tables: B.1 describes the Variable Length Array facility, and B.2 describes the TDIMn keyword. (See http://archive.stsci.edu/fits/fits_standard/node74.html). *There is also a third appendix, B.3, dealing with substring arrays, but only a portion of this proposed convention has ever been used or implemented (to my knowledge), so only part of this appendix, if any, should be considered for incorporation into the FITS Standard. If anyone has any issues or concerns about making these appendices part of the official FITS format Standard, now would be a good time to raise them here on this newsgroup. *The FITS committees will likely be considering this issue soon, later this year. I see no problems with the B.2 (TDIMn) proposal as it only deals with the interpretation of a one dimensional array. Certainly it would be good to have the appendix B.2 included officially into the standard soon. With respect to B.1 (variable length arrays), I have some reservations. This proposal requires a full implementation of the HEAP and the P column format. Technically, the B.1 proposal is good and I would endorse it as such. The issue is that (as least to my taste) FITS files which conform to the standard should be readable by most readers. The success of FITS is to some extend based on this, namely that one having a standard FITS file does not need to worry - it is, by default, readable by almost all systems. If B.1 is included into the standard I am concerned that a significant number of systems will be unable to read and access such data. It is not just a matter of if HEAPs can be accessed by software libraries but more if full data processing systems can access such data as if they had been in a 'normal' table column. Beside leaving B.1 as it is, there may the following options to include it into the standard: 1) Adding an clear, explicit preample which states that users cannot assume the HEAP concept to be implemented by all readers and therefore should be cautious in using it. 2) Verify that most readers actually already implements the HEAP which would satisfy my concern. On B.3, it would be good to know who is actually using it? One should only include, into the standard, features which are used by a broad community. Preben Grosbol |
#5
|
|||
|
|||
I share much of Preben's uncertainties about adopting the appendices.
The TDIMn (B.2) is absolutely needed and is or soon will be used. It is also a convention that can be ignored by general readers that do not need to know the internal details of a table column's data. It uses a previously non-standard keyword and so has been ignored by general readers. The substring convention (B.3) is also relatively harmless in that a general routine may ignore all the substring notation. However, that notation appears as part of the data value of a standard keyword and so may (should??) confuse a general reader that does not support the extension. Has this been used by anyone? The heap convention however does worry me. It changes the data structure in the FITS file and I expect that general FITS readers will not normally support the convention internally. In fact, I have always regarded this convention with the worry that it encourages a lack of discipline in designing the data tables in the first place. Thus an over-worked designer may decide to dump all sorts of data into a single table piling most of the "table" data into an unstructured heap, rather than building two or more cleanly structured tables separating unlike data into separate columns/tables. I do not doubt that the heap may be an excellent choice for an internal format in a data system geared to handle it. But as a communication-of-data choice I expect that it will not be implemented generally and hence stand in the way of communication. Eric Greisen |
#6
|
|||
|
|||
The NASA/ESA SOHO project has been using TDIMn since 1995. I strongly support
adopting TDIMn officially. The IDL software I wrote many years ago for binary tables does support variable length arrays, but I'm not aware of any projects within the solar community actually using it. I can ask around. I'm not aware of any projects using the substring array convention, and am not sure how well it's supported. My own software does not explicitely support it, but wouldn't be confused by its presence. Bill Thompson Eric Greisen wrote: I share much of Preben's uncertainties about adopting the appendices. The TDIMn (B.2) is absolutely needed and is or soon will be used. It is also a convention that can be ignored by general readers that do not need to know the internal details of a table column's data. It uses a previously non-standard keyword and so has been ignored by general readers. The substring convention (B.3) is also relatively harmless in that a general routine may ignore all the substring notation. However, that notation appears as part of the data value of a standard keyword and so may (should??) confuse a general reader that does not support the extension. Has this been used by anyone? The heap convention however does worry me. It changes the data structure in the FITS file and I expect that general FITS readers will not normally support the convention internally. In fact, I have always regarded this convention with the worry that it encourages a lack of discipline in designing the data tables in the first place. Thus an over-worked designer may decide to dump all sorts of data into a single table piling most of the "table" data into an unstructured heap, rather than building two or more cleanly structured tables separating unlike data into separate columns/tables. I do not doubt that the heap may be an excellent choice for an internal format in a data system geared to handle it. But as a communication-of-data choice I expect that it will not be implemented generally and hence stand in the way of communication. Eric Greisen |
#7
|
|||
|
|||
Although I can't name a list of sites or packages from memory, my impression
has been that the variable length array facility is used quite a lot. One can't always allocate a fixed size array for data arrays, in which case there is a real need for the variable length array facility. In a table containing such columns a reader which does not support variable length arrays should still be able to read the main table, which has fixed length records, although it would not be able to read the variable length arrays. Doug From Preben: With respect to B.1 (variable length arrays), I have some reservations. This proposal requires a full implementation of the HEAP and the P column format. Technically, the B.1 proposal is good and I would endorse it as such. The issue is that (as least to my taste) FITS files which conform to the standard should be readable by most readers. The success of FITS is to some extend based on this, namely that one having a standard FITS file does not need to worry - it is, by default, readable by almost all systems. If B.1 is included into the standard I am concerned that a significant number of systems will be unable to read and access such data. It is not just a matter of if HEAPs can be accessed by software libraries but more if full data processing systems can access such data as if they had been in a 'normal' table column. Beside leaving B.1 as it is, there may the following options to include it into the standard: 1) Adding an clear, explicit preample which states that users cannot assume the HEAP concept to be implemented by all readers and therefore should be cautious in using it. 2) Verify that most readers actually already implements the HEAP which would satisfy my concern. On B.3, it would be good to know who is actually using it? One should only include, into the standard, features which are used by a broad community. |
#8
|
|||
|
|||
Preben Grosbol wrote:
.... With respect to B.1 (variable length arrays), I have some reservations. This proposal requires a full implementation of the HEAP and the P column format. Technically, the B.1 proposal is good and I would endorse it as such. The issue is that (as least to my taste) FITS files which conform to the standard should be readable by most readers. The success of FITS is to some extend based on this, namely that one having a standard FITS file does not need to worry - it is, by default, readable by almost all systems. If B.1 is included into the standard I am concerned that a significant number of systems will be unable to read and access such data. It is not just a matter of if HEAPs can be accessed by software libraries but more if full data processing systems can access such data as if they had been in a 'normal' table column. Beside leaving B.1 as it is, there may the following options to include it into the standard: 1) Adding an clear, explicit preample which states that users cannot assume the HEAP concept to be implemented by all readers and therefore should be cautious in using it. 2) Verify that most readers actually already implements the HEAP which would satisfy my concern. The table at http://fits.gsfc.nasa.gov/fits_libraries.html suggests that most FITS libraries that support binary tables do support variable length arrays. Preben's comments could have been made about binary tables (or even more strongly ASCII tables) which are not supported by substantial numbers of FITS readers. Indeed, if one were to look at the distinct software implementations of FITS readers I suspect that the vast majority are simple readers that handle only primary FITS images. (I've written many such readers over the past couple of decades for a variety of special projects). However when we look at the libraries that are being used in the development of new major software, then I think that it is clear that both binary tables and variable length arrays are widely supported and used. That said I'm not sure I agree with Preben's premise that putting something in the standard implies a mandate that FITS readers should be able to handle it. Certainly the many FITS readers that only handle images will continue to ignore binary tables regardless of whether they have variable length records, and FITS readers that wish to ignore heap data in binary tables should feel free to do so. In a similar vein, the new WCS papers suggest some quite sophisticated and complex ways that coordinate information can be encoded. I expect that the vast majority of readers will support only elements of the WCS that are necessary to the kinds of science they are intended to support rather than providing explicit support for all of the WCS features. Regards, Tom McGlynn On B.3, it would be good to know who is actually using it? One should only include, into the standard, features which are used by a broad community. Preben Grosbol |
#9
|
|||
|
|||
In response to questions about the extent to which the FITS BINTABLE
conventions (defined in Appendix B of the NOST FITS Standard) have been used, here are the cases that I'm aware of. Please let me know of any other examples where these conventions have been used. B.1 "Variable Length Array" facility ------------------------------------ This convention is supported by many FITS libraries (see http://fits.gsfc.nasa.gov/fits_libraries.html) and has been used for the past decade in a number of publicly distributed FITS data sets, including: - The Copernicus satellite (OAO-3) spectral data, distributed by the Multimission Archive at Space Telescope (MAST). - The standard Response Matrix calibration files (RMF) that are used by almost all X-ray satellite missions since the mid 1990s. - Arecibo Observatory radio telescope data. The format of these data files is based on the SDFITS convention for interchange of single dish data in radio astronomy which explicitly supports the variable length array convention. - The IRAF data analysis software uses the variable length array convention when storing 'Pixel Lists' in FITS format. In addition, the 'Tiled' image compression convention (see http://heasarc.gsfc.nasa.gov/docs/he...ess_image.html) is layered on top of the variable length array convention and is currently used by several groups, including: - Mosaic CCD camera data taken at the Canada-France-Hawaii Telescope (CFHT) and publicly distributed by the CADC. - The CCD image archive taken for the Optical Gravitational Lensing Experiment (OGLE) using the 1.3m Warsaw telescope at Las Campanas Observatory, Chile. B.2 The TDIMn keyword convention --------------------------------- This convention is widely supported by many software packages and is used in many FITS data sets, too numerous to mention individually. B.3 The "Substing Array" convention ------------------------------------ I'm not aware of any software packages or any FITS data files that support or use the full form of this proposed convention, as in these examples: TFORMn = '30A:SSTR10' / fixed length 10-char substrings TFORMn = '30A:SSTR10/032' / variable-length substrings The CFITSIO library does support the simpler convention for specifying fixed-length substrings, e.g., TFORMn = '30A10' however I don't know of any FITS files that have actually used this convention. -Bill Pence -- __________________________________________________ __________________ Dr. William Pence NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice) Greenbelt MD 20771 +1-301-286-1684 (fax) |
#10
|
|||
|
|||
William Pence writes:
- Mosaic CCD camera data taken at the Canada-France-Hawaii Telescope (CFHT) and publicly distributed by the CADC. ??? As far as I know, mosaic images taken at CFHT are distributed as multiple IMAGE extensions (one extension per CCD). It is of course possible that the CADC repackages them in a new format, but that would be new to me. Do you have a pointer to where you got this information from? |
|
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
[fitsbits] New draft of WCS Paper IV | Mark Calabretta | FITS | 0 | April 27th 04 05:20 AM |
[fitsbits] WCS Paper III | Eric Greisen | FITS | 2 | March 17th 04 09:24 PM |
[fitsbits] Dataset identifications. | Thomas McGlynn | FITS | 0 | March 10th 04 07:20 PM |
[fitsbits] On-line FITS-WCS visualisation | David Berry | FITS | 0 | March 4th 04 03:36 PM |
[fitsbits] (fwd) Windows Explorer Fits Viewer | Eddie Trimarchi | FITS | 0 | February 24th 04 06:46 PM |