A Space & astronomy forum. SpaceBanter.com

Go Back   Home » SpaceBanter.com forum » Astronomy and Astrophysics » FITS
Site Map Home Authors List Search Today's Posts Mark Forums Read Web Partners

[fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod



 
 
Thread Tools Display Modes
  #1  
Old September 29th 06, 10:14 PM posted to sci.astro.fits
William Pence
external usenet poster
 
Posts: 66
Default [fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod

This is to announce the start of the Public Comment Period on the "Foreign
File Encapsulation" FITS convention that has been submitted by the IRAF
group at NOAO for inclusion in the "Registry of FITS Conventions" that is
maintained by the IAU FITS Working Group.

A document describing this convention, and a sample FITS file that uses
it, are available for public review and comment from the FITS registry web
page at

http://fits.gsfc.nasa.gov/fits_registry.html

This convention defines a new FITS extension type with

XTENSION= 'FOREIGN '

as a means of putting a FITS format wrapper about an arbitrary file. This
convention enables a file or tree of files to be wrapped up in a FITS file
and later restored in the original form to disk. A set of optional
keywords are defined to store information about the encapsulated file. An
example of the full set of header keywords is shown below:

XTENSION= 'FOREIGN ' / NOAO xtension type
BITPIX = 8 / Bits per pixel (byte)
NAXIS = 0 / No Image matrix
PCOUNT = 10756 / File size in bytes
GCOUNT = 1 / One group
EXTNAME = 'kp-B-pgr_120.png' / Filename
EXTVER = 1
EXTLEVEL= 1 / Directory level
FG_GROUP= 'kp-ftrB-dps-mdppgr' / Group Name
FG_FNAME= 'kp-B-pgr_120.png' / Filename
FG_FTYPE= 'binary ' / File type
FG_LEVEL= 1 / Directory level
FG_FSIZE= 10756 / Data size (bytes)
FG_FMODE= '-rw-r--r--' / File mode
FG_FUOWN= 'smith ' / File UID
FG_FUGRP= 'jones ' / File GID
FG_CTIME= '2006-06-15T02:32:49' / file ctime (GMT)
FG_MTIME= '2006-06-15T02:32:49' / file mtime (GMT)
END

This Public Comment Period will remain open for 30 days. Comments may be
posted here on the FITSBITS mail exploder or the sci.astro.fits newsgroup.
Minor typographical issues may be sent directly to the submitters of the
convention.

Bill Pence
(on behalf of the IAU FITS Working Group)
--
__________________________________________________ __________________
Dr. William Pence
NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice)
Greenbelt MD 20771 +1-301-286-1684 (fax)
  #2  
Old October 3rd 06, 08:00 AM posted to sci.astro.fits
Peter Bunclark
external usenet poster
 
Posts: 6
Default [fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod

William Pence wrote:
This is to announce the start of the Public Comment Period on the "Foreign
File Encapsulation" FITS convention that has been submitted by the IRAF
group at NOAO for inclusion in the "Registry of FITS Conventions" that is
maintained by the IAU FITS Working Group.



FG_FMODE
The file mode as a string ("rwx-rwx-rwx", bits not set given as
"-").


x ? Why would a file be executable? In the tradition of the
omni-platform transportability of FITS files, it is unlikely that
a file would be really executable. If the executable was intended
to go onto a machine of the same architecture (and libc....) then
why use FITS?

An example of an executable file, however, would be a virus.
Surely we don't want to go there?

Peter.
  #3  
Old October 3rd 06, 06:54 PM posted to sci.astro.fits
[email protected]
external usenet poster
 
Posts: 1
Default Start of "Foreign File Encapsulation" Public Comment Period


Peter Bunclark wrote:

x ? Why would a file be executable? In the tradition of the
omni-platform transportability of FITS files, it is unlikely that
a file would be really executable. If the executable was intended
to go onto a machine of the same architecture (and libc....) then
why use FITS?

An example of an executable file, however, would be a virus.


Another example would be a shell-script (requiring the execute bits),
bash/tcsh are pretty ubiquitous and portable. The extension is meant
to capture any foreign blob, this could include say a perl script used
as a pipeline recipe but stored in a foreign extension because the
archive
saving it is tuned to FITS files. Compiled binaries are allowed as
well,
but you'd be nuts to save them in this way.

-Mike

  #4  
Old October 4th 06, 04:43 PM posted to sci.astro.fits
Rob Seaman
external usenet poster
 
Posts: 49
Default [fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod

Hi Pete,

Why would a file be executable? In the tradition of the omni-
platform transportability of FITS files, it is unlikely that a file
would be really executable.


Just to provide some context, foreign file encapsulation was created
as part of a prior NOAO archive project several years ago to
encapsulate various non-FITS metadata products generated on Kitt Peak
– observing logs, weather pictures, etc. NOAO is using it again to
transport PNG files from the Mosaic camera pipeline to the next
generation NSA archive. It was designed to be very general purpose,
hence the authors' interest in reminding the community of its
existence. You can certainly create file collections that a third
party would be foolish to unpack on their own system. The same is
true of tar. In any event, if the fgread program isn't setuid, such
files would only be executable as the UID of user, containing any risk.

Nelson Zárate will have an ADASS poster, and perhaps we can expect
foreign encapsulation and the general topic of FITS extension types
to enliven the FITS BoF.

If the executable was intended to go onto a machine of the same
architecture (and libc....) then why use FITS?


We've long since moved past the point of regarding FITS as a pure
transport format. It's used in all our archives, for one thing. One
might well want to package up processing scripts with the data they
were applied to, for instance.

An example of an executable file, however, would be a virus.
Surely we don't want to go there?


One person's virus is another's intelligent agent. I suspect that a
lot of us have entertained the notion of an IRAF or IDL virus in the
past. Why not FITS? VO will certainly have its hands full with XML
and web services, many of whose features are intended to be very
"intelligent" indeed.

I suspect that foreign encapsulation is a sufficiently narrow niche
market that we need not fret too much.

Rob


  #5  
Old October 9th 06, 04:03 PM posted to sci.astro.fits
LC's NoSpam Newsreading account
external usenet poster
 
Posts: 3
Default [fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod

I have been following the discussion on the FOREIGN extension for quite
a bit, but found no time so far to communicate my views.

My opinions fall in three different categories :

a) procedural comments

There was some confusion about the fact that a thing like the FOREIGN
extension should go in the convention registry, and how. They are
probably due to the fact here we are dealing with a "conforming
extension" and not a mere convention on keyword or column naming

This was first pointed out on Mon, 2 Oct 2006 by Thomas McGlynn

The foreign file encapsulation doesn't really fit this framework.
It describes an entirely new type of FITS HDU. In the past such
a new type would require an amendment to the FITS standard.


(which is not really the case ... a new extension has just to be
registered IN A SEPARATE REGISTRY, it is not necessarily part of the
standard)

Also on Mon, 2 Oct 2006, William Pence wrote:

The basic requirements for a convention to be included in the Registry
a

1 It should conform to the FITS Standard.


So I guess that until XTENSION='FOREIGN' is not registered in the
registry of conforming extensions, it does not conform, and cannot
be registered in the registry of conventions.

AFAIK, things are dealt with in the IAU-FWG, so this is not a real
issue

I also agree with Bill when he says

From my own perspective I would hope that the number of new extension
types that are created can be kept to a minimum because of the impact
that it has on existing software and the added confusion that it can
cause for FITS users.


I will continue this sort of discussion on the IAUFWG list. For what
the comments requested on FITSBITS about a new convention, they
should properly deal ONLY with the fact the documentation is complete,
well written, unambiguous, not redundant etc.

b) formal comments

So we are not really entitled to say anything about a particular
implementation of a (local) convention. If the authors use it, they'd
have good reasons to do so.

So the only really formal comments are those by Thomas McGlynn about
some undefined acronyms, and about the fact that section 6 is part of an
internal specification document and not relevant for the registry

I'm not sure I understand what the acronyms MEF, FITS-MEF and EHU mean
in this proposal. They never seem to be defined.


Section 6 seems to be irrelevant to the proposal.



c) substantial comments

Now I'm a bit perplexed by a number of other items which have been
pointed out by some participants to the discussion :

On Mon, 2 Oct 2006, Thomas McGlynn wrote:

The encoding of directories (and symbolic links) is unclear.
Directories are not portable across systems.


The potential system dependencies of directory structures/file names
are not discussed. E.g., is c:\x\y\z\ a valid directory name? How is
a absolute file path written on a Unix machine to be read on a Windows

[...]
What happens when we go from a case sensitive to a case insensitive

[...]

I don't think Windows supports symbolic links either.


On the same date William Thompson wrote:

7. I'm not sure I understand the point of encoding the user and group
ID of the file. Surely that's platform specific.


And finally on Tue, 3 Oct 2006, Peter Bunclark wrote:

The file mode as a string ("rwx-rwx-rwx", bits not set given as "-").

x ? Why would a file be executable? In the tradition of the
omni-platform transportability of FITS files, it is unlikely [...]



As gut feeling I tend to AGREE with all above comments (and the rest by
Thomas McGlynn), which can be grouped as "the proposal has clearly in
mind Unix as target system".

For instance I could detail on Peter Bunclark's comment more than
concerning with the execute bit being set, by the fact that a "file
mode" made of 3 triplets rwxrwxrwx (NOT rwx-rwx-rwx !) for user, group
and other is also NOT SYSTEM INDEPENDENT. If I recall well for instance
VMS had 4 quadruplets RWED for user, group, world and system. I confess
I ignore the Windows protection scheme.

These are technical comments.

To these one might add that, if the origin and target systems are both
Unix systems I fail to see the need to devise such a GENERAL convention
in lieu of using a well established one like tar files (I may see the
need of a simpler convention to append a single foreign file to a FITS
file, like a PNG preview). This is a philosophical comment.

BUT WE ARE NOT ENTITLED TO MAKE SUCH COMMENTS WHEN DISCUSSING A
CONVENTION TO GO IN THE CONVENTION REGISTRY.

We are not even entitled to ask the authors of such a convention to
change anything (keyword names or contents) they already use. We can
only say whether it is well documented.

Then, if NOAO has its own reasons to use such convention to move files
around their site, they are in full right to do so and we ARE NOT
ENTITLED to say anything like all the above comment items.


I believe it would be different if the FOREIGN extension had to be
accepted as a STANDARD extension. ONOY in such case I believe we could
question the need to "replace tar" and include a "directory structure"
into a "FITS FOREIGN" file, and we SHALL question the conventions used
to store things like file names, case sensitivity, uid and gid,
permissions in a system independent way, maybe requiring some limitation
on character set, syntax etc. I actually wonder if all these sort of
things have not already been standardised in the way CDs are written by
utilities like makeisofs / cdrecord.


A minor technicality, when Thomas McGlynn says
I don't think Windows supports symbolic links either.


I do not believe that is true. Windows has at user level some kind of
links providing pointers to a file, which resemble soft links, even VMS
had, at system level, something like that (which resembled more hard
links).

----------------------------------------------------------------------------
Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133 Milano (Italy)
For more info : http://www.iasf-milano.inaf.it/~lucio/personal.html
----------------------------------------------------------------------------

--
----------------------------------------------------------------------
is a newsreading account used by more persons to
avoid unwanted spam. Any mail returning to this address will be rejected.
Users can disclose their e-mail address in the article if they wish so.
  #6  
Old October 10th 06, 03:47 AM posted to sci.astro.fits
Doug Tody
external usenet poster
 
Posts: 19
Default [fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod

Hi All -

Some more quick comments are in order here. First, what we are discussing
is a convention, hence revisiting the design is not appropriate (unless
someone wants to propose moving towards a standard; I probably would
not suggest that myself in this case). Second, on the specific issue of
Unix-oriented concepts in the design, this is not necessarily a problem
for something like we have here. Unix Tar for example, exists in Cygwin
(or Winzip etc.) and clearly has some utility even on such a platform.
Things like mode bits, paths, and case sensitivity do not map exactly but
they map adequately; some information is lost when going to a non-Unix
platform, but the extra refinement is useful on reasonably POSIX systems
such as the Unix-derivatives like Linux or Mac. I suspect that if
we produced a rigorous, explicit model for all this, with an explicit
mapping to each platform, we would not do much better and would only
complicate things. If something like a symbolic link is not mentioned
that means it is not supported due to being too platform specific (I
agree that perhaps it would be better to say this explicitly).

This was a convenient convention for what it was intended for, which
was extending an existing FITS multi-extension system for transporting
a telescope data stream to an archive, without having to redesign the
whole thing to use something other than FITS. If we had to do something
similar today I might prefer to use something more standard like ZIP as
the container, keeping the FITS part simple and focused on single files.
This would solve various problems such as providing a directory for
efficient random access to extract files from the container - probably
today it is no longer useful to attempt to do such a thing in FITS.
But the point here is merely to document existing conventions, so it
is not very useful to revisit such matters, except maybe as we consider
where to take FITS itself in the future.

- Doug
  #7  
Old October 10th 06, 09:32 AM posted to sci.astro.fits
Peter Bunclark
external usenet poster
 
Posts: 6
Default [fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod

Doug Tody wrote:
Hi All -

Some more quick comments are in order here. First, what we are discussing
is a convention, hence revisiting the design is not appropriate (unless
someone wants to propose moving towards a standard; I probably would
not suggest that myself in this case). Second, on the specific issue of
Unix-oriented concepts in the design, this is not necessarily a problem
for something like we have here.

SNIP

- Doug


I understand better from this and preceding comments the essential
difference between a standard proposal and a convention registry. But
one of the strong cases for having a convention registry is that others
with similar needs might adopt an existing convention for their own
purpose, thus short-cutting design and testing, and making a bigger pool
of complying systems.

Thus, although we might not criticise a convention in the same rigorous
was as we would a standards proposal, surely it is apposite to append a
SHORTCOMINGS section where it is admitted that, say, this convention
doesn't generalise very well and that some of its provisions (and I
maintain executable is not a good idea) may not be advisable.

Pete
 




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Free Commodities Are Abused Len Policy 46 December 5th 05 06:21 AM
[fitsbits] Start of the HEALPix "Public Comment Period" William Pence FITS 0 November 15th 05 10:42 PM
[fitsbits] Start of the FITS MIME type Public Comment Period William Pence FITS 8 June 17th 04 06:08 AM
Our Moon as BattleStar Rick Sobie Astronomy Misc 93 February 8th 04 10:31 PM
Something more interesting for you to read! Greg Dortmond UK Astronomy 12 December 22nd 03 05:51 AM


All times are GMT +1. The time now is 09:31 AM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 SpaceBanter.com.
The comments are property of their posters.