|
|
Thread Tools | Display Modes |
#1
|
|||
|
|||
[fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod
This is to announce the start of the Public Comment Period on the "Foreign
File Encapsulation" FITS convention that has been submitted by the IRAF group at NOAO for inclusion in the "Registry of FITS Conventions" that is maintained by the IAU FITS Working Group. A document describing this convention, and a sample FITS file that uses it, are available for public review and comment from the FITS registry web page at http://fits.gsfc.nasa.gov/fits_registry.html This convention defines a new FITS extension type with XTENSION= 'FOREIGN ' as a means of putting a FITS format wrapper about an arbitrary file. This convention enables a file or tree of files to be wrapped up in a FITS file and later restored in the original form to disk. A set of optional keywords are defined to store information about the encapsulated file. An example of the full set of header keywords is shown below: XTENSION= 'FOREIGN ' / NOAO xtension type BITPIX = 8 / Bits per pixel (byte) NAXIS = 0 / No Image matrix PCOUNT = 10756 / File size in bytes GCOUNT = 1 / One group EXTNAME = 'kp-B-pgr_120.png' / Filename EXTVER = 1 EXTLEVEL= 1 / Directory level FG_GROUP= 'kp-ftrB-dps-mdppgr' / Group Name FG_FNAME= 'kp-B-pgr_120.png' / Filename FG_FTYPE= 'binary ' / File type FG_LEVEL= 1 / Directory level FG_FSIZE= 10756 / Data size (bytes) FG_FMODE= '-rw-r--r--' / File mode FG_FUOWN= 'smith ' / File UID FG_FUGRP= 'jones ' / File GID FG_CTIME= '2006-06-15T02:32:49' / file ctime (GMT) FG_MTIME= '2006-06-15T02:32:49' / file mtime (GMT) END This Public Comment Period will remain open for 30 days. Comments may be posted here on the FITSBITS mail exploder or the sci.astro.fits newsgroup. Minor typographical issues may be sent directly to the submitters of the convention. Bill Pence (on behalf of the IAU FITS Working Group) -- __________________________________________________ __________________ Dr. William Pence NASA/GSFC Code 662 HEASARC +1-301-286-4599 (voice) Greenbelt MD 20771 +1-301-286-1684 (fax) |
#2
|
|||
|
|||
[fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod
William Pence wrote:
This is to announce the start of the Public Comment Period on the "Foreign File Encapsulation" FITS convention that has been submitted by the IRAF group at NOAO for inclusion in the "Registry of FITS Conventions" that is maintained by the IAU FITS Working Group. FG_FMODE The file mode as a string ("rwx-rwx-rwx", bits not set given as "-"). x ? Why would a file be executable? In the tradition of the omni-platform transportability of FITS files, it is unlikely that a file would be really executable. If the executable was intended to go onto a machine of the same architecture (and libc....) then why use FITS? An example of an executable file, however, would be a virus. Surely we don't want to go there? Peter. |
#3
|
|||
|
|||
Start of "Foreign File Encapsulation" Public Comment Period
Peter Bunclark wrote: x ? Why would a file be executable? In the tradition of the omni-platform transportability of FITS files, it is unlikely that a file would be really executable. If the executable was intended to go onto a machine of the same architecture (and libc....) then why use FITS? An example of an executable file, however, would be a virus. Another example would be a shell-script (requiring the execute bits), bash/tcsh are pretty ubiquitous and portable. The extension is meant to capture any foreign blob, this could include say a perl script used as a pipeline recipe but stored in a foreign extension because the archive saving it is tuned to FITS files. Compiled binaries are allowed as well, but you'd be nuts to save them in this way. -Mike |
#4
|
|||
|
|||
[fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod
Hi Pete,
Why would a file be executable? In the tradition of the omni- platform transportability of FITS files, it is unlikely that a file would be really executable. Just to provide some context, foreign file encapsulation was created as part of a prior NOAO archive project several years ago to encapsulate various non-FITS metadata products generated on Kitt Peak – observing logs, weather pictures, etc. NOAO is using it again to transport PNG files from the Mosaic camera pipeline to the next generation NSA archive. It was designed to be very general purpose, hence the authors' interest in reminding the community of its existence. You can certainly create file collections that a third party would be foolish to unpack on their own system. The same is true of tar. In any event, if the fgread program isn't setuid, such files would only be executable as the UID of user, containing any risk. Nelson Zárate will have an ADASS poster, and perhaps we can expect foreign encapsulation and the general topic of FITS extension types to enliven the FITS BoF. If the executable was intended to go onto a machine of the same architecture (and libc....) then why use FITS? We've long since moved past the point of regarding FITS as a pure transport format. It's used in all our archives, for one thing. One might well want to package up processing scripts with the data they were applied to, for instance. An example of an executable file, however, would be a virus. Surely we don't want to go there? One person's virus is another's intelligent agent. I suspect that a lot of us have entertained the notion of an IRAF or IDL virus in the past. Why not FITS? VO will certainly have its hands full with XML and web services, many of whose features are intended to be very "intelligent" indeed. I suspect that foreign encapsulation is a sufficiently narrow niche market that we need not fret too much. Rob |
#5
|
|||
|
|||
[fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod
I have been following the discussion on the FOREIGN extension for quite
a bit, but found no time so far to communicate my views. My opinions fall in three different categories : a) procedural comments There was some confusion about the fact that a thing like the FOREIGN extension should go in the convention registry, and how. They are probably due to the fact here we are dealing with a "conforming extension" and not a mere convention on keyword or column naming This was first pointed out on Mon, 2 Oct 2006 by Thomas McGlynn The foreign file encapsulation doesn't really fit this framework. It describes an entirely new type of FITS HDU. In the past such a new type would require an amendment to the FITS standard. (which is not really the case ... a new extension has just to be registered IN A SEPARATE REGISTRY, it is not necessarily part of the standard) Also on Mon, 2 Oct 2006, William Pence wrote: The basic requirements for a convention to be included in the Registry a 1 It should conform to the FITS Standard. So I guess that until XTENSION='FOREIGN' is not registered in the registry of conforming extensions, it does not conform, and cannot be registered in the registry of conventions. AFAIK, things are dealt with in the IAU-FWG, so this is not a real issue I also agree with Bill when he says From my own perspective I would hope that the number of new extension types that are created can be kept to a minimum because of the impact that it has on existing software and the added confusion that it can cause for FITS users. I will continue this sort of discussion on the IAUFWG list. For what the comments requested on FITSBITS about a new convention, they should properly deal ONLY with the fact the documentation is complete, well written, unambiguous, not redundant etc. b) formal comments So we are not really entitled to say anything about a particular implementation of a (local) convention. If the authors use it, they'd have good reasons to do so. So the only really formal comments are those by Thomas McGlynn about some undefined acronyms, and about the fact that section 6 is part of an internal specification document and not relevant for the registry I'm not sure I understand what the acronyms MEF, FITS-MEF and EHU mean in this proposal. They never seem to be defined. Section 6 seems to be irrelevant to the proposal. c) substantial comments Now I'm a bit perplexed by a number of other items which have been pointed out by some participants to the discussion : On Mon, 2 Oct 2006, Thomas McGlynn wrote: The encoding of directories (and symbolic links) is unclear. Directories are not portable across systems. The potential system dependencies of directory structures/file names are not discussed. E.g., is c:\x\y\z\ a valid directory name? How is a absolute file path written on a Unix machine to be read on a Windows [...] What happens when we go from a case sensitive to a case insensitive [...] I don't think Windows supports symbolic links either. On the same date William Thompson wrote: 7. I'm not sure I understand the point of encoding the user and group ID of the file. Surely that's platform specific. And finally on Tue, 3 Oct 2006, Peter Bunclark wrote: The file mode as a string ("rwx-rwx-rwx", bits not set given as "-"). x ? Why would a file be executable? In the tradition of the omni-platform transportability of FITS files, it is unlikely [...] As gut feeling I tend to AGREE with all above comments (and the rest by Thomas McGlynn), which can be grouped as "the proposal has clearly in mind Unix as target system". For instance I could detail on Peter Bunclark's comment more than concerning with the execute bit being set, by the fact that a "file mode" made of 3 triplets rwxrwxrwx (NOT rwx-rwx-rwx !) for user, group and other is also NOT SYSTEM INDEPENDENT. If I recall well for instance VMS had 4 quadruplets RWED for user, group, world and system. I confess I ignore the Windows protection scheme. These are technical comments. To these one might add that, if the origin and target systems are both Unix systems I fail to see the need to devise such a GENERAL convention in lieu of using a well established one like tar files (I may see the need of a simpler convention to append a single foreign file to a FITS file, like a PNG preview). This is a philosophical comment. BUT WE ARE NOT ENTITLED TO MAKE SUCH COMMENTS WHEN DISCUSSING A CONVENTION TO GO IN THE CONVENTION REGISTRY. We are not even entitled to ask the authors of such a convention to change anything (keyword names or contents) they already use. We can only say whether it is well documented. Then, if NOAO has its own reasons to use such convention to move files around their site, they are in full right to do so and we ARE NOT ENTITLED to say anything like all the above comment items. I believe it would be different if the FOREIGN extension had to be accepted as a STANDARD extension. ONOY in such case I believe we could question the need to "replace tar" and include a "directory structure" into a "FITS FOREIGN" file, and we SHALL question the conventions used to store things like file names, case sensitivity, uid and gid, permissions in a system independent way, maybe requiring some limitation on character set, syntax etc. I actually wonder if all these sort of things have not already been standardised in the way CDs are written by utilities like makeisofs / cdrecord. A minor technicality, when Thomas McGlynn says I don't think Windows supports symbolic links either. I do not believe that is true. Windows has at user level some kind of links providing pointers to a file, which resemble soft links, even VMS had, at system level, something like that (which resembled more hard links). ---------------------------------------------------------------------------- Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133 Milano (Italy) For more info : http://www.iasf-milano.inaf.it/~lucio/personal.html ---------------------------------------------------------------------------- -- ---------------------------------------------------------------------- is a newsreading account used by more persons to avoid unwanted spam. Any mail returning to this address will be rejected. Users can disclose their e-mail address in the article if they wish so. |
#6
|
|||
|
|||
[fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod
Hi All -
Some more quick comments are in order here. First, what we are discussing is a convention, hence revisiting the design is not appropriate (unless someone wants to propose moving towards a standard; I probably would not suggest that myself in this case). Second, on the specific issue of Unix-oriented concepts in the design, this is not necessarily a problem for something like we have here. Unix Tar for example, exists in Cygwin (or Winzip etc.) and clearly has some utility even on such a platform. Things like mode bits, paths, and case sensitivity do not map exactly but they map adequately; some information is lost when going to a non-Unix platform, but the extra refinement is useful on reasonably POSIX systems such as the Unix-derivatives like Linux or Mac. I suspect that if we produced a rigorous, explicit model for all this, with an explicit mapping to each platform, we would not do much better and would only complicate things. If something like a symbolic link is not mentioned that means it is not supported due to being too platform specific (I agree that perhaps it would be better to say this explicitly). This was a convenient convention for what it was intended for, which was extending an existing FITS multi-extension system for transporting a telescope data stream to an archive, without having to redesign the whole thing to use something other than FITS. If we had to do something similar today I might prefer to use something more standard like ZIP as the container, keeping the FITS part simple and focused on single files. This would solve various problems such as providing a directory for efficient random access to extract files from the container - probably today it is no longer useful to attempt to do such a thing in FITS. But the point here is merely to document existing conventions, so it is not very useful to revisit such matters, except maybe as we consider where to take FITS itself in the future. - Doug |
#7
|
|||
|
|||
[fitsbits] Start of "Foreign File Encapsulation" Public CommentPeriod
Doug Tody wrote:
Hi All - Some more quick comments are in order here. First, what we are discussing is a convention, hence revisiting the design is not appropriate (unless someone wants to propose moving towards a standard; I probably would not suggest that myself in this case). Second, on the specific issue of Unix-oriented concepts in the design, this is not necessarily a problem for something like we have here. SNIP - Doug I understand better from this and preceding comments the essential difference between a standard proposal and a convention registry. But one of the strong cases for having a convention registry is that others with similar needs might adopt an existing convention for their own purpose, thus short-cutting design and testing, and making a bigger pool of complying systems. Thus, although we might not criticise a convention in the same rigorous was as we would a standards proposal, surely it is apposite to append a SHORTCOMINGS section where it is admitted that, say, this convention doesn't generalise very well and that some of its provisions (and I maintain executable is not a good idea) may not be advisable. Pete |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Free Commodities Are Abused | Len | Policy | 46 | December 5th 05 06:21 AM |
[fitsbits] Start of the HEALPix "Public Comment Period" | William Pence | FITS | 0 | November 15th 05 10:42 PM |
[fitsbits] Start of the FITS MIME type Public Comment Period | William Pence | FITS | 8 | June 17th 04 06:08 AM |
Our Moon as BattleStar | Rick Sobie | Astronomy Misc | 93 | February 8th 04 10:31 PM |
Something more interesting for you to read! | Greg Dortmond | UK Astronomy | 12 | December 22nd 03 05:51 AM |