[fitsbits] Clarification of EXTEND, please

Without naming names, I just today hit a conflict between two pieces of
software. One created a FITS file with only a single HDU whose header
contained the line 'EXTEND = F'. A second piece of software interpreted
the presence of the EXTEND keyword in the primary HDU as meaning it
should check for extensions, tried to fseek() beyond the end of the
primary HDU and data array, and died.

Who did what right and wrong in this scenario? I'll send bug reports
with the correct answers to the software authors.

--
Stephen Walton
Dept. of Physics & Astronomy, Cal State Northridge

Stephen, the standard is pretty clear on this
e.g.
http://archive.stsci.edu/fits/fits_standard/node39.html

Even if EXTEND = T, it only means that there *may* be
further extensions, so doing something that will die if
extensions are not present is always incorrect.
You should be prepared to handle the possibility of EOF
when checking for extensions; it's not that hard to do.

EXTEND=F is in fact a guarantee that further extensions
are NOT present and by my reading of the standard is
a perfectly legal thing to do.
- Jonathan McDowell

Stephen, the standard is pretty clear on this
e.g.
http://archive.stsci.edu/fits/fits_standard/node39.html

Even if EXTEND = T, it only means that there *may* be
further extensions, so doing something that will die if
extensions are not present is always incorrect.
You should be prepared to handle the possibility of EOF
when checking for extensions; it's not that hard to do.

EXTEND=F is in fact a guarantee that further extensions
are NOT present and by my reading of the standard is
a perfectly legal thing to do.
- Jonathan McDowell

On Fri, 2004-05-14 at 10:31, Thierry Forveille wrote:
the presence of the EXTEND keyword in the primary HDU as meaning it
should check for extensions, tried to fseek() beyond the end of the
primary HDU and data array, and died.

The second piece did wrong, twice :-) EXTEND indicates only that the file
may contain an extension, not that it does, and then only when set to T.
It looks like they took two unwarranted shortcuts...

Time to name names. The offending piece of software is the fitsread.m
which The Mathworks distributes with MATLAB. In addition to above
problem, Mathworks's FITS software contains another bug, this one in
fitsinfo.m: it seeks 2880 bytes past an HDU which is exactly n*2880
bytes in length before trying to read data, and thus skips the first
2880 bytes of actual data. I have a patch for the second problem which
I'd be happy to share; I haven't tried to fix the first one yet, but it
is probably a simple matter of checking for EXTEND = T rather than the
mere existence of EXTEND.

Thanks to all.

--
Stephen Walton
Dept. of Physics & Astronomy, Cal State Northridge

On Fri, 2004-05-14 at 10:31, Thierry Forveille wrote:
the presence of the EXTEND keyword in the primary HDU as meaning it
should check for extensions, tried to fseek() beyond the end of the
primary HDU and data array, and died.

The second piece did wrong, twice :-) EXTEND indicates only that the file
may contain an extension, not that it does, and then only when set to T.
It looks like they took two unwarranted shortcuts...

Time to name names. The offending piece of software is the fitsread.m
which The Mathworks distributes with MATLAB. In addition to above
problem, Mathworks's FITS software contains another bug, this one in
fitsinfo.m: it seeks 2880 bytes past an HDU which is exactly n*2880
bytes in length before trying to read data, and thus skips the first
2880 bytes of actual data. I have a patch for the second problem which
I'd be happy to share; I haven't tried to fix the first one yet, but it
is probably a simple matter of checking for EXTEND = T rather than the
mere existence of EXTEND.

Thanks to all.

--
Stephen Walton
Dept. of Physics & Astronomy, Cal State Northridge

Stephen Walton writes:
Time to name names. The offending piece of software is the fitsread.m
which The Mathworks distributes with MATLAB. In addition to above
problem, Mathworks's FITS software contains another bug, this one in
fitsinfo.m: it seeks 2880 bytes past an HDU which is exactly n*2880
bytes in length before trying to read data, and thus skips the first
2880 bytes of actual data. I have a patch for the second problem which
I'd be happy to share; I haven't tried to fix the first one yet, but it
is probably a simple matter of checking for EXTEND = T rather than the
mere existence of EXTEND.

That's a first step, but even if EXTEND is present and set to T it remains
legal to not have any extension, and there could also be multiple extensions.
FITS reading software should really be prepared to recover if it does not
find an extension.

Stephen Walton writes:
Time to name names. The offending piece of software is the fitsread.m
which The Mathworks distributes with MATLAB. In addition to above
problem, Mathworks's FITS software contains another bug, this one in
fitsinfo.m: it seeks 2880 bytes past an HDU which is exactly n*2880
bytes in length before trying to read data, and thus skips the first
2880 bytes of actual data. I have a patch for the second problem which
I'd be happy to share; I haven't tried to fix the first one yet, but it
is probably a simple matter of checking for EXTEND = T rather than the
mere existence of EXTEND.

That's a first step, but even if EXTEND is present and set to T it remains
legal to not have any extension, and there could also be multiple extensions.
FITS reading software should really be prepared to recover if it does not
find an extension.

On Fri, 2004-05-14 at 09:57, Jonathan McDowell wrote:
Stephen, the standard is pretty clear on this
e.g.
http://archive.stsci.edu/fits/fits_standard/node39.html

I have sent a bug report to the Mathworks about fitsinfo.m, specifically
that it should not treat the absence of an extension on an otherwise
legal FITS file as a fatal error even if EXTEND=T is stated in the
primary HDU. I'll see what response I get.

Thanks to everyone who responded.

--
Stephen Walton
Dept. of Physics & Astronomy, Cal State Northridge

On Fri, 2004-05-14 at 09:57, Jonathan McDowell wrote:
Stephen, the standard is pretty clear on this
e.g.
http://archive.stsci.edu/fits/fits_standard/node39.html

I have sent a bug report to the Mathworks about fitsinfo.m, specifically
that it should not treat the absence of an extension on an otherwise
legal FITS file as a fatal error even if EXTEND=T is stated in the
primary HDU. I'll see what response I get.

Thanks to everyone who responded.

--
Stephen Walton
Dept. of Physics & Astronomy, Cal State Northridge