Details on Spirit Breakdown

The following message was sent to me by a buddy who works closely with
VXWorks, a commercial RTOS that's used on Spirit. Unfortunately he
couldn't identify the original poster, but it has loads of good technical
details regarding why Spirit broke down. Unfortunately, the OP's English
needs a little work, but the details are decipherable.

Incidentally, the message also says something about opportunity having a
"stuck heater" on its instrumentation arm. I hadn't heard about that.
Does anybody have any more details about it or a link describing the
situation? Is it fixed, or are they still working on it?


--------------------------------------------------------------------------

The long and short of the Spirit debacle: Test what you intend to
do, and Do what you tested for. In the lab, they tested for 9 days
at a time, then cleaned things up, and ran again.

On Mars, they ran for over 18 days without doing any housecleaning.
You can get away with that on earth, you just have to vacuum twice...

As has been noted, Spirit has relatively little RAM for applications
and the OS - very little margin of error. The FLASH directory structure
and file structure required enough RAM for the dosFs cache that one last
malloc() was all it took to go over the top (malloc() puts the fat in the
FAT file system). The last malloc() failed, the task is suspended, the
dosvd semaphore never released, a high-prio task waiting on an "open"
causes a prio-inversion, another high priority task needs to write to a
heath monitor board (the RTI clock) but is block from running in time.

The RTI clock resets the system.

The system then re-initializes, rebuilds the dosFs cache... and the whole
thing happens all over again.

Current status:

They've sent commands to removed a coupld of directories of files from
the system and rebooted. They'll bring back files we're interested in,
then send another script command to reformat TFFS, and reboot. In other
words... we're back in business, Spirit is up and running, and we're
ready to finish cleaning house and start doing science again.

The heater on Opportunity that is stuck on is in the arm that holds the
RAT and two of the spectrometers. It's a very minor concern at this point,
and should not cause any operational constraints.

Opportunity had sent status early-on that the Mossbauer spectrometer had
been damaged on launch. After landing the spectrometer was checked again,
and it is working fine. It won't be long untill all 12 wheels are rolling
towards objectives!

John Schutkeker wrote:
... Incidentally, the message also says something about opportunity having a
"stuck heater" on its instrumentation arm. I hadn't heard about that.
Does anybody have any more details about it or a link describing the
situation? Is it fixed, or are they still working on it?

Check the status page at http://www.spaceflightnow.com/mars/mera/status.html

Search the page for "heater" and you will find the section. It's near
the bottom of the page.

--
bp
Proud Member of the Human O-Ring Society Since 2003

Bruce Palmer wrote in
. net:

Check the status page at
http://www.spaceflightnow.com/mars/mera/status.html
Search the page for "heater" and you will find the section. It's near
the bottom of the page.

Thanks. These problems always make me very nervous, because my initial
feeling is that they sound like hardware problems. In this case, it sounds
like a short circuit that bypasses the switch (or just a broken switch)
upstream of the heater load. Of course, that's just my gut feeling, and
I've never seen the schematics. But if the only repair option we have is
to remotely reprogram the rovers, the only thing you can do is to stop
using the parts that had the hardware failures.

Yes, I know I'm just being paranoid, and the Spirit team really did a bang
up job getting that breakdown fixed. I hope that they can do as well with
this one. Three cheers for the software guys! Hip, hip...

John Schutkeker wrote in message . 93...
The following message was sent to me by a buddy who works closely with
VXWorks, a commercial RTOS that's used on Spirit. Unfortunately he
.....
As has been noted, Spirit has relatively little RAM for applications
and the OS - very little margin of error. The FLASH directory structure
and file structure required enough RAM for the dosFs cache that one last
malloc() was all it took to go over the top (malloc() puts the fat in the
FAT file system).

Quota management, anyone ? So it actually ran out of RAM, not flash
space, due to large fat ?
In any case, i woulda thought that it would have given a low memory or
low disk warning beforehand ...

-kert

(Kaido Kert) wrote in
om:

In any case, i woulda thought that it would have given a low memory or
low disk warning beforehand ...

....which does have to be trapped, and their trap may have also involved a
reboot. If I were writing it, I'd have the trap routine call home for
instructions and then do a busy wait while looking for the return call.

John Schutkeker ) wrote:
: (Kaido Kert) wrote in
: om:

: In any case, i woulda thought that it would have given a low memory or
: low disk warning beforehand ...

: ...which does have to be trapped, and their trap may have also involved a
: reboot. If I were writing it, I'd have the trap routine call home for
: instructions and then do a busy wait while looking for the return call.

Busy wait or request for interrupt? The latter would allow the processor
to actually sleep rather than have to fetch and execute instructions.

Eric

Kaido Kert ) wrote:
: John Schutkeker wrote in message . 93...
: The following message was sent to me by a buddy who works closely with
: VXWorks, a commercial RTOS that's used on Spirit. Unfortunately he
: ....
: As has been noted, Spirit has relatively little RAM for applications
: and the OS - very little margin of error. The FLASH directory structure
: and file structure required enough RAM for the dosFs cache that one last
: malloc() was all it took to go over the top (malloc() puts the fat in the
: FAT file system).

: Quota management, anyone ? So it actually ran out of RAM, not flash
: space, due to large fat ?
: In any case, i woulda thought that it would have given a low memory or
: low disk warning beforehand ...

My immediate thought was a memory leak problem (too many mallocs as
compared to the number of free statements). Disk? I would have thought the
computer system to be radiation shielded CMOS (or modern variant)
throughout. Cripes, I get disk problems first and foremost on the systems
I use.

Eric

: -kert

Eric Chomko ) wrote:
: John Schutkeker ) wrote:
: : (Kaido Kert) wrote in
: : om:

: : In any case, i woulda thought that it would have given a low memory or
: : low disk warning beforehand ...

: : ...which does have to be trapped, and their trap may have also involved a
: : reboot. If I were writing it, I'd have the trap routine call home for
: : instructions and then do a busy wait while looking for the return call.

: Busy wait or request for interrupt? The latter would allow the processor
: to actually sleep rather than have to fetch and execute instructions.

Actually a combination would be best: request for interrupt and sleep for
a period of say 20 minutes and then wake up and request again, sleep, and
the RFI, rather than a steady busy wait loop.

Eric

(Eric Chomko) wrote in
:

My immediate thought was a memory leak problem (too many mallocs as
compared to the number of free statements). Disk? I would have thought
the computer system to be radiation shielded CMOS (or modern variant)
throughout. Cripes, I get disk problems first and foremost on the
systems I use.

Do you know for a fact that this system has a disk? Since the working
software is completely static (except for bug fixes from earth), it's
memory requirements are fixed and known, and ROM alone should be
sufficient.

I would also expect the delicate moving parts of a disk drive to be
extremely sensitive to the comparatively violent landing process of bounce-
and-roll. I wouldn't want disk drives to be subjected to that trauma, if I
were the designer.

John Schutkeker ) wrote:
: (Eric Chomko) wrote in
: :

: My immediate thought was a memory leak problem (too many mallocs as
: compared to the number of free statements). Disk? I would have thought
: the computer system to be radiation shielded CMOS (or modern variant)
: throughout. Cripes, I get disk problems first and foremost on the
: systems I use.

: Do you know for a fact that this system has a disk?

I do not. And in another post I questioned the use of a disk given the
mechanical nature being suspect to a higher probability to failure.

: Since the working
: software is completely static (except for bug fixes from earth), it's
: memory requirements are fixed and known, and ROM alone should be
: sufficient.

Scratchpad RAM, at least, to work with the ROM will also be needed. Also,
if the ROM code has problems you can program around it in RAM as subtle
bug fixes. Not pretty, but works.

: I would also expect the delicate moving parts of a disk drive to be
: extremely sensitive to the comparatively violent landing process of bounce-
: and-roll. I wouldn't want disk drives to be subjected to that trauma, if I
: were the designer.


Yes, check earlier posts in this thread. Someone else mentioned a disk and
I questioned the reliability of using a disk in this type of situation.


Eric