Details on Spirit Breakdown

The following message was sent to me by a buddy who works closely with
VXWorks, a commercial RTOS that's used on Spirit. Unfortunately he
couldn't identify the original poster, but it has loads of good technical
details regarding why Spirit broke down. Unfortunately, the OP's English
needs a little work, but the details are decipherable.

Incidentally, the message also says something about opportunity having a
"stuck heater" on its instrumentation arm. I hadn't heard about that.
Does anybody have any more details about it or a link describing the
situation? Is it fixed, or are they still working on it?


--------------------------------------------------------------------------

The long and short of the Spirit debacle: Test what you intend to
do, and Do what you tested for. In the lab, they tested for 9 days
at a time, then cleaned things up, and ran again.

On Mars, they ran for over 18 days without doing any housecleaning.
You can get away with that on earth, you just have to vacuum twice...

As has been noted, Spirit has relatively little RAM for applications
and the OS - very little margin of error. The FLASH directory structure
and file structure required enough RAM for the dosFs cache that one last
malloc() was all it took to go over the top (malloc() puts the fat in the
FAT file system). The last malloc() failed, the task is suspended, the
dosvd semaphore never released, a high-prio task waiting on an "open"
causes a prio-inversion, another high priority task needs to write to a
heath monitor board (the RTI clock) but is block from running in time.

The RTI clock resets the system.

The system then re-initializes, rebuilds the dosFs cache... and the whole
thing happens all over again.

Current status:

They've sent commands to removed a coupld of directories of files from
the system and rebooted. They'll bring back files we're interested in,
then send another script command to reformat TFFS, and reboot. In other
words... we're back in business, Spirit is up and running, and we're
ready to finish cleaning house and start doing science again.

The heater on Opportunity that is stuck on is in the arm that holds the
RAT and two of the spectrometers. It's a very minor concern at this point,
and should not cause any operational constraints.

Opportunity had sent status early-on that the Mossbauer spectrometer had
been damaged on launch. After landing the spectrometer was checked again,
and it is working fine. It won't be long untill all 12 wheels are rolling
towards objectives!