Who made Spirit's flash memory? How Now Dow Jones

Hi,

Been busy today so sorry in advance if my info is dated.


From what I understand Spirit has bad flash memory. Who made
it? If that is their best then what will users here on Earth think of
their product?


If you're pro-space then reward the company that makes space
work. Punish the company that fails.

Hmm, on what ship did Darwin travel?.....


-Steve

As I was browsing in another newsgroup, I noticed a comment that
Spirit's flash memory problem root cause might be in the memory
management software.

What is that flash memory there for? Is it a space-style hard drive?
I'd like to see some discussion here describing Spirit's computer and
communication systems.

Cheers -- Martha Adams

On Mon, 26 Jan 2004 05:19:03 +0000 (UTC), (Martha H
Adams) wrote:

As I was browsing in another newsgroup, I noticed a comment that
Spirit's flash memory problem root cause might be in the memory
management software.

Actually, NASA has just discounted that theory, when testing of this
flash memory has highlighted that it is working just fine.

The latest main theory is that it is a software "bug" in the file
management section.

In other words the software simply wrote to an area of memory that it
should not have due to faulty calculations or memory mapping.

I done that once myself when trying to get a device to hold more data
than what was physically possible. Naturally it failed.

What is that flash memory there for?

It is used in the same sort of way that you use memory on your home
computer system.

In the lower (or sometimes upper) area of memory is where all the
variables are stored to be used by the operating system. The rest of
the memory is for running programs and storing data.

Is it a space-style hard drive?

No, but if they come with a hard drive type system I cannot say. I
guess that this is unlikely due to the weight, where their flash
memory can just be made to be non-volatile in case of temporary power
loss.

In other words they should store all those nice pictures and science
data in that very flash memory.

I'd like to see some discussion here describing Spirit's computer and
communication systems.

What do you desire to know?

Cardman
http://www.cardman.com
http://www.cardman.co.uk

(Martha H Adams) writes:

What is that flash memory there for? Is it a space-style hard drive?
I'd like to see some discussion here describing Spirit's computer and
communication systems.

Based only on recent press conferences:

Three types of memory: 128 MBytes volatile RAM, 256 MBytes
non-volatile Flash, some amount of non-volatile EEPROM. Said to be
"just like your PC", but I wonder; it must be radiation hardened, but
I don't know if that's by special devices or special boxes or both.
RAM is fastest and is probably the only memory directly used by the
CPU as it's memory. Flash is used rather like a non-mechanical disk
drive. (Their interfaces often look to the CPU exactly like a disk
drive and have disk drive filesystem data structures. It sounded like
that is the case here, but I couldn't be sure; they might have just
been over-analogizing again for the Unwashed Masses.) EEPROM is
sort of like flash, but much slower (and I suspect, more reliable);
not sure why both flash and EEPROM in one system.

I heard they keep two non-volitile copies of (main?) software aboard,
which can be loaded into RAM during restarts, at least one from
EEPROM, I conclude.

They said they could work in a degraded mode without flash, but it
seems that won't be needed.

I was interrupted and missed a bit this morn, but got the impression
that they intended to not rewrite the Flash until they dumped it all
for debugging, but goofed and rewrote it, and in the process showed
that the Flash was working OK (now, at least).

Three types of radios: X-band omni-directional antenna, low speed
(8-1000? bits/sec); X-band directional antenna, medium speed (11000-
128000 bits/sec); UHF antenna, high speed (up to 256000? bits/sec).
X-band only direct to Earth, UHF only to the two orbiters. There
was two low-speed X-band broadcasting through the bags during
landing; maybe one was the main one, the other at one corner of
the lander bottom, radiating two opposite hemispheres, I guess.

It seems that the radios are told when to broadcast by the computer
and if they get no data from the computer, they broadcast random
numbers in the data packets (or tones on the omni-antenna). The
UHF communications are two-way, using "handshaking" protocols
(and probably re-transmissions of garbled data).

On Mon, 26 Jan 2004 08:35:32 GMT, (Gary W.
Swearingen) wrote:

(Martha H Adams) writes:

What is that flash memory there for? Is it a space-style hard drive?
I'd like to see some discussion here describing Spirit's computer and
communication systems.

Based only on recent press conferences:

Three types of memory: 128 MBytes volatile RAM,

That would be best used as a work area and not for long term storage.

256 MBytes non-volatile Flash,

No doubt with a limited amount of write cycles before it could fail,
just like your good old USB pen drives.

some amount of non-volatile EEPROM.

Not very much I would presume.

Said to be "just like your PC",

Yes somewhat like that.

but I wonder; it must be radiation hardened,

Certainly. In a metal box as well.

but I don't know if that's by special devices or special boxes or both.

With the processor it is like having two copies of everything just in
case some circuit or other gets nuked. For the memory it is hard to
say, when they can always map around faults or erase any unwanted bit
flipping.

RAM is fastest and is probably the only memory directly used by the
CPU as it's memory.

Most likely, but there would be some things that it would not wish to
lose in case of power loss or a reset.

Flash is used rather like a non-mechanical disk drive.

Just like a USB pen drive. And you can actually get yourself USB pen
drives with a larger capacity than with these MERs. :-]

(Their interfaces often look to the CPU exactly like a disk
drive and have disk drive filesystem data structures.

Hey, just like a USB pen drive. :-]

Does these MERs have a virtual USB port and pen drive for about $50,
or has NASA spent millions making their own version? :-]

It sounded like
that is the case here, but I couldn't be sure; they might have just
been over-analogizing again for the Unwashed Masses.)

Sounds clear enough to me.

EEPROM is sort of like flash, but much slower (and I suspect, more
reliable);

Yes, one big chip instead of thin memory banks.

not sure why both flash and EEPROM in one system.

The EEPROM would contain the BIOS of these MERs, which will always be
fixed there.

So when one of these MERs boots up, then first of all it loads it's
brain pattern from the EEPROM, where it then goes and uses the RAM and
FLASH like any other computer.

I am not fully sure where the MER's O/S is stored, but given no other
option that is likely to be in the EEPROM. Since most firmware devices
do that, then I guess so.

I heard they keep two non-volitile copies of (main?) software aboard,

Yes in the EEPROM I presume, which could be two EEPROMs I guess. Even
your modern PCs come with two EEPROMs these days.

One is for erasing and loading in a new BIOS/CMOS system, where one is
the backup should you mess up your first EEPROM during update.

which can be loaded into RAM during restarts, at least one from
EEPROM, I conclude.

I expect so.

They said they could work in a degraded mode without flash, but it
seems that won't be needed.

Not now that they have excluded this as the fault.

What I guess they mean by that is that they no longer use the flash
and just make use of the RAM instead.

Doing that as a temporary measure now would get Spirit working again
just fine with only a small risk of losing your science data during a
power cut or reboot.

I was interrupted and missed a bit this morn, but got the impression
that they intended to not rewrite the Flash until they dumped it all
for debugging, but goofed and rewrote it, and in the process showed
that the Flash was working OK (now, at least).

LOL. Well it is hard to stop the natural function of the system when
you have to power it on first.

Three types of radios: X-band omni-directional antenna, low speed
(8-1000? bits/sec); X-band directional antenna, medium speed (11000-
128000 bits/sec); UHF antenna, high speed (up to 256000? bits/sec).

One more than I expected.

X-band only direct to Earth, UHF only to the two orbiters. There
was two low-speed X-band broadcasting through the bags during
landing; maybe one was the main one, the other at one corner of
the lander bottom, radiating two opposite hemispheres, I guess.

I see.

It seems that the radios are told when to broadcast by the computer
and if they get no data from the computer, they broadcast random
numbers in the data packets (or tones on the omni-antenna). The
UHF communications are two-way, using "handshaking" protocols
(and probably re-transmissions of garbled data).

Well Spirit's brain lives on, where once they solve this bug, then it
will be back to working just fine.

From what I know this problem should affect Opportunity as well, when
it would not even have to get off this lander to blow a fuse.

As sure enough this problem must have come about when they started
packing full that memory of theirs and it wrapped around to reuse the
lower section.

So Opportunity could well suffer this same problem if they take lots
of data with it, where the best temporary solution to keep using this
flash is to just clear whatever data they got in the flash (following
upload) whenever the flash is over say half full and to start to reuse
it from the start.

They go and spend a couple of days doing a rock scan and filling up
that flash, then sure enough that is what will cause this.

Keep your stored data volume within reason and wipe the lot frequently
should prevent this problem cropping up.

Actually I just had an idea. As if this flash memory can start to fail
after say a 1,000 or more writes, then they would desire to spread the
data over the full 256MB.

And so their problem cropped up when they reached the end of this
256MB and the counter told it to go back around and start from the
beginning. Or basically failed to noticed when the 256MB ended.

I could have that fixed no problem in one of these "dirty fixes" (a
coders term to describe working around a bug instead of fixing it) had
I known the system.

Well as I said their easiest solution is to just "switch off the
flash" for a short time in order to have it doing science work instead
of sitting ideal for days or weeks awaiting a bug fix.

That delay would be unhelpful when those 90 days are ticking down
quite quickly now.

Naturally, they will want this bug fully detailed before they move
Spirit another inch, but sure enough what I just said is what they
should certainly find.

They could have it working fine right now, but they won't when they
don't yet fully understand what happened. I do, when most firmware
coders working with a 2GL or 3GL end up doing just that at some point.

Cardman
http://www.cardman.com
http://www.cardman.co.uk

Cardman writes:

On Mon, 26 Jan 2004 05:19:03 +0000 (UTC), (Martha H
Adams) wrote:

[delurk and snip]

What is that flash memory there for?

It is used in the same sort of way that you use memory on your home
computer system.

I seriously doubt that.

In the lower (or sometimes upper) area of memory is where all the
variables are stored to be used by the operating system. The rest of
the memory is for running programs and storing data.

Storing program variables in flash? That strikes me as rather improbable.
Extremely improbable, in fact, given the speed of flash memory, especially
with respect to operations of writing the data into it (and the way it
is done, with a read-the-block-into-RAM/clear-the-block-in-flash/update-
data-in-RAM/write-updated-data-to-flash). To the best of my knowledge, you
can't just "poke" a value into a random address within flash. At least you
can't do it with flashes that I'm familiar with.

Is it a space-style hard drive?

No, but if they come with a hard drive type system I cannot say. I
guess that this is unlikely due to the weight, where their flash
memory can just be made to be non-volatile in case of temporary power
loss.

Hmmmm... I think the original poster was asking whether flash memory
on Spirit is used as a sort of the permanent storage device, which is
very likely a correct assumption.

In other words they should store all those nice pictures and science
data in that very flash memory.

So, you are also saying it's a "space-style hard drive" after all.

Beside storing those nice pictures, I'd also guess that the flash is holding
the complete configuration data (parameters required for the rest of Spirit's
software to function... "ini files" or "repository", for windows-addicts, or
the /etc directory for the rest of the world), making it possible to re-boot
cleanly and continue working with the most recent parameterization, as
received from Earth.


Cheers, alex.
--
"Linux means never having to delete your love mail." -- Don Marti

On Mon, 26 Jan 2004 22:31:45 GMT, Alex Pozgaj
wrote:

Cardman writes:

On Mon, 26 Jan 2004 05:19:03 +0000 (UTC), (Martha H
Adams) wrote:
What is that flash memory there for?

It is used in the same sort of way that you use memory on your home
computer system.

I seriously doubt that.

A USB "flash" pen drive comes to mind.

In the lower (or sometimes upper) area of memory is where all the
variables are stored to be used by the operating system. The rest of
the memory is for running programs and storing data.

Storing program variables in flash?

It may be slow, but it is also non-volatile. After all there are not
many other places that you can keep important data following a reset.

That strikes me as rather improbable.

Know what non-volatile is? :-]

Sure, you would keep most system variables in the much faster RAM, or
more correctly the SDRAM on these RAD6000 boards, but these types of
memory are volatile.

Extremely improbable, in fact, given the speed of flash memory, especially
with respect to operations of writing the data into it (and the way it
is done, with a read-the-block-into-RAM/clear-the-block-in-flash/update-
data-in-RAM/write-updated-data-to-flash).

Yes it is slow, but then Spirit has been slowly writing up to 256 mb
of picture files to it.

To the best of my knowledge, you
can't just "poke" a value into a random address within flash. At least you
can't do it with flashes that I'm familiar with.

Ever used a USB pen drive? This is also flash memory, where you can
write, read and erase in a random way like any other drive.

The only real problem with flash memory is that it has a limited
amount of write cycles before chance of failure, which is one reason
why these MERs would prefer to use up the full 256mb before restarting
again on the second write.

Is it a space-style hard drive?

No, but if they come with a hard drive type system I cannot say. I
guess that this is unlikely due to the weight, where their flash
memory can just be made to be non-volatile in case of temporary power
loss.

Hmmmm... I think the original poster was asking whether flash memory
on Spirit is used as a sort of the permanent storage device, which is
very likely a correct assumption.

Maybe so, but flash memory is certainly not like a traditional hard
drive. I am sure that just thinking about the technical difference
will highlight this.

As I mentioned in a later posting this flash memory is very much like
your common USB pen drives, which also makes use of non-volatile flash
memory.

In other words they should store all those nice pictures and science
data in that very flash memory.

So, you are also saying it's a "space-style hard drive" after all.

No, a space style "flash" drive. :-]

After all you will find no read/write heads here, a spinning disk or
it involving magnetism.

Beside storing those nice pictures, I'd also guess that the flash is holding
the complete configuration data (parameters required for the rest of Spirit's
software to function...

Exactly what I was thinking when I wrote that, but I guess that I
should have been more clear.

One thing that should certainly be stored in this non-volatile memory
is the command sequence received direct from Earth or via Odyssey or
MGS.

As if Spirit has a power fluctuation and resets, then it will start
back up knowing exactly what it has to do. Even if more likely it
would go into a safe state first.

So what happens if your picture files messed up and over wrote
important system data like this? As then Spirit may well go and do
things that it is not supposed to.

Covering several square miles is unlikely, but it is possible that it
may have moved.

That is of course one of many types of data that could be stored here,
where others may be time/date, communication sequence, last recorded
position of the Sun, the latest Odyssey and MSG schedules, then most
likely much more that what I could ever guess.

"ini files" or "repository", for windows-addicts, or
the /etc directory for the rest of the world), making it possible to re-boot
cleanly and continue working with the most recent parameterization, as
received from Earth.

Yes, where now you know just how Spirit could have broke down when
that 256mb all got used.

Cardman
http://www.cardman.com
http://www.cardman.co.uk

Cardman writes:

Yes, where now you know just how Spirit could have broke down when
that 256mb all got used.

Based on Monday's briefing, that's not quite the suspected problem.
The problem seems to be that the number of files stored in Flash is so
large that some "metadata" associated with the files in RAM filled up
(or overflowed?) the RAM (or at least the area reserved for such
data). Each file probably has a fixed number of bytes associated with
it, with info about the data or maybe some filesystem info like
location in Flash. They still had "hundreds" of files of "cruise"
(pre-EDL) data stored in there, plus a lot of trash built up over the
first 18 days on Mars. Their simulations apparently were not as
thorough as they should have been to turn up the file handling bug.
They said the second rover would probably deleting their cruise files
soon, to avoid the same fate.

On Tue, 27 Jan 2004 06:22:21 GMT, (Gary W.
Swearingen) wrote:

Cardman writes:

Yes, where now you know just how Spirit could have broke down when
that 256mb all got used.

Based on Monday's briefing, that's not quite the suspected problem.

Oh?

The problem seems to be that the number of files stored in Flash is so
large that some "metadata" associated with the files in RAM filled up
(or overflowed?) the RAM (or at least the area reserved for such
data).

That was not unlike my theory, not too well pointed out above, when
this 256 mb all got filled up and the next write ended up somewhere
where it should not have.

Each file probably has a fixed number of bytes associated with
it,

True, but the placement within the flash should be variable and just
wrote in a sequential way.

with info about the data or maybe some filesystem info like
location in Flash.

Yes, the flash would need to store a kind of FAT telling the system
the status of the files on it. It would be very unusual to fill up
your allocation table, when they are normally made plenty big enough
for such large number of files.

They still had "hundreds" of files of "cruise"
(pre-EDL) data stored in there,

Hopefully they managed to get a copy before they recently wrote over
this flash memory.

plus a lot of trash built up over the
first 18 days on Mars.

It remains to be seen if the data is erased following upload, or only
when the flash gets full.

Their simulations apparently were not as
thorough as they should have been to turn up the file handling bug.

That would have been unusual. Still, since the vital data then becomes
corrupt in a random way, then things can be seemingly fine in some
cases.

They said the second rover would probably deleting their cruise files
soon, to avoid the same fate.

A good idea.

Cardman
http://www.cardman.com
http://www.cardman.co.uk

"Cardman" wrote
Yes, the flash would need to store a kind of FAT telling the system
the status of the files on it. It would be very unusual to fill up
your allocation table, when they are normally made plenty big enough
for such large number of files.

The explanation given is not that they ran out of space on the flash, but in
the structures in RAM.

The rover uses the VxWorks OS. The following usenet message from 2000 may be
of interest:

---------------------------------------------------------------------
Newsgroups: comp.os.vxworks
Subject: Processing Overhead for Sockets = ??
Date: Sat, 01 Jul 2000 21:19:18 GMT
From: "Dan Perlberger"

Keep in mind that VxWorks has a maximum number of file descriptors which can
be open at once. This value (I think it is MAX_FILES or MAX_FILES_OPEN) is
defined in one of the config files when building the kernel.

On our system, it is set to 50...so if you have several dozen open sockets,
you might hit your limit, or you might prevent other files from being able
to be opened.
-------------------------------------------------------------------------

Another:

---------------------------

Newsgroups: comp.os.vxworks
Subject: Max Resources ?
Date: Sat, 22 Jul 2000 19:40:31 GMT
From:
Organization: Deja.com - Before you buy.
Message-ID:
References:


Hi,

In article ,
Madhu wrote:
Sorry for the HTML posting. We have tried here increasing the
NUM_FILES
but it does not seem to make a difference. How does the memory get
allocated

for these resources i.e the memory for the system data structures,
probably
might
need some system cluster tuning.

The memory for the fd table is allocated when the kernel is initialised.
You shouldn't need to "tune" anything to increase the number of file
descriptors that can be open at any one time, other than re-build your
kernel having changed NUM_FILES of course.

As for other system data structures, they are allocated dynamically
using the equivalent of malloc() from the system memory partition (the
heap). You can see if you are running out of memory using memShow(). How
many objects are you creating, and how much memory does your board have?

Regards,

John...
----------------------------------------------------------------

They still had "hundreds" of files of "cruise"
(pre-EDL) data stored in there,

Hopefully they managed to get a copy before they recently wrote over
this flash memory.

Might be difficult if it is a lot of data. Each satellite pass sees to be
reported as about 20MB, so it would take ~12 passes (6 days with one pass of
MGS and ODY per day) to get 256MB?

Joe