Naval Reactor success ...

On or about Mon, 15 Sep 2003 12:23:59 CST, Rand Simberg
made the sensational claim that:
Of course, as von Braun pointed out, you can't make something
completely idiot proof--idiots are too ingenious...

I have it as a Douglas Adams quote: "A common mistake people make when trying
to design something completely foolproof is to underestimate the ingenuity of
complete fools."
--
This is a siggy | To E-mail, do note | This space is for rent
It's properly formatted | who you mean to reply-to | Inquire within if you
No person, none, care | and it will reach me | Would like your ad here

Pat Flannery wrote:


[snip]

There is also the mystery of why Lockheed Martin apparently couldn't
afford a complete set of bolts for each of the turn over carts, so that
one team felt it necessary to abscond with the other team's bolts. That
is somewhat difficult to fathom; but I'm sure there is a very involved
bureaucratic reason for the situation- "Requisition bolts? Do you have
any idea of the paperwork involved? The Congress is always getting down
on us about launch costs- by using just on set of mil spec bolts we save
the taxpayer hundreds of dollars! We'll just put them back on NOAA-N
when we're done; they don't even need to know that we took them..." That
may sound like a bad joke, but I bet the truth comes out pretty close to
that in the end.

[snip]

Just a guess...but I bet the other 24 bolts were locked away in some
critical parts tool crib. Being after hours, instead of calling someone
in to unlock the crib, they found a convenient set near by. And the
rest is history.

LooseChanj wrote:
On or about Mon, 15 Sep 2003 12:23:59 CST, Rand Simberg
made the sensational claim that:

Of course, as von Braun pointed out, you can't make something
completely idiot proof--idiots are too ingenious...


I have it as a Douglas Adams quote: "A common mistake people make when trying
to design something completely foolproof is to underestimate the ingenuity of
complete fools."

And it's this uncanny ability that will keep risk managers and tech.
journalists employed for a loooonnngggg time.

rk wrote:
Thoughts? This is an area that I'm thinking about a lot now.

Keep in mind that Naval Reactor and the SUBSAFE program both have two
things that *no* space program has, not even Soyuz, to wit:

An extensive practical background extending back over half a century,
and a large number of in service units.

D.
--
The STS-107 Columbia Loss FAQ can be found
at the following URLs:

Text-Only Version:
http://www.io.com/~o_m/columbia_loss_faq.html

Enhanced HTML Version:
http://www.io.com/~o_m/columbia_loss_faq_x.html

Corrections, comments, and additions should be
e-mailed to , as well as posted to
sci.space.history and sci.space.shuttle for
discussion.

stmx3 wrote:
Oh, and the really final last resort for critical systems where even a
trained operator may not follow the procedure correctly, is to have two
operators perform the procedure; or have an independent verification of
an equipment lineup, or something along these lines.

Rig-for-dive is also independently verified, as are many switch and
valve configurations for other work. Working on a submarine is
definitely *not* a lonely profession.

That's what's done when calculating where a reactor goes critical upon startup
or when working on submarine seawater piping.

For reference; here's the steps followed by the crew to work on
SUBSAFE systems;

1- QA paperwork is prepared that provides the procedure to be
followed, any inspections required during the course of the work, the
closeout tests to be made, and *explicitly* references the plans,
procedures, documents, and standards used to prepare the paperwork.

Ship's drawings are referenced to describe the exact components and
scope of work. If tag-outs are required, the paperwork is included in
the package, and must be approved via the tag-out chain prior to
submission of the package. Tools and materials are frequently
specified by stock number, MIL-STD, or other traceable/verfiable
methods.

In many cases copies of the plans, procedures, documents, and
standards are included as part of the package. (Thus they are
available at the worksite to workers, supervisors, and QA personnel.)
There is no such thing as a standard package, a fresh one is prepared
every time for every job. (This prevents 'tribal knowledge' from
propagating.)

2- The QA package is approved up the chain of command. While all
packages are carefully scrutinized, the approval process can be quite
adversarial for non-routine or complex tasks.

It's worth noting that pen-and-inks or white out is not allowed on
these documents. *Any* fault leading to rejection means the package
must be prepared again and resubmitted again in toto, starting at the
bottom of the approval chain.

3- The work is performed, with the presence of QA personnel as
required. (Which can be quite intrusive, which is by design. One job
I did routinely required stowage of parts removed during the
procedure. I had to show him that the parts were properly bagged,
tagged, and stowed. The nuts were of a special design, and he had to
watch me remove them and independently inspect them before bagging and
tagging.)

4- Closeout testing is done, in the presence of QA personnel, who are
certified for the general task at hand. (I.E. the QA guys who worked
ships systems were not the same guys who worked QA on the weapons
systems, and there was a separate pool for nuclear work. There was
some overlap in individuals, but qualification for each task group was
independent.)

5- Completion of the task, along with the QA package are re-submitted
up the chain for review.

D.
--
The STS-107 Columbia Loss FAQ can be found
at the following URLs:

Text-Only Version:
http://www.io.com/~o_m/columbia_loss_faq.html

Enhanced HTML Version:
http://www.io.com/~o_m/columbia_loss_faq_x.html

Corrections, comments, and additions should be
e-mailed to , as well as posted to
sci.space.history and sci.space.shuttle for
discussion.

(Derek Lyons) writes:

rk wrote:
Thoughts? This is an area that I'm thinking about a lot now.

Keep in mind that Naval Reactor and the SUBSAFE program both have two
things that *no* space program has, not even Soyuz, to wit:

An extensive practical background extending back over half a century,
and a large number of in service units.

They had also a thing space programs had: Accidents. AFAIR USS Thresher
was lost in 1963 because of QA issues. If there were as many space
crafts operating as submarines and budgets for them as for subs, things
might have been different. And of course there have been more
submariners been killed in accidents than astronauts.


Jochem

--
"A designer knows he has arrived at perfection not when there is no
longer anything to add, but when there is no longer anything to take
away." - Antoine de Saint-Exupery

Jochem Huhmann wrote:

(Derek Lyons) writes:

rk wrote:
Thoughts? This is an area that I'm thinking about a lot now.

Keep in mind that Naval Reactor and the SUBSAFE program both have two
things that *no* space program has, not even Soyuz, to wit:

An extensive practical background extending back over half a century,
and a large number of in service units.

They had also a thing space programs had: Accidents.

There has not been one single accident or loss related to Naval
Reactors or traceable to failure of SUBSAFE systems.

AFAIR USS Thresher was lost in 1963 because of QA issues.

Thresher was lost due to poor design, not poor QA, not directly. The
silver brazed joints, an acceptable practice at the time, could not be
reliably tested after installation. No amount of QA effort can
overcome what cannot be tested and verified.

D.
--
The STS-107 Columbia Loss FAQ can be found
at the following URLs:

Text-Only Version:
http://www.io.com/~o_m/columbia_loss_faq.html

Enhanced HTML Version:
http://www.io.com/~o_m/columbia_loss_faq_x.html

Corrections, comments, and additions should be
e-mailed to , as well as posted to
sci.space.history and sci.space.shuttle for
discussion.

rk wrote:
stmx3 wrote:


I think in the NASA community, there is some attempt at this
(viz ISO 900X), but it is either not rigorous enough or
standardized across the centers.


ISO? Did someone say ISO???

The CAIB report has something to say about ISO (which has been
discussed in sci.space.* previously, as far as its suitability to an
R&D environment)
[snip CAIB excerpt]

I missed that discussion earlier. Certainly NASA conducts R&D and
requires latitude in developing procedures and performing certain
operations. However, there are repetitive processes that lend
themselves to a standardized procedural base. NASA attempted to capture
this with ISO but did so with a broad brush stroke. ISO, or some
standard process, could still work in a wide range of areas at NASA, as
long as they pinpoint areas to which it shouldn't pertain.

Jochem Huhmann wrote:

They had also a thing space programs had: Accidents. AFAIR USS Thresher
was lost in 1963 because of QA issues. If there were as many space
crafts operating as submarines and budgets for them as for subs, things
might have been different. And of course there have been more
submariners been killed in accidents than astronauts.


Jochem


Loss of USS Thresher was a design issue. The submarine community
learned many valuable lessons, not the least of which was to minimize
amount of sea water piping on future subs. I don't believe QA played a
significant role.

I suppose you could say the loss of Apollo 1, Challenger and Columbia
were also design issues and NASA also learned many valuable lessons.
However, I think the NAVY has done a better job in crafting their
lessons learned into a much more credible safety and QA program. I'm
sure the funding has much to do with this.

Derek Lyons wrote:
[snip before]


4- Closeout testing is done, in the presence of QA personnel, who are
certified for the general task at hand. (I.E. the QA guys who worked
ships systems were not the same guys who worked QA on the weapons
systems, and there was a separate pool for nuclear work. There was
some overlap in individuals, but qualification for each task group was
independent.)

5- Completion of the task, along with the QA package are re-submitted
up the chain for review.

D.

Very informative post. Thanks for an in depth view. I would like to
see further posts contrasting this with NASA processes.

One point I've noticed relating to this is the effectiveness of QA. On
subs, the QA inspector for, say, work on a sea water pump, comes from
the same division (mechanical) that is performing the work. You would
think this is an incestuous relationship. Many times the QAI is the LPO
(Leading Petty Officer) for the division because of his experience. So,
when there's pressure to get a boat underway and everyone is waiting on
the seawater pump work to be completed, how does the process insure an
unbiased and independent inspection?

I bring up the issue because I've seen QA work on subs and at NASA and I
place more trust in submarine QA, by ORDERS OF MAGNITUDE (if you could
quantify trust).

My own initial answer was "integrity"...but that was off the cuff. I
think "pride" in a job well done comes in to play, but mostly I believe
it's "training". Of the NASA QA I have experience with, there members
can be called to any particular job site to sign off a procedural step.
They may have no idea why that step is being performed or how it
affects the system, but if the technician torques a nut to 7.5 ft. lbs
+/- .01 ft. lbs. in accordance with the procedure, then that's enough
for the QA stamp. In the NAVY, the QAI implicitly understands the work
being performed and can tell that, yeah...although the nut was torqued
properly, it was the wrong material or it was in the wrong position, etc.

Any thoughts?