Calculation of Shuttle 1/100,000 probability of failure

I have read that pre-Challenger, NASA actually calculated a 1/100,000
(or something like that) probability of catastrophic failure for the
shuttle. (I think it was mentioned in the book 'A Major Malfunction')

Ever since, I have wondered how NASA did that, and if the rationale
for such a calculation was published or not. It would make a great
case-study of scientific irrationality for some high-school science class.

Was there ever such a calculation, and is it published online anywhere?

I have read that pre-Challenger, NASA actually calculated a 1/100,000
(or something like that) probability of catastrophic failure for the
shuttle. (I think it was mentioned in the book 'A Major Malfunction')

Ever since, I have wondered how NASA did that, and if the rationale
for such a calculation was published or not. It would make a great
case-study of scientific irrationality for some high-school science class.

Was there ever such a calculation, and is it published online anywhere?



Dunno about that one, but I know in 1979 (when flights were planned to begin
initially) a failure analysis concluded that one in fifty flights would
encounter a catastrophic accident during ascent and one in 100 would fail to
land successfully (Jenkins, p 281) and that analysis for awhile didn't change.

I do find it interesting that the "one in 100" estimate of a failure to land
seems to have come pretty damn close.

-A.L.

(perfb) wrote in
om:

I have read that pre-Challenger, NASA actually calculated a 1/100,000
(or something like that) probability of catastrophic failure for the
shuttle. (I think it was mentioned in the book 'A Major Malfunction')

Ever since, I have wondered how NASA did that, and if the rationale
for such a calculation was published or not. It would make a great
case-study of scientific irrationality for some high-school science
class.

Was there ever such a calculation, and is it published online
anywhere?

The "rationale" was little more than management fiat, and was exposed as
such by Dr. Richard Feynmann in his appendix to the Presidential Commission
Report:

http://history.nasa.gov/rogersrep/v2appf.htm

--
JRF

Reply-to address spam-proofed - to reply by E-mail,
check "Organization" (I am not assimilated) and
think one step ahead of IBM.

The "rationale" was little more than management fiat, and was exposed as
such by Dr. Richard Feynmann in his appendix to the Presidential Commission
Report:

Too bad Feynmann died. He was DIRECTLY ON TARGET

"We have also found that certification criteria used in Flight Readiness
Reviews often develop a gradually decreasing strictness. The argument that the
same risk was flown before without failure is often accepted as an argument for
the safety of accepting it again. Because of this, obvious weaknesses are
accepted again and again, sometimes without a sufficiently serious attempt to
remedy them, or to delay a flight because of their continued presence."


HAVE A GREAT DAY!

"Jorge R. Frank" wrote

(perfb) wrote

Was there ever such a calculation, and is it published online
anywhere?

The "rationale" was little more than management fiat, and was
exposed as such by Dr. Richard Feynmann in his appendix to
the Presidential Commission Report:

http://history.nasa.gov/rogersrep/v2appf.htm

The thing I wonder is if, pre-Challenger, anyone ever took
the next step and said,

"OK, some people say the probability of loss is 1e-2 per launch
and others say it's 1e-5. The first is consistent with the
experience of other launch vehicles and engineering judgment,
the second is intermediate between launch vehicle experience and
mature aircraft experience.

"STS is supposed to be four shuttles, each with a design life
of 100 flights. If 1e-2/launch is right, there's a 98%
probability that we'll lose at least one orbiter. If it's
1e-5/launch, it's only 0.4%. That's the difference between a
show-stopper and a green light -- let's be very sure we know
which of the two choices is closer to being right."

Reports bearing on that, or even discussions of such matters
pre-1986, would be interesting to find and read.

"Allen Thomson" wrote in message
m...
"Jorge R. Frank" wrote

(perfb) wrote

Was there ever such a calculation, and is it published online
anywhere?

The "rationale" was little more than management fiat, and was
exposed as such by Dr. Richard Feynmann in his appendix to
the Presidential Commission Report:

http://history.nasa.gov/rogersrep/v2appf.htm

The thing I wonder is if, pre-Challenger, anyone ever took
the next step and said,

"OK, some people say the probability of loss is 1e-2 per launch
and others say it's 1e-5. The first is consistent with the
experience of other launch vehicles and engineering judgment,
the second is intermediate between launch vehicle experience and
mature aircraft experience.

"STS is supposed to be four shuttles, each with a design life
of 100 flights. If 1e-2/launch is right, there's a 98%
probability that we'll lose at least one orbiter. If it's
1e-5/launch, it's only 0.4%. That's the difference between a
show-stopper and a green light -- let's be very sure we know
which of the two choices is closer to being right."

Reports bearing on that, or even discussions of such matters
pre-1986, would be interesting to find and read.


Not sure exactly when he said it but in an interview with F. Borman (whom I
despised for what he did to EAL) he indicated that he was almost sure we'd
lose another one, in a different fashion. ( IIRC, this was right after 51-L)
It would appear he knew more about Shuttles than airlines...

T

The 1/100,000 mission reliability figure is a legacy of the sales hype that
we used to peddle the shuttle to the White House, Congress, the OMB and the
American public during the Phase A and B efforts in 1970-71 when we were
touting the shuttle's supposed "airliner-like operability" (I worked on the
shuttle TPS at McDonnell Douglas during that time). This, of course, was
pure crapola since at that time no reusable launch vehicle or spacecraft had
been developed and flown, so all these reliability numbers were pure
guesstimates. This excessively optimistic number was repeated during the
shuttle development period in several of the annual shuttle program reports
to the congressional oversight committees.

After the Challenger disaster, NASA tried to get a better handle on shuttle
reliability. I posted the following info more than a year ago, a few months
after the Columbia disaster. It may be of some help to clear up the
confusion concerning this topic:

So, what's the skinny on shuttle reliability, failure frequency, etc. in
light of the Columbia disaster? I checked some of the reports in my files
and Googled a bit to see if any sense could be made out of all the chatter
about probabilistic risk assessment (PRA), failure statistics, probability
of the next shuttle failure, etc. Here's what I've found so far.

The earliest info I could locate is the post-Challenger PRA calculations
done to satisfy the environmental impact requirements for the Galileo launch
in 1988 (the Galileo spacecraft had more than 50 pounds of plutonium
onboard). These are calculations of shuttle LAUNCH failure frequencies:

Oct88 Galileo PRA excluding 51-L:
5th-%ile Median Mean 95th-%ile
1/350 1/78 1/55 1/18

Oct88 Galileo PRA including 51-L:
5th-%ile Median Mean 95th-%ile
1/202 1/50 1/36 1/13

The median value is the failure frequency at the 50th percentile (half of
the statistical samples are greater and half are smaller that the median
value). For example, the first case can be interpreted as follows: with 90%
confidence, the shuttle launch failure frequency lies between 1/350 and 1/18
with 1/78 median and 1/55 mean if the 51-L failure is excluded. Also, the
median value is the usual "single-point" number used by statisticians as a
shorthand description of a statistical distribution. Note the relatively
wide spread (a factor of about 20 or more) in the failure frequency values
at the ends of the 90% confidence interval.

SAIC took another look at shuttle LAUNCH failure frequency in 1993:

Apr93 SAIC PRA excluding 51-L:
5th-%ile Median Mean 95th-%ile
1/223 1/90 1/73 1/31

Apr93 SAIC PRA including 51-L:
5th-%ile Median Mean 95th-%ile
1/118 1/52 1/44 1/21

Apr93 SAIC PRA excluding 51-L and SRB/RSRM failure frequency at 1/327
5th-%ile Median Mean 95th-%ile
1/394 1/134 1/83 1/30

For the last estimate, SAIC assumed that the post-Challenger improvements
incorporated in the redesigned solid rocket motor (RSRM) decreased the
failure rate of the SRB/RSRM to 1 in 327 (an educated guess). In the final
report, SAIC made further calculations that reduced the median launch
failure frequency to 1/145, a number frequently cited in late 1990s
literature on shuttle reliability. Also, the spread in the 90% confidence
interval has shrunk (it's a factor of 7 to10 in the SAIC calculations), but
I don't know if this has much significance, considering the approximations
and simplifications used in these calculations.

There have been some reports lately about an unpublished 1999 NASA analysis
of shuttle risk that calculated the LAUNCH failure frequency at 1/556
(median value). This sounds like something from an obscure NASA briefing,
but I haven't been able to locate the briefing charts. Need some help here.

So much for launch risk. What about the risk associated with the entire
shuttle flight regime (launch, orbit, reentry/landing)? In 1997 the National
Research Council was tasked to estimate the risk to the shuttle from orbital
debris and meteoroids. In doing this work, the NRC considered the entire
flight regime and came up with the following:

Without meteoroid and orbital debris
Ascent Reentry Debris Total
Median 1/248 1/350 na 1/145
Mean 1/219 1/326 na 1/131

With meteoroid and orbital debris
Ascent Reentry Debris Total
Median 1/248 1/350 1/200 1/84
Mean 1/219 1/326 1/200 1/79

NRC used 1/200 as the failure frequency associated with orbital debris and
meteoroids, a supposedly "worst case" value obtained from NASA, DOD and
industry experts. I don't know if there have been more recent updates for
this number and I don't whether NASA really agrees with it today.
Regardless, the NRC was pretty disturbed by the relatively high failure
frequency (1/84, median) that fell out of this calculation and tried to
light a fire under NASA to pay more attention to meteoroid and orbital
debris risk. In fact, every shuttle flight is analyzed by NASA for orbital
debris/meteoroid risks and the results are presented at a Cargo Integration
Review that's held about 12 months before launch. I assume that the CAIB
will make NASA do other things to mitigate this particular risk (inspecting
the TPS via milsats, using the ISS arms and cameras for closeup TPS
inspection, etc).

In that same unpublished 1999 NASA risk analysis mentioned above, the
failure frequency for a complete shuttle mission was estimated at 1/256
(median), according to media reports following the Columbia disaster.

In March 2001, Michael G. Stamatelatos, NASA's risk assessment manager, gave
a briefing on PRA that contained a chart with the following info for the
shuttle:

1997 2002 2007
2012
(current) (objective) (objective)
(objective)
Vehicle loss 1/148 1/250 1/325
1/500

The 1/148 number looks suspiciously like the 1/145 median value that NRC
calculated, ignoring the risk of orbital debris and meteoroids. Or maybe it'
s just a near coincidence.

I haven't been able to locate any more recent shuttle failure frequency
data, but there's probably more out there somewhere. The article in the
14Feb2003 issue of "Science" (pp. 1001-2), 2 weeks after the Columbia
disaster, mentions the Ulysses PRA, the later 1/145 number, a 1/245 number
from 1998, and has a few quotes from Mike Stametelatos, but otherwise adds
nothing new. Any help here is appreciated.

Forget about these PRAs for moment. What about the actual shuttle failure
frequency data? Well, there's only two random samples available to date, one
from Challenger (1/25) and one from Columbia (1/88). These two statistical
samples constitute a very meager data set as far as the statisticians are
concerned since they could have come from an infinite number of failure
frequency distributions. With only two samples, it's not possible to
determine the characteristics of the underlying shuttle failure frequency
distribution, i.e. the percentiles, the median value, the mean value,
skewness etc. The only recourse is PRA simulations that can run thousands of
statistical experiments encompassing hundreds of shuttle failures to produce
an estimate of the underlying shuttle failure frequency distribution.

So, what benefits have resulted from all of this PRA effort extending over
the past 15 years? Good question. Well, one question that's probably on a
lot of minds is what's the probability that another crew and vehicle will be
lost before the shuttle program ends. You can take a stab at answering this
by using the median failure frequency from whatever PRA you believe in and
assume an underlying statistical distribution, such as the binomial (=
Bernoulli) distribution. If the 1/148 number is taken as the median shuttle
failure frequency and it's assumed that the shuttle returns to operation in
Jan 2004 and flies through December 2020, then you can calculate the
following:

Launch failure frequency = 1 failure in 148
Launch failure probability = 0.0068
Year in which shuttle program ends 2020
Number of shuttle launches per year
2 4 6
Total number of launches (now to end of program) 34
68 102
Likelihood of at least 1 launch failure before end of program 20.6% 36.9%
49.9%

If you use the 1/84 number that fell out of the NRC analysis of orbital
debris/meteoroid risk to the orbiter, then these likelihoods become 33.4%,
55.7% and 70.5%, respectively.

Note: see http://www.airsafe.com/risk/shuttle.htm for the rationale behind
this calculation.

Personally, I'm not surprised that there could be a 50/50 chance of losing
another orbiter in the next 100 flights. I think most people have a gut
feeling that this is roughly correct. After all, there are several thousand
Criticality 1 failure points in the shuttle and nothing that NASA does to
fix the problems that caused the Columbia disaster will change this
fundamental fact. And, if you add the unknowns associated with aging of the
orbiter fleet, which is apparently causing some CAIB members to lose sleep,
then that ominous feeling becomes even more intense.

Later
Ray Schmitt






"perfb" wrote in message
om...
I have read that pre-Challenger, NASA actually calculated a 1/100,000
(or something like that) probability of catastrophic failure for the
shuttle. (I think it was mentioned in the book 'A Major Malfunction')

Ever since, I have wondered how NASA did that, and if the rationale
for such a calculation was published or not. It would make a great
case-study of scientific irrationality for some high-school science class.

Was there ever such a calculation, and is it published online anywhere?

Allen Thomson wrote:

"STS is supposed to be four shuttles, each with a design life
of 100 flights. If 1e-2/launch is right, there's a 98%
probability that we'll lose at least one orbiter. If it's
1e-5/launch, it's only 0.4%. That's the difference between a
show-stopper and a green light -- let's be very sure we know
which of the two choices is closer to being right."


Why would 98% probability of losing one shuttle have been a
showstopper?

--
Sander

+++ Out of cheese error +++

Sander Vesik wrote

Allen Thomson wrote:

"STS is supposed to be four shuttles, each with a design life
of 100 flights. If 1e-2/launch is right, there's a 98%
probability that we'll lose at least one orbiter.


Why would 98% probability of losing one shuttle have been a
showstopper?

Note the "at least."

Alas, probability isn't all that easy to think about, but look
it on a per-orbiter basis. Each one, with a 98% of launching
successfully, has a 0.98^100 = 13.25% = 1/7.5 chance of getting
through its nominal 100-launch life.