In article , Dan Nagle
writes:

I think your post is a very nice example of the point I was making,
which, if you will remember, was:

"Funny how something at the 2-sigma level can be of great interest or
of no interest depending on point of view and expectations."

When the data are chosen from the point of view of being of great interest,
then they are of no interest.

I think this is the main point. Certainly no-one will claim a detection
from a 2-sigma result. The difference is that they set up their
experiment, took their data, and reported a 2-sigma result. This is
more significant than choosing the data BECAUSE they show a 2-sigma
result.

Even a 5-sigma result would be useless if you looked at a huge number of
possible data sets and chose the one which has a 5-sigma result.

If you choose a person and this person then wins the lottery, then that
is a significant result. Suppose that a psychic says that he can
predict the lottery, he predicts that a certain person will win, and
indeed that person wins. That would be strong evidence that he really
does have psychic powers. But consider that someone wins the lottery
every week. A news reporter who then visits the lottery winner and
pitches his report as a report about an extremely improbable event
doesn't have a good command of statistics.