International Skeptics Forum - View Single Post - How to manipulate data, get results as good as PEAR, and be $1,000,000 richer

Baron Samedi · 7th March 2007, 02:11 PM

Originally Posted by Cuddles

As an interesting thought that just popped into my head, there are often reports of medical trials that have been stopped early because of obvious negtive effects. It is possible that at least some of these could be due to this effect and not actually due to the treatment? Of course, if this does happen it would be false negatives rather than false positives, and with experimental drugs it is always better safe than sorry, but it would be interesting to know if we could be losing potential treatments due to this effect.

That's an interesting idea. I'll have to think about this some more, but I don't it would be as big of an issue. In the first case, you're looking for a high side effect, and you stop on a high side effect. In this case, you're looking for a high side effect, but stop on a low side effect. I think it's a wash.

For example, the tests are usually for no difference between drug A on the market and drug B which you just developed. You're thinking that we stop too early and say B is worse when in fact B is better. So if observed - expected is too small, we stop. This is assuming that expected is 0. If B is in fact better, then expected should be +5 or +10. So:

Z obs: (observed - 0)/se < -3.1 we stop, and p inflates from 0.001 to 0.02
We stop in error when: (observed - +i)/se < -3.1. However, the chances of making this mistake are much smaller than Z obs.

Or let's think of 100 coin tosses. I'll stop if the person get's 40% correct or worse. Therefore, the chances of me stopping the test assuming no ability are: Z = (.4 - .5)/(0.5/10) = -2, therefore p=2.5%
However, if the person really and truly had ESP and was up to 60% success rate, the probability that I would stop and say no evidence is now
Z = (.4 - .6)/sqrt(.6*.4/100) = -4, therefore p << 0.1%

I made my own head now. Ouch.

7th March 2007, 02:11 PM	#24
Baron Samedi Critical Thinker Join Date: Dec 2006 Posts: 476	Originally Posted by Cuddles As an interesting thought that just popped into my head, there are often reports of medical trials that have been stopped early because of obvious negtive effects. It is possible that at least some of these could be due to this effect and not actually due to the treatment? Of course, if this does happen it would be false negatives rather than false positives, and with experimental drugs it is always better safe than sorry, but it would be interesting to know if we could be losing potential treatments due to this effect. That's an interesting idea. I'll have to think about this some more, but I don't it would be as big of an issue. In the first case, you're looking for a high side effect, and you stop on a high side effect. In this case, you're looking for a high side effect, but stop on a low side effect. I think it's a wash. For example, the tests are usually for no difference between drug A on the market and drug B which you just developed. You're thinking that we stop too early and say B is worse when in fact B is better. So if observed - expected is too small, we stop. This is assuming that expected is 0. If B is in fact better, then expected should be +5 or +10. So: Z obs: (observed - 0)/se < -3.1 we stop, and p inflates from 0.001 to 0.02 We stop in error when: (observed - +i)/se < -3.1. However, the chances of making this mistake are much smaller than Z obs. Or let's think of 100 coin tosses. I'll stop if the person get's 40% correct or worse. Therefore, the chances of me stopping the test assuming no ability are: Z = (.4 - .5)/(0.5/10) = -2, therefore p=2.5% However, if the person really and truly had ESP and was up to 60% success rate, the probability that I would stop and say no evidence is now Z = (.4 - .6)/sqrt(.6*.4/100) = -4, therefore p << 0.1% I made my own head now. Ouch.