How many studies have been done and how many were positive?
I will give you an illustrative example using Dextromethorphan.
Looking only at controlled studies for acute cough (RCT's where appropriate)....
There were 14 studies in animals and all were positive.
There were 5 studies in humans using induced cough and all were positive.
There were 5 studies in humans with acute upper respiratory tract infection and 3 were positive (one of the positive studies was a metanalysis of 6 other studies, but I'm only counting it as 1 instead of 6).
Assuming that's true, homeopathy easily meets that standard.
It doesn't. I'll explain why using two different approaches.
First I have to digress a bit to explain what p<0.05 means, because you have made a common mistake (I am not singling you out as I have seen this mistake from many people, including scientists) and since I see this oft-repeated, this is my chance to set the record straight. <grin>
Many people say that a study with a p<0.05 means that it is more than 95% likely that the treatment is effective. In reality, the 95% doesn't necessarily say anything about the possibility that the treatment is effective. Most drug trials consist of taking a group of people, dividing them into two groups, giving all the people in one group a drug and all the people in another group a comparable placebo, and then looking for any differences in (pre-specified) outcomes days/week/months/years later. Now, anytime you divide people into two groups, you are likely to find differences between them due to chance. The 95% confidence interval recognizes that by coming up with a way to say, "yes the differences may be due to chance, but it is so unlikely that you should consider that there is
some other way in which the groups are different". All the p<0.05 says is that there is
some other way in which the groups are different. It says nothing about
what that difference may be. In a meticulously designed and executed RCT, the
only way that the groups are different is that one group is receiving a drug and one group is receiving a placebo. It is only then that one can say that the possibility that the treatment is effective is greater than 95%.
Any deviations from that ideal mean that we cannot be sure why the groups are different, so we need to look at
all the plausible explanations.
Common reasons for groups to be different - selection (the population that is selected for study), sampling (the method of choosing people from the population for the study), non-randomization (the way that people are divided into two groups), non-pharmaceutical treatments (attention, comfort, coercion, etc.), variation in measurement of outcomes.
If any of those reasons are present, then for all we know, our p<0.05 reflects bias rather than treatment effect.
Now for my first approach. We can let the results of the studies stand as reported and agree that the majority show a positive result (the groups are different in some way). But let's look a bit more closely at the demonstrated positive results.
Many of the studies do not involve an untreated group, so the positive result says "if you start with people who are unwell, give them drops of water, and then ask them how they feel a day/week/month later, many of them will state they feel better". However, we already know that "if you start with people who are unwell and then ask them how they a day/week/month later, many of them will state they feel better". There's no way to differentiate between those that took in gallons of water during that time period from those that took in gallons of water plus a few more drops during that time period.
Many of the studies involve two groups that are treated differently - one group gets attention, the psychological advantage of being told they are receiving a treatment, and a few drops of water. The other group gets none of this. We already know that people that get attention, and who are told that they are receiving a treatment feel better, regardless of whether or not they also receive a few drops of water.
Some of the studies are partially placebo-controlled - one group gets attention and both groups get the psychological advantage of receiving a treatment. We already know that people that get attention feel better regardless of whether or not they also are told that they are receiving treatment.
A few of the studies are fully placebo-controlled - both groups get the complete therapeutic package of attention and treatment, so that the only difference between the two groups is the presence or absence of a few drops of water. In this case, p<0.05 could be reasonable assumed to represent the treatment effect.
So what it boils down to is that the bulk of homeopathic research that is "positive" demonstrates that the groups are different in ways that we already know will create group differences through well-established conventional methods. There are a handful of studies that suggest that the differences could be due to treatment effects. But considering that all positive results will get published and only some negative results will get published, this handful could easily represent the few studies that we expect to see as positive due to chance.
And that does not even take into account that improper use of multiple comparisons and the use of homeopathic preparations that are not dilute (i.e. they contain active pharmacologic ingredients) accounts for some of these results.
My second approach refers to something I mentioned in a previous post - playing with a weighted coin. It is possible to generate a normal distribution with a weighted coin - it simply changes the mean and variance. I mentioned before that the study design/execution of most of the homeopathic studies introduces a bias similar to playing with a weighted coin. It is possible to compare the distribution you would expect from a weighted coin (in this case "the placebo effect") with the distribution of homeopathic studies and conventional medicine studies. In this type of analysis, it turns out that the distribution of homeopathic studies is not distinguishable from the distribution of the placebo effect, whereas the distribution of the conventional medicine studies is.
http://www.ncbi.nlm.nih.gov/entrez/...Retrieve&dopt=abstractplus&list_uids=16125589
So even though the homeopathic studies are "positive", they are not indistinguishable from a chance distribution taking the placebo effect into consideration.
Also, there is a strong correlation with "bias introduced during the design and execution of the study" and "magnitude of the result" in homeopathic studies which suggests that positive results are due to chance. If there was a true effect, there would not be a correlation between the two or it would be in the opposite direction (as you reduce the effects of noise)
Can you be more specific as to which studies you're talking about?
Power is affected by variance and the number of subjects. In studies where the variance is low and/or the number of subject is high (such as the animal and induced-cough studies I referenced above for Dextromethorphan), the power is high and none of the studies are negative. In studies where the variance is high, the power is low and some of the results are negative. That is, there is a correlation between the power and the results. That suggests that negative results are due to chance. If the medications were truly ineffective, then there would not be a correlation between the two.
Linda