Timescale of a trial

Maybe I should have started this in "Latest Commentary Issues", as it does relate to something in the current commentary, but I'm interested in Kramer's view, so I've put it here.

The question is, does an applicant have to be able to complete the test in a few hours? If so, doesn't this very seriously limit what sorts of claims can be tested? The relevant quote from the commentary -

I'm trying to find out just how silly this can get. I'm also trying to imagine how one would find a group of people with simultaneous headaches, and I'm unable to....

Click to expand...

OK, the case in point was silly. But with the last sentence, Randi seems to be saying that a (pseudo)medical claim can't be tested unless a sufficient number of suitable test subjects can be gathered together at one time, and probably to be implying that the applicant has to be able to decide whether or not the cure has happened within a very short time. For many "medical" claims, this simply may not be possible.

I'm thinking particularly of Homoeopath Syed, who is currently making an ass of himself by demanding that posters in the "General Scepticism and the Paranormal" forum area should write a protocol for him (in 10 hours!) and guarantee that it will be acceptable for the challenge. He claims that when he gives people homoeopathic remedies, certain things will happen.

Actually, he doesn't really seem to have much grasp of what is necessary to fulfill the terms of the challenge, and I anticipate him flouncing off in the huff before anything sensible can be worked out. However, is it actually practical for anything to be worked out with regard to that claim?

There are two problems. First, some homoeopaths declare that it could take them some time to be sure whether a person (or themselves) was responding to the remedy. If someone said they could tell for sure, but it would take several days, how might that be accommodated? Second, it would be necessary to do maybe 15 repetitions in order to reach the required level of statistical confidence that the applicant wasn't just a lucky guesser. Is it necessary that all these be done simultaneously, or packed into one day?

First, I can't necessarily see that the applicant needs to be supervised while he is making up his mind. After all, there is no restriction on what one is allowed to do to tell whether a given bottle is the real thing or the sham, so theoretically, is their any barrier to the applicant simply being given the test substance, real or sham, and told to go away and decide which it is?

Second, is there any barrier to the repetitions being done in series, one after the other? I ask because one of the H'pathy Forums crowd, Anna somebody, once thought about going for the challenge using herself as a test subject. She would take the remedy herself, and then declare whether or not it was the real thing. But it might take her a few days to make up her mind each time, and then she'd need a period to recover before she did it again. Thus the test would have taken probably several weeks.

In fact she backed off, ostensibly because she couldn't face the thought of "making herself so ill" so many times, even for a million dollars. (Yeah, right!) But would it have been possible to do the test?

It seems to me that it would be possible, so long as the blinding was rigorous enough. Published papers about homoeopathic trials have managed suitable systems, often involving an independent lawyer mailing out the remedy/sham and keeping the records of which was which, while having no contact with anyone in the study.

If a homoeopath could be found who was willing to put his mouth where his money is, would the JREF be able to work out a protocol around these restrictions? I would have thought so, personally, if the applicant was serious and sensible. However, if the guy with the headache cure wasn't going to be allowed to take the test unless enough people with headaches could be found on the same day, then I fear there may be serious difficulties for this type of claim.

Rolfe.

Yes & No

A test that CAN be completed within hours should be conducted as such. If the only way to test a claim is over time (as Randi is presently negotiating the protocol for a long-term test of homeopathy with a group of UK homeopaths), then that's how we do it.

But, if a test can be done in a few hours and the applicant insists that the only way to prove his claim is over time (or with hundreds of trials, in an attempt to show that even a slight statistical result is a positive result nonetheless), well, we consult our experts, and decide accordingly. Generally speaking, we'll allow the applicant some leeway in this area, but we won't "cater to vanities".

The most important thing to keep in mind is that each claim is different, and each claim requires its own test. Two tests for homeopathy, or dowsing, or ESP, may be quite different, depending on the specific nature of the claim and the applicant's proposed protocol. Remember, the JREF does NOT prescribe the test, and we have no real standards for testing a specific claim.
It is clearly stated in the Challenge rules that the APPLICANT proposes a test protocol that he/she belives will verify the claim.
That's how the process and the negotiations begin.

So, purely hypothetical proposal. Both myself and the JREF witness the preparation by independent manufacturers of the necessary homoeopathic remedies and "sham" preparations, and the appropriate labelling, and their delivery to an independent and mutually trusted lawyer.

The lawyer agrees to send or deliver to me, one at a time and on request, either the remedy I ask for, or a sham. I take my time and decide whether I got the requested remedy or not (any way I please), and notify another trusted third party, who does nothing but keep a record of my decisions. I then notify the lawyer that I'm ready for the next one, and he sends it to me. And so on.

We do this, say, 15 times (which I think would allow for sufficient statistical power). After I have delivered my 15 decisions, the lawyer who was sending out the material compares notes with the person recording my decisions, and the number of correct calls counted. If this makes it to a less than 1 in 1,000 chance that I was guessing, the preliminary phase has been passed.

Naturally, I pay all professional fees accrued by this process.

I'm asking because sometimes homoeopaths complain that it's impossible to satisfy the demands of the challenge, and it would be nice to have a simple and practical protocol that (probably, I'm not asking for guarantees here) meets the requirements.

This one lets them pick the remedy they think they can identify according to what volunteers they can scare up with what symptoms, or "prove" it themselves or on other volunteers or anything, and it lets them take their time about deciding whether the effect is there, without anyone being locked up in a room or having to take a long holiday or provide a number of people with the same disease simultaneously.

I think it's fair, I think it's secure, and I think it would work. And frankly, if I believed in homoeopathy I'd be rushing to get my application in quick before somebody beat me to it. The fact that they're all so coy and can never agree on what they think they can do says to me as clearly as anything that they know they can't do anything.

Rolfe.

Apply. Protocol negotiations do not begin until you have done that.

KRAMER said:

Apply. Protocol negotiations do not begin until you have done that.

Click to expand...

Didn't protocol negotiations begin with that EFT guy before he actually applied?

KRAMER said:

Apply. Protocol negotiations do not begin until you have done that.

Click to expand...

Don't be daft, I don't think it's possible to tell magic sugar pills from ordinary sugar pills.

But about every couple of months, we get somebody here who does. And invariably they demand that we give them a suitable protocol to go for the challenge. So we tell them again and again, go and read the rules, go and discuss this with Kramer. But the real killer is that they actually have no idea where to start. Syed was the pits. Give me one healthy person, and I will make them slightly ill with my ultra-dilute preparation. No idea who was to determine whether the person was slightly ill or not, or how it might be possible to say whether or not his magic potion had caused it. Or his second proposal, give me one person with a cold. I will observe the symptoms, give him a remedy, and in 20 minutes the cold will be gone. One subject, no blinding, and no agreement on who would decide whether or not the cold was gone.

Hopeless.

Now, personally, I'd dearly love to get a homoeopath to stand up to the challenge. If your currently-in-discussion proposal gets somewhere, I'll be right in the front row cheering. But mostly the ones that seem as if they might go for it are so dense that they can't even think of a protocol that wouldn't get laughed out of court.

It would be nice to be able to say to them, look, no guarantees, and you have to negotiate with Kramer and the JREF yourself, but if you propose something like this, it's probably a good start and doesn't have any obvious fatal torpedoes in it.

If you don't want to give a provisional, qualified nod to anything, well, fair enough. It would just have been nice to know if one was anywhere near the right lines.

Rolfe.

PS. Thanks for clarifying that 9 right out of 10 trials is acceptable for the preliminary test. That's actually a big help. I don't suppose you'd indicate what sort of performance would be required for the definitive test? (Anna Bryant, probably the least dim homoeo to get into this so far, was really insistent that she know that, what a dimbo, she was so convinced she could do it but also convinced that every time she got the real thing she'd make herself ill!!!)

Good point, Rolfe. I think there should be some tentative protocols available for the most likely potential claims, like dowsing, telling the future, homeopathy, tekinesis, etc, etc - Of course pending a finalizing negotiation in each case (homeopaths should understand the demand for individualization ), at least for preliminary testing. These should include suggestions for self-testing prior to applying.

We so often meet the question: What would I have to do to win the prize?

I'm sure members of this forum would be delighted to help in designing such protocols. All the JREF would have to do was endorse them.

Hans

Well, the confirmation of 9 out of 10 was very helpful. Hans, you remember Anna Bryant, don't you? She was very close to coming here and giving it a shot, and she was bright enough to follow what we were saying. It was just her reluctance to "make myself very sick" multiple times that seemed to be stopping her. And the fact that we couldn't say for sure how many trials she'd have to do didn't help.

Of course she demanded to know how many for the final test as well, because she was so sure she could pass the preliminary, but that it wasn't worth her while going through that if the number of tests for the two phases together was in her view impossible.

Whackjob, of course, but I wish she'd gone for it.

I was really interested to know if the postal distribution of the remedies held by a trusted neutral party would be acceptable, for a test where the only possible way to cheat would be to break the blinding. If it was, it would make the whole thing a lot more doable.

You'd just need to get the applicant to nominate a commercial homoeopathic pharmacy in his area, then find a few local sceptics who were prepared to act as witnesses to ensure that the remedies were prepared absolutely honestly. Both the applicant (and his mates, within reason) and (maybe the same number of) the sceptics would watch while the usual technicians did the diluting and succussing according to the applicant's preferred method. At the same time a sufficient quantity of identical bottles of stock carrier material would be prepared, the only difference being that they'd never been near a mother tincture. The applicant could choose which remedies were included, as many different ones as he liked, so long as they were all ultramolar.

An unbreakable random code would be agreed on, so that while the identity of the bottle dispensed could definitely be established after the trial, there would be no possibility of the applicant guessing what he'd been given (obviously this needs thought and negotiation). All the observers would make sure that all this was done honestly, and all the materials passed over to lawyer 1.

Everybody would go away. The applicant would decide which remedy he wanted to try first, either depending on the symptoms of any volunteer he could scare up, or what he thought he could "prove" on himself or a healthy volunteer. He would tell lawyer 1 what he wanted. Lawyer 1 would toss a coin, and depending on whether it was heads or tails (assignation of heads or tails to real remedy to be kept constant throughout the trial) he would send the applicant either the remedy he asked for, or one of the sham bottles.

The applicant would do whatever he liked with the stuff, presumably dose either himself or a volunteer with it, and decide by what he observed whether he'd been given the real thing or the sham. He would notify his decision to lawyer 2, who would not be in communication with lawyer 1. He would then wait until he was ready to try again, and repeat the exercise.

After he's had ten bottles to try, and made the ten decisions, the two lawyers would communicate, break the blinding, and announce whether he'd managed nine right or not.

Oh yes, and the applicant pays the professional fees for the lawyers and the homoeopathic pharmacy.

You could do this anywhere in the world so long as you could scare up a few sceptics who knew enough about homoeopathy to witness that the remedy preparation was on the level. And nobody would be under pressure, or have to rush anything. And it allows for any individualisation they like.

The only worry might be that somebody else might win the money before the ten trials were done, as far as I can see!

I'd like to be able to say to Syed and the rest of them, look, there is a protocol that Kramer might look favourably on. You have to negotiate the fine details, but that's at least an outline of somewhere to start that dosn't have any obvious no-nos in it.

But if Kramer doesn't feel he can give that sort of nod, well, he's the boss.

Rolfe.

Beleth said:

Didn't protocol negotiations begin with that EFT guy before he actually applied?

Click to expand...

No, that was just Randi engaging the guy informally and trying to figure out what in blazes he was talking about. Randi had never heard of EFT before, so he chose to engage him in a discussion about what might be an acceptable test.

No matter what is discussed prior to submission of an application, the official protocol negotiations begin following the submission of a protocol, and not before.

Don't nitpick. Randi is free to chat with whomever he likes.

KRAMER said:

No, that was just Randi engaging the guy informally and trying to figure out what in blazes he was talking about. Randi had never heard of EFT before, so he chose to engage him in a discussion about what might be an acceptable test.

No matter what is discussed prior to submission of an application, the official protocol negotiations begin following the submission of a protocol, and not before.

Click to expand...

Official protocol negotiations are indistinguishable from unofficial protocol negotiations, then. Telling some people to "apply or disappear" but discussing protocol with other people before they have applied appears, to us outsiders, to be a capricious adherence to your own rules. And doing so just causes you problems and stress. Trust me; I speak from experience.

Don't nitpick. Randi is free to chat with whomever he likes.

Click to expand...

Calm down. It was a question, not an accusation. Of course Randi can chat with whomever he likes. And it's a skeptic's job to nitpick.

I think your protocol can be simplified Rolfe.

Prepare the bottles of water and bottles of "water" as you say - lets imagine of just one type of remedy for the moment. A randomly chosen code is then associated with each bottle, in the presence of all the preparing parties, who then leave the room. An independent party with no knowledge of the coding then enters, and assigns their own random code to each bottle (not encrypted, completely random so therefore unbreakable), noting, of course, what the correspondence with the original code is. This independent parties' code is then actually affixed to each bottle.

Now ALL bottles are simply given to the homeopath, who can use any means whatsoever to determine which ones are active and which are dummies....

The homoeopaths won't go for that.

The protocol is designed to get round their perpetual griping about individualisation, and not wanting to repeatedly "prove" the same remedy on themselves in case they make themselves ill. (Real safe stuff this....)

This way, they can decide what remedy is the right one for the circumstances as they present themselves - volunteer with a cold, or a headache, or spondylitis or whatever, and know they've been given either that, or nothing.

When you try them with the "choose one remedy and stick with that", you've no idea the excuses they produce. If you let them go on with their usual practice, but just let them decide on a circumstance where they're sure they'll be able to tell the remedy which is the right individual one for the moment if they're given it, you avoid all that.

Actually, a friend of mine tried a variation of this. He asked for homoeopaths to volunteer who thought they for sure could prove any remedy they chose. Each one just had to say what remedy they wanted, and they were given either that, or the blank pills. The idea was to avoid complaints about one person repeatedly proving a remedy and doing harm. Each one only had to take one preparation, and make one choice. The hope was that by getting enough volunteers, the numbers would still be significant.

As there was no money involved, the controls were less rigorous, they just used commercially purchased remedies, and a couple of vets arranged the blinding and dispatch between them.

However, although he advertised it widely, he only got a handful of volunteers, not enough to publicise the results. Last I heard they were running 50% right, 50% wrong though, what a surprise.

Rolfe.

Rolfe said:

The homoeopaths won't go for that.

The protocol is designed to get round their perpetual griping about individualisation, and not wanting to repeatedly "prove" the same remedy on themselves in case they make themselves ill. (Real safe stuff this....)

This way, they can decide what remedy is the right one for the circumstances as they present themselves - volunteer with a cold, or a headache, or spondylitis or whatever, and know they've been given either that, or nothing.

When you try them with the "choose one remedy and stick with that", you've no idea the excuses they produce. If you let them go on with their usual practice, but just let them decide on a circumstance where they're sure they'll be able to tell the remedy which is the right individual one for the moment if they're given it, you avoid all that.

Actually, a friend of mine tried a variation of this. He asked for homoeopaths to volunteer who thought they for sure could prove any remedy they chose. Each one just had to say what remedy they wanted, and they were given either that, or the blank pills. The idea was to avoid complaints about one person repeatedly proving a remedy and doing harm. Each one only had to take one preparation, and make one choice. The hope was that by getting enough volunteers, the numbers would still be significant.

As there was no money involved, the controls were less rigorous, they just used commercially purchased remedies, and a couple of vets arranged the blinding and dispatch between them.

However, although he advertised it widely, he only got a handful of volunteers, not enough to publicise the results. Last I heard they were running 50% right, 50% wrong though, what a surprise.

Rolfe.

Click to expand...

I'm not saying do it with only one remedy - you can do it with multiple different ones! My point is a suitable double blinding/randomization can be trvially designed such that at the end of the day the homeopath has a bunch of yellow bottles, half of which "have" active ingredient X, a bunch of blue bottles, half of which have active ingredient Y and so on. They can individualize to their hearts content, or come up with any number of different ways of distinguishing them. Who cares. AT the end of the day they must, for example, turn over 10 bottles (or the codes for them rather) which are correctly identified. No need for writing away to other parties etc during the test...

I'm not sure I'm totally following you. You need to make sure what you suggest is in accordance with how homoeopaths actually practice. No use saying "do it like this", if they don't claim to work like that.

Also, we're talking about "experimenting" on volunteers. Doesn't your method require more than 10 patients?

Rolfe.

Rolfe said:

I'm not sure I'm totally following you. You need to make sure what you suggest is in accordance with how homoeopaths actually practice. No use saying "do it like this", if they don't claim to work like that.

Also, we're talking about "experimenting" on volunteers. Doesn't your mehtod require more than 10 patients?

Rolfe.

Click to expand...

All I'm saying is this. At the end of the day, using the process I outlined above, you can leave a homeopath sitting in their office with all their standard remedies. However, in any batch of remedies only half of them will actually be active. All the bottles will be labelled (with a label saying what type of remedy, and a random code which corresponds to whether its active).

Now, they can practise their standard homeopathic treatment. The only modification to whatever they normally do is whenever they use a particular remedy, they decide whether its actually active or not. In fact, since you've actually given them hundreds of bottles, they can have many "I dunno" moments - they dont need to decide on every bottle. Who cares? The only requirement is that at the end of X many weeks/months/years they hand over 10 of the bottles they're most certain about, and say "this bottle was ok, this one was dud etc".

How can this POSSIBLY not conform with they actually practise???

Do homeopaths believe that homeopathic remedies "go off"? That is, do they believe that they get "stale" after a period of time?

MRC_Hans said:

Good point, Rolfe. I think there should be some tentative protocols available for the most likely potential claims, like dowsing, telling the future, homeopathy, tekinesis, etc, etc - Of course pending a finalizing negotiation in each case (homeopaths should understand the demand for individualization ), at least for preliminary testing. These should include suggestions for self-testing prior to applying.

We so often meet the question: What would I have to do to win the prize?

I'm sure members of this forum would be delighted to help in designing such protocols. All the JREF would have to do was endorse them.

Hans

Click to expand...

I think Rolfe's suggestion of some guideline protocols is a good one. They could be integrated into the FAQ.

But, Hans, I don't think JREF needs to "endorse" them. The FAQ could simply state something like, "For testing <whatever>, here is a sample protocol that can serve as a starting point for designing your own individual protocol for subsequent Challenge negotiations."

PS: Kramer's snippy response to Rolfe was just plain uncalled for, IMO.

SezMe said:

I think Rolfe's suggestion of some guideline protocols is a good one. They could be integrated into the FAQ.

But, Hans, I don't think JREF needs to "endorse" them. The FAQ could simply state something like, "For testing <whatever>, here is a sample protocol that can serve as a starting point for designing your own individual protocol for subsequent Challenge negotiations."

My point for endorsing is that when "they" ask: "What will the JREF accept as proof?" We could answer: "Something closely resembling this."

PS: Kramer's snippy response to Rolfe was just plain uncalled for, IMO.

Let's face it: Kramer's snippy answers mostly are .

Kramer's feel for situations leaves much to be desired.

Click to expand...

Hans

Winny said:

Do homeopaths believe that homeopathic remedies "go off"? That is, do they believe that they get "stale" after a period of time?

Click to expand...

Generally not, although some controversy persists.

Hans

MRC_Hans said:

Generally not, although some controversy persists.

Click to expand...

Does that mean that the water retains the properties of anything that it has ever been in contact with? Surely they would see the absurdity of that?

Maybe it only retains the properties of the thing that they really, really want it to retain the properties of? Is there a magic spell when the water gets imbued with the appropriate property?

Winny