Merged: Bem's latest experiments / Researcher says we can see the future

[fls]

Originally Posted by Arouet

Here's a paper that has apparently been submitted for publication to the JPSP critically looking at the Bem paper. One proponent on Skeptiko has already called it "a load of crap dressed up in a nice pdf package", but I thought that I would post it anyway, to see what the thoughts are here.

http://dl.dropbox.com/u/1018886/Bem6.pdf

I've read it. They cover a number of points which have already been made here about Bem's paper specifically and parapsychology in general. I might quibble over some of their details (for example, the choice of prior probability at 10^-20 seems excessive), but it is otherwise generally sound. An interesting discussion could be had over the extent to which psychology needs an evidence-based overhaul. This is something where we would benefit from the input of those within the field here. For example, it's not enough to show that researchers write up their results in a way that puts them into a better light than they deserve. This happens in any field to some extent. What we want to know is the extent to which their colleagues are fooled by this. In medicine, part of the evidence-based overhaul included putting the ability to recognize this sort of trickery into the hands of physicians, so that the way in which research is written up doesn't necessarily reflect what practicing physicians take away from that research.

What specific criticisms have been leveled at it?

Linda

[Arouet]

Originally Posted by fls

What specific criticisms have been leveled at it?

Linda

There were only a few posts on this in that thread since it was a hijack. I may start a discussion thread on it once I've had a chance to read the paper in more detail.

Just wanted to say that I appreciate your analysis of these matters. Your detial is fantastic, even though you make my head spin sometimes!

[GnaGnaMan]

I, too, find nothing substantially wrong with it. Caveat: If there is anything wrong with the Bayes factors they calculate, I wouldn't have noticed it.
I'm eager to hear the counter arguments.

Two (scientist) bloggers have also written critiques which raise many of the same points, not quite coincidentally, I guess.

http://www.psychologytoday.com/blog/...n-precognition

http://www.psychologytoday.com/blog/...rs-really-mean

http://www.psychologytoday.com/blog/...omena-research

[Arouet]

This discussion, as well as the one about the dog experiment seems to highlight just how vulnerable these types of studies are to statistical manipulation. Frankly, for a lay person, how are we to differentiate between these different statistical interpretations? Everyone seems to be writing from a position of authority and accusing the others of not knowing what they are doing!

Is it even feasible for the scientific community to reach a consensus on the proper way to proceed? I can't imagine that this problem lies only with paraspychology. It must permeate into the mainstream sciences as well.

[Beth]

Originally Posted by Arouet

This discussion, as well as the one about the dog experiment seems to highlight just how vulnerable these types of studies are to statistical manipulation. Frankly, for a lay person, how are we to differentiate between these different statistical interpretations? Everyone seems to be writing from a position of authority and accusing the others of not knowing what they are doing!

Is it even feasible for the scientific community to reach a consensus on the proper way to proceed? I can't imagine that this problem lies only with paraspychology. It must permeate into the mainstream sciences as well.

Well, they did title their paper Why Psychologists Must Change the Way They Analyze
Their Data: The Case of Psi

and conclude with this:

Quote:

our assessment suggests that something is deeply wrong with the way experimental psychologists design their studies and report their statistical results. It is a disturbing thought that many experimental findings, proudly and confidently reported in the literature as real, might in fact be based on statistical tests that are explorative and biased. We hope the Bem article will become a signpost for change, a writing on the wall: psychologists must change the way they analyze their data.

My conclusion is that BEM's evidence for psi in this paper is equal to or better than the evidence presented in many psychological papers for less controversial theories.

[fls]

Originally Posted by Arouet

This discussion, as well as the one about the dog experiment seems to highlight just how vulnerable these types of studies are to statistical manipulation. Frankly, for a lay person, how are we to differentiate between these different statistical interpretations?

That's a good question. I dont have an answer for you.

Quote:

Is it even feasible for the scientific community to reach a consensus on the proper way to proceed? I can't imagine that this problem lies only with paraspychology. It must permeate into the mainstream sciences as well.

Not so much. Speaking about medicine...you might find multiple comparisons in exploratory research, but it won't be reported as the main effect and will be followed up with confirmatory research, sometimes as part of the same article. Failure to include control groups is considered a fault. Analyses are based on the empirical information, rather than the expected results of random-number generators. For example, drug trials are analyzed based on how many people were actually randomized into each group, not on the 'mean chance expectation' that equal numbers would be in each group. Failure to include control groups is considered a fault. Much work is put into establishing prior probability so that experiments are generally performed in the setting of even odds. And a failure to include control groups is viewed as a fault.

I don't know the state of affairs in psychology, but it would be interesting to hear from people who would know.

Linda

[blobru]

Originally Posted by Paul C. Anagnostopoulos

I thought folks might be interested in Daryl Bem's recent experiments, to be published in Journal of Personality and Social Psychology.

http://dbem.ws/FeelingFuture.pdf

~~ Paul

Rereading Bem's write-up a little closer, I'm noticing what may be a basic flaw with experiments 3 & 4: ("Retroactive Priming I & II"). In each experiment, there are 32 trials: 16 congruous and 16 incongruous. The data seem to show that participants categorize quicker with congruous than incongruous 'retro-primes'. But note that, given the even split between congruous and incongruous pairings of picture-affect and retro-prime, it is more likely that a congruous pairing will have been preceded by an incongruous pairing, and vice versa. Allowing for the first pairing, which is either congruous or incongruous and preceded by nothing, on average the participants will experience an extra incongruity (incongruous to congruous or congruous to incongruous VS congruous to congruous or incongruous to incongruous) between pairings. Iow, each participant on average will see and be asked to categorize one more congruous pairing after seeing and categorizing an incongruous pairing than congruous after congruous; or, see and categorize one more incongruous pairing after a congruous pairing than incongruous after incongruous.

Note especially that this means that on average for every two participants there will have been one extra congruous pairing followed by an incongruous pairing, and one extra incongruous pairing followed by a congruous pairing.

So what? Well, given this, we can easily construct an alternate, non-psi hypothesis: that the difference in response times is not a psi effect; but rather, an averaged measure of the effect of categorizing the extra congruous after incongruous pairing and the extra incongruous after congruous pairing. That is, it may be that anomalies in participant response-times are tied to these extra incongruities between pairings (affect & prime) over the run of the experiment, and not to psi-retro-primed congruities or incongruities within the pairings themselves (as a possible non-psi mechanism: having just seen an incongruous pairing, where the picture and prime don't match, the participant may experience some tension, and be more likely to categorize the next picture -- which is more likely to be part of a congruous pairing -- more quickly in order to move on from the tension; conversely, having just seen a congruous pairing, the participant may be pleased, and be less likely to categorize the next picture -- more likely to be part of an incongruous pairing -- more quickly, not wanting to move past the lingering pleasure).

[fls]

Originally Posted by blobru

Rereading Bem's write-up a little closer, I'm noticing what may be a basic flaw with experiments 3 & 4: ("Retroactive Priming I & II"). In each experiment, there are 32 trials: 16 congruous and 16 incongruous. The data seem to show that participants categorize quicker with congruous than incongruous 'retro-primes'. But note that, given the even split between congruous and incongruous pairings of picture-affect and retro-prime, it is more likely that a congruous pairing will have been preceded by an incongruous pairing, and vice versa. Allowing for the first pairing, which is either congruous or incongruous and preceded by nothing, on average the participants will experience an extra incongruity (incongruous to congruous or congruous to incongruous VS congruous to congruous or incongruous to incongruous) between pairings. Iow, each participant on average will see and be asked to categorize one more congruous pairing after seeing and categorizing an incongruous pairing than congruous after congruous; or, see and categorize one more incongruous pairing after a congruous pairing than incongruous after incongruous.

Note especially that this means that on average for every two participants there will have been one extra congruous pairing followed by an incongruous pairing, and one extra incongruous pairing followed by a congruous pairing.

So what? Well, given this, we can easily construct an alternate, non-psi hypothesis: that the difference in response times is not a psi effect; but rather, an averaged measure of the effect of categorizing the extra congruous after incongruous pairing and the extra incongruous after congruous pairing. That is, it may be that anomalies in participant response-times are tied to these extra incongruities between pairings (affect & prime) over the run of the experiment, and not to psi-retro-primed congruities or incongruities within the pairings themselves (as a possible non-psi mechanism: having just seen an incongruous pairing, where the picture and prime don't match, the participant may experience some tension, and be more likely to categorize the next picture -- which is more likely to be part of a congruous pairing -- more quickly in order to move on from the tension; conversely, having just seen a congruous pairing, the participant may be pleased, and be less likely to categorize the next picture -- more likely to be part of an incongruous pairing -- more quickly, not wanting to move past the lingering pleasure).

That's a good point. It's all part of the larger picture that these experiments almost all include an effect occurring in the usual direction, in addition to the proposed pre-cognitive effect. And this obviously larger effect has not been accounted for in the analyses.

Linda

[Paul C. Anagnostopoulos]

On Monday I again asked Bem for the data from experiment 1. I have not heard back from him.

~~ Paul

[Beth]

Originally Posted by Paul C. Anagnostopoulos

On Monday I again asked Bem for the data from experiment 1. I have not heard back from him.

~~ Paul

You might want to check your email address. I wrote him last week-end and got a form reply Monday. In it he states:

Originally Posted by BEM

Finally, some of you have asked for my raw data. I am a firm believer in providing access to all data—especially in such a controversial area. But it will take me a while to get to those requests. (And I have to strip the data files of all subject-identifying information). I am giving priority to the replication packages right now. So please be patient.

[Ersby]

That's good to hear. He sent me all the PRL ganzfeld data when I asked for them, too.

[Paul C. Anagnostopoulos]

Originally Posted by Beth

You might want to check your email address. I wrote him last week-end and got a form reply Monday.

The rest of the world can reply to my email address, so he may just have missed my emails or they never got to him. PM me the email address you used for him.

Why would the data files have subject-identifying information, other than subj1, subj2, etc.?

~~ Paul

[Paul C. Anagnostopoulos]

Beth: Have you heard anything more from Bem?

~~ Paul

[Beth]

Originally Posted by Paul C. Anagnostopoulos

Beth: Have you heard anything more from Bem?

~~ Paul

No

[Almo]

Just saw this.

http://www.newscientist.com/article/...published.html

Quote:

In another study, Bem adapted research on "priming" – the effect of a subliminally presented word on a person's response to an image. For instance, if someone is momentarily flashed the word "ugly", it will take them longer to decide that a picture of a kitten is pleasant than if "beautiful" had been flashed. Running the experiment back-to-front, Bem found that the priming effect seemed to work backwards in time as well as forwards.

This has passed peer review at the Journal of Personality and Social Psychology. The guy himself says it sounds goofy.

I'm not sure I'm impressed by it, but I found it interesting anyway.

[Brainache]

Researcher says we can see the future ...

Tomorrow...

[Gord_in_Toronto]

The Word of the Day is: Replication. I'll wait.

[Skeptical Greg]

'Saw that coming..

Oh, wait..

[rjh01]

I wonder how far into the future they can see? Maybe a few hours? Maybe it can be used to predict tatts results? Big money for all if it works.

[Jeff Corey]

There's another thread which was started a few weeks ago about that.

[Gord_in_Toronto]

Originally Posted by Jeff Corey

There's another thread which was started a few weeks ago about that.

Are you sure? I thought it was going to start next week.

[Steve001]

...and a Failure to Replicate the Same Results
http://papers.ssrn.com/sol3/papers.c...act_id=1699970

A preliminary study.

[Jeff Corey]

Originally Posted by Steve001

...and a Failure to Replicate the Same Results
http://papers.ssrn.com/sol3/papers.c...act_id=1699970

A preliminary study.

Thanks for the reference. Ever since Linda asked me to check this out a few weeks ago, I've been puzzling over a cleaner way to falsify this. I'm going to pick one of his tests and see if I can get some students interested in helping me run the systematic replication.
But I do have a niggle. Who will publish all the negative results?

[Robin]

Originally Posted by Gord_in_Toronto

The Word of the Day is: Replication. I'll wait.

Not to mention better experimental design.

[Paul C. Anagnostopoulos]

Originally Posted by Jeff Corey

Thanks for the reference. Ever since Linda asked me to check this out a few weeks ago, I've been puzzling over a cleaner way to falsify this. I'm going to pick one of his tests and see if I can get some students interested in helping me run the systematic replication.
But I do have a niggle. Who will publish all the negative results?

Well, you can certainly publish them here. But submit them to the same journal as Bem.

~~ Paul

[GnaGnaMan]

Originally Posted by Jeff Corey

Thanks for the reference. Ever since Linda asked me to check this out a few weeks ago, I've been puzzling over a cleaner way to falsify this. I'm going to pick one of his tests and see if I can get some students interested in helping me run the systematic replication.
But I do have a niggle. Who will publish all the negative results?

The original journal will be hard pressed to come up with an excuse not to follow up.

There's also this:
http://failuretoreplicate.com/
Failure to Replicate is, as the name suggests, a database of failed attempts to replicate previously-reported psychological findings.

It seems to have completely failed to take off. Quite a pity.

[Peristarkawan]

Originally Posted by Woomaster

If it were true that people could indeed foresee future events, could this be explained within a materialistic worldview? What do you think?

Possibly. You should visit Wikipedia and read up on the arrow of time. The basic gist is that there are many different methods we can use to distinguish the past from the future, but as far as we can tell, they all essentially boil down to thermodynamics.

If that is true, then it logically follows that entropy is the only real obstacle preventing us from predicting the future. We can easily see into the past because the past has lower entropy than the present, but it is difficult to see into the future because the future has higher entropy than the present. As an analogy, it would be like riding a bike -- going downhill is easy, but climbing up a steep hill takes a lot of work.

[Rramjet]

Originally Posted by Jeff Corey

Thanks for the reference. Ever since Linda asked me to check this out a few weeks ago, I've been puzzling over a cleaner way to falsify this. I'm going to pick one of his tests and see if I can get some students interested in helping me run the systematic replication.
But I do have a niggle. Who will publish all the negative results?

Yes of course! That is precisley how science should be conducted. Announce the predetermined results and only then undertake an experiment to find exactly according to what the experimental bias has been set up to predetermine. Nice. I knew there was something missing from my concept of scientific methodology!

[GnaGnaMan]

Originally Posted by blobru

Rereading Bem's write-up a little closer, I'm noticing what may be a basic flaw with experiments 3 & 4: ("Retroactive Priming I & II"). In each experiment, there are 32 trials: 16 congruous and 16 incongruous.

I think you misunderstand Bem on this point. My understanding is that the number of congruous vs. incongruous is not predetermined.

Nevertheless, that's an interesting idea you introduce. I'm pretty positive that it is possible to come up with an alternative explanation based on making the right assumptions about between trials priming.
The tricky thing is that these assumptions would have to match what is known about priming. The implication would be that a lot of mainstream work done, using a similar priming paradigm, would become worthless.

[Garrette]

Originally Posted by Rramjet

Yes of course! That is precisley how science should be conducted. Announce the predetermined results and only then undertake an experiment to find exactly according to what the experimental bias has been set up to predetermine. Nice. I knew there was something missing from my concept of scientific methodology!

Not that I am a scientist, but I suppose I approximate one as closely as you, but I think you completely misunderstand Jeff Corey's use of the term "falsify."

[roger]

Originally Posted by Garrette

Not that I am a scientist, but I suppose I approximate one as closely as you, but I think you completely misunderstand Jeff Corey's use of the term "falsify."

To be fair, I think he was referring to the last sentence, where Jeff questioned who would publish the negative results.

In reality, the world many of us try to live in, scientists very often have a very good idea about how an experiment is going to turn out. They, of course, are open to being wrong about it. Who hasn't read in a press release "We are very excited. This experiment confirmed our expectation that the quasar ....".

It's just a cheap shot at a professional.

[Garrette]

Originally Posted by roger

To be fair, I think he was referring to the last sentence, where Jeff questioned who would publish the negative results.

On re-reading, I have to agree.

My apologies to Rramjet.

Originally Posted by roger

In reality, the world many of us try to live in, scientists very often have a very good idea about how an experiment is going to turn out. They, of course, are open to being wrong about it. Who hasn't read in a press release "We are very excited. This experiment confirmed our expectation that the quasar ....".

It's just a cheap shot at a professional.

I can't improve on this response.

[technoextreme]

Originally Posted by Jeff Corey

Thanks for the reference. Ever since Linda asked me to check this out a few weeks ago, I've been puzzling over a cleaner way to falsify this. I'm going to pick one of his tests and see if I can get some students interested in helping me run the systematic replication.

You probably don't have to. Check the statistical tests used to reject the null hypothesis. If the criticism in Psychology Today is correct I think he ********** up the statistics.
EDIT:
SERIOUSLY?????? I was right. Really????? How the hell does an electrical engineer who has very limited experience with statistical analysis know more than a supposed scientist??? Wait I know.

Originally Posted by Gord_in_Toronto

The Word of the Day is: Replication. I'll wait.

Actually the word of the day is make sure there is an effect before replication.

[Beth]

Originally Posted by technoextreme

You probably don't have to. Check the statistical tests used to reject the null hypothesis. If the criticism in Psychology Today is correct I think he ********** up the statistics.
EDIT:
SERIOUSLY?????? I was right. Really????? How the hell does an electrical engineer who has very limited experience with statistical analysis know more than a supposed scientist??? Wait I know.

What criticism of the statistics are you talking about? The worst thing the linked article said was "some of the null hypotheses are rejected only by way of the flat-footed one-tailed test".

[technoextreme]

Originally Posted by Beth

What criticism of the statistics are you talking about? The worst thing the linked article said was "some of the null hypotheses are rejected only by way of the flat-footed one-tailed test".

Yeah if you knew anything about statistics its almost tantamount to faking the results. When you use the one tailed test you are making assumptions that your data will show you a specific result. In general you can't make that assumption. For example in medicine you can't assume that a new medicine will make you better. It could do worst. That in turn means you have to use a two tailed test or something similar than that. The issue with the two tailed test is that its actually harder to reject the null hypothesis as opposed to the one tailed test. There are lies, dam lies, and statistics and in this case he went for the easiest trick in the book to make your data look legitimate.

[Beth]

Originally Posted by technoextreme

Yeah if you knew anything about statistics its almost tantamount to faking the results.

Why would you say that? I'm reasonably well versed in statistics and I don't see it that way.

[technoextreme]

Originally Posted by Beth

Why would you say that? I'm reasonably well versed in statistics and I don't see it that way.

You make assumptions in a one tailed test that I can't possibly fathom being possible in this scenario especially when he admits he has no *********** clue what is going on. In that case the test should revert to a two tailed test but he did the statistics using the one tailed test.

[Gord_in_Toronto]

Originally Posted by technoextreme

<SNIP>

Actually the word of the day is make sure there is an effect before replication.

It's sure going to be damn hard to replicate if there is no effect.

[blobru]

Originally Posted by GnaGnaMan

I think you misunderstand Bem on this point. My understanding is that the number of congruous vs. incongruous is not predetermined.

On rerereading Bem's write-up even a little more closely... you're right. For "Retroactive Priming I" the number of positive and negative affects and primes is evenly split at 16 apiece, but the mix is randomized (which does make it possible to anticipate congruous or incongruous towards the end of the trials by 'card-counting' so to speak, but "Retroactive Priming II" appears to correct for this with open deck sampling).

Quote:

Nevertheless, that's an interesting idea you introduce. I'm pretty positive that it is possible to come up with an alternative explanation based on making the right assumptions about between trials priming.
The tricky thing is that these assumptions would have to match what is known about priming. The implication would be that a lot of mainstream work done, using a similar priming paradigm, would become worthless.

Possibly, but if so the alternate non-psi mechanism is not as obvious as my misreading implied (I'll have to think about it, but I think randomization should balance out the between-trials priming I was complaining about). In any event, the other experiments in Bem's study don't suggest any non-psi mechanism (that I can see), experimenters having moved on to replication (with one failure to already cited by Steve001).

[Beth]

Originally Posted by technoextreme

You make assumptions in a one tailed test that I can't possibly fathom being possible in this scenario especially when he admits he has no *********** clue what is going on. In that case the test should revert to a two tailed test but he did the statistics using the one tailed test.

The only different assumption is whether you are predicting a particular direction for the change. It's arguable about whether or not he made the prediction in advance, but he did cite previous non-retroactive experiments that established the expected direction. I don't think that sort of difference of opinion on technical details is in any way "tantamount to faking the results".