What's the difference between "cherry-picking" and "confirmation bias"?

[pnerd]

What's the difference between "cherry-picking" and "confirmation bias"?

Thanks in advance.

[athon]

Not much, really. You could probably use both terms interchangably in some circumstances. Cherry picking tends to convey a sense of deliberateness, as if somebody is actively selecting information that agrees with them. Confirmation bias tends to indicate a lack of awareness of conflicting information.

Athon

[Wolfman]

Also, both terms begin with a "c", but end with a "g" and a "s" (respectively).





You're welcome.

[athon]

Originally Posted by Wolfman

Also, both terms begin with a "c", but end with a "g" and a "s" (respectively).

Your day job is on Sesame Street, right?

Athon

[Wolfman]

Originally Posted by athon

Your day job is on Sesame Street, right?

Athon

Right up until the day that the show was brought to them by the letter "F".

[BPScooter]

I would accuse someone of "cherry picking" if they had masses of data and only chose to analyze or discuss those that confirmed the hypothesis, leaving the rest to be found by others. I would accuse someone of "confirmation bias" when, given few data, they chose to interpret them in a manner that supported the hypothesis without considering alternate explanations.

[Puppycow]

Like Athon said, cherry picking is deliberate, confirmation bias is unconscious or half-conscious.

[Jeff Corey]

I agree that cherry picking involves selecting the data after being collected to support some hypothesis (and the file drawer effect involves hiding the nonsupporting data), but confirmation bias occurs when one only tries to collect data that could support the hypothesis and avoids efforts to falsify it. The Wason card selection task demonstrates this quite nicely.

[Jeff Corey]

double post

[Denver]

I agree with Athon's explanation, but the question also enters as to when is it appropriate to use either?

Depending on the person, if I were trying to illuminate their own confirmation bias, I might instead use the term cherry-picking, because it sounds less technical, and it's a word they probably are already familiar with and so they would more quickly grok my meaning.

But since cherry-picking does also have that sense of deliberateness, I'd have to be careful to not come across as accusing them of purposeful self-deception (if I felt that did not apply).

[quadraginta]

Originally Posted by Denver

I agree with Athon's explanation, but the question also enters as to when is it appropriate to use either?

Depending on the person, if I were trying to illuminate their own confirmation bias, I might instead use the term cherry-picking, because it sounds less technical, and it's a word they probably are already familiar with and so they would more quickly grok my meaning.

But since cherry-picking does also have that sense of deliberateness, I'd have to be careful to not come across as accusing them of purposeful self-deception (if I felt that did not apply).

In general I don't associate the term "cherry-picking" with "self-deception", but rather a quite considered effort to mislead someone else, since it involves a conscious selection of data supporting a predetermined POV.

If not "confirmation bias", which I think of as more subtle and subconscious, I'd be more apt to label "purposeful self-deception" as "denial".

[GreyICE]

Confirmation bias is when you discard events that don't fit the hypothesis, in favor of ones that do. For instance, lets say you believe in astrology. There's 3-4 things that are supposed to happen today that are predicted by your horoscope. One of them definitely happens, and you think 'OH! The horoscope predicted that.' The other three don't. You forget about them. Eventually you've built up a firm stock of things the horoscope has predicted.

Cherry picking is basically the same thing, but deliberate. You can see it in a lot of the global warming threads, the deniers will chose one tree ring study or one weather station or one particular problem and try and argue global warming isn't happening because of a single weather station.

[Jeff Corey]

Originally Posted by GreyICE

Confirmation bias is when you discard events that don't fit the hypothesis, in favor of ones that do. For instance, lets say you believe in astrology. There's 3-4 things that are supposed to happen today that are predicted by your horoscope. One of them definitely happens, and you think 'OH! The horoscope predicted that.' The other three don't. You forget about them. Eventually you've built up a firm stock of things the horoscope has predicted...

That is technically termed "belief persistence - the tendency to hang on to beliefs in the face of contradictory evidence" (Weiten, Introductory Psychology, p.337)

[GreyICE]

Originally Posted by Jeff Corey

That is technically termed "belief persistence - the tendency to hang on to beliefs in the face of contradictory evidence" (Weiten, Introductory Psychology, p.337)

Not quite. The person believing in horoscopes has plenty of evidence they work. Why once they predicted that she would meet a man, but not to trust him. She met a guy she thought was cute, and three months later he cheated on her! And just last week it said that unexpected events could bring large windfalls, and the company she worked at announced that because of everyone's hard work, they'd give out extra bonuses this year!

Grasp the difference?

[Jeff Corey]

No. You said, "The other three don't. You forget about them." That the contradictory evidence.

[novaphile]

Where I struggle with the concepts, is where "cherry picking" might be appropriate.

For example, (please forgive my vague memory about the details, I read about this study a long time ago, and can't find any internet references):

[NB. I mentioned this study in a thread about IQ yesterday, so it's in my mind again.]

A post-war study on the effect of "school dinners" on IQ found that there had been no improvement in IQ scores for the children who participated in the school dinners programs.

A much later review of the raw data found that there were a small percentage of children whose IQ jumped more than 20 points, suggesting that there were circumstances where the provision of school dinners had generated (for those children) a significant benefit.

These days a similar study would try to control for factors like parents income, nutrition at home etc. to help identify possible factors for further investigation.

[novaphile]

Here's a nicely written up explanation of one of the most famous examples of "cherry picking" from a set of observations. Milikan's Oil Drop Experiment

[novaphile]

Richard Feynman made an interesting comment about biases in experimentation in regard to the Millikan experiment, reproduced here on Wikipedia: Millikan's experiment and cargo cult science

[athon]

Originally Posted by Jeff Corey

I agree that cherry picking involves selecting the data after being collected to support some hypothesis (and the file drawer effect involves hiding the nonsupporting data), but confirmation bias occurs when one only tries to collect data that could support the hypothesis and avoids efforts to falsify it. The Wason card selection task demonstrates this quite nicely.

I fold and defer to Jeff. His explanation is better.

Athon

[BPScooter]

In my professional life, I really try hard to avoid all bias. Perhaps the big thing here is intent, or will. If a person intends to deceive, they can use the tools of cherry-pick and what I'd call "drive the point" to create their own variation on the ultimate un-known. Yet a person with good intent may fall into error, for reasons of ignorance, envy, greed, jealousy, avarice, or any of the other sins.

So, once one realizes that all agendas are indeed someone's agendas, and we're all prone to the sins to some extent, where does science, as created by individual egoistic all-too-human people like us, go from here?

Post-modernist critics of science are invited to reply.

[BPScooter]

GreICE I think has hit the intentional or concsious/unintended or unconscious here. It's right to consider the attention even paid to the data. I'm sort of talking about these big flipping things like global warming or Bigfoot where the data are more or less public domain. On the other hand, my private notice of the numerous nice things my mother-in-law has done, through confirmation bias, I've failed to note and respond to. Entirely unconsciously.

[Jeff Corey]

Originally Posted by BPScooter

GreICE I think has hit the intentional or concsious/unintended or unconscious here. It's right to consider the attention even paid to the data. I'm sort of talking about these big flipping things like global warming or Bigfoot where the data are more or less public domain. On the other hand, my private notice of the numerous nice things my mother-in-law has done, through confirmation bias, I've failed to note and respond to. Entirely unconsciously.

i have seen no such distinction in the psychological literature on confirmation bias.

[Jeff Corey]

Originally Posted by athon

I fold and defer to Jeff. His explanation is better.

Athon

Actually, it wasn't mine. It was Peter Wason's, who coined the term.
Wason, Peter C. (1960). "On the failure to eliminate hypotheses in a conceptual task"

[Aitch]

As a matter of interest, how do you ensure, when filtering out 'irrelevant' data, that you don't lapse into 'cherry-picking'?

[Jeff Corey]

Originally Posted by Aitch

As a matter of interest, how do you ensure, when filtering out 'irrelevant' data, that you don't lapse into 'cherry-picking'?

I'm not sure any data are irrelevant. Making sure that data are reliably gathered and avoiding the threat to internal validity termed instrumentation error or checking on interobserver reliability before running a full blown experiment should help to avoid gathering "irrelevant data".
In other words, gathering data on whether your data collection produces reliable data is relevant.



'

[Jeff Corey]

Let me try a replication of Wason's original demonstration. I will give you a triplet of whole numbers that follows a rule. Your task is to figure out that rule by giving me some triplets and finding out whether they follow that rule from my feedback - I'll tell you yes or no.
The triplet is 2,3,5.

[pnerd]

Originally Posted by jeff corey

the triplet is 2,3,5.

.
7, 11, 13
17, 19, 23

[GreyICE]

Originally Posted by Jeff Corey

No. You said, "The other three don't. You forget about them." That the contradictory evidence.

Ah. Forgetting something represents an effort to ignore it. Thus, when you forget random details, like what exactly was in the news article you read last week, or who sang a song that you like, you've actively ignored it.

Or your brain has just discarded less useful information. One of the two...

[GreyICE]

Originally Posted by jeff corey

let me try a replication of wason's original demonstration. I will give you a triplet of whole numbers that follows a rule. Your task is to figure out that rule by giving me some triplets and finding out whether they follow that rule from my feedback - i'll tell you yes or no.
The triplet is 2,3,5.

8,12,17
9,17,33
7,11,13

[Jeff Corey]

Originally Posted by GreyICE

Ah. Forgetting something represents an effort to ignore it. Thus, when you forget random details, like what exactly was in the news article you read last week, or who sang a song that you like, you've actively ignored it.

Or your brain has just discarded less useful information. One of the two...

The two main sources of forgetting are interference (either proactive or retroactive) and encoding failure. For an example of the latter, which penny is right? http://www.dcity.org/braingames/pennies/
No effort to ignore that has occurred.

[GreyICE]

Originally Posted by Jeff Corey

The two main sources of forgetting are interference (either proactive or retroactive) and encoding failure. For an example of the latter, which penny is right? http://www.dcity.org/braingames/pennies/
No effort to ignore that has occurred.

But suggesting it must be one and not the other because one wishes to interpret the actions of certain people in certain ways is most certainly not a good idea.

[Jeff Corey]

Originally Posted by GreyICE

But suggesting it must be one and not the other because one wishes to interpret the actions of certain people in certain ways is most certainly not a good idea.

Who was suggesting what? You said forgetting was caused by "...an effort to ignore it...Or your brain has just discarded less useful information. One of the two."
Neither explanation has any validity.

[GreyICE]

Originally Posted by Jeff Corey

Who was suggesting what? You said forgetting was caused by "...an effort to ignore it...Or your brain has just discarded less useful information. One of the two."
Neither explanation has any validity.

Uh huh. Let me follow: You claim it's belief persistence (a tendency to hold onto beliefs even when evidence contradicts them). This is best evidenced in things like doomsday cults, where even multiple passings of the 'end of the world' date don't necessarily debunk it for the believers.

I point out that remembering things that came true, and forgetting things that never happened is more like discarding useless information in favor of 'useful' information (or information that gives the illusion of usefulness), and that that is an entirely different thing.

Since then, you've rationalized your use of the term 'belief persistence' repeatedly.

Would my summary be correct or incorrect, and would the quoted post be more rationalization, or a useful addition to this discussion?

[Jeff Corey]

Incorrect.

[fls]

Originally Posted by pnerd

.
7, 11, 13
17, 19, 23

This so perfectly demonstrates Jeff Corey's point that I smell a set-up.

Linda

[quadraginta]

Originally Posted by fls

This so perfectly demonstrates Jeff Corey's point that I smell a set-up.

Linda

It's not a set up. It's a real deal.

I can't play, 'cause if he's doing the original I know the answer, but it isn't about the answer, it's about how the answer is arrived at.

[Jeff Corey]

It's a prime example.

[Jeff Corey]

OK, it looks like there are no more takers, but all the triplets offered did fit the rule, which was "The triplets must be ascending numbers".
Wason's original triplet was "2,4,6" and approximately 80% of his subjects never generated exemplars that could falsify their hypotheses, like "6,4,2", or "1,2,223400012". This has been generally the case with subsequent replications.
I used 2,3,5 as the exemplar because I figured some people here would go the prime or fibonacci series route.
Wason called the avoidance of testing violations of the subject's hypothesis confirmation bias. And that surely is different from cherry picking, data mining or belief perseverance.

[AgeGap]

The Stanford Prison Experiment (1971)......Subjects volunteered by simply responding to a newspaper ad ...

Confirmation bias

[Jeff Corey]

Originally Posted by AgeGap

The Stanford Prison Experiment (1971)......Subjects volunteered by simply responding to a newspaper ad ...

Confirmation bias

How so? This is the education thread, and should be relatively free of half baked, uninformed opinions.
The OP was quite clear and a good question. Last week I used a similar question in the Critical Thinking final exam. It was to define confirmation bias, the file drawer effect and to tell how they differ.
A number of answers here would not have passed. Thankfully, none of the students said the difference was based on some conscious/unconscious fake dichotomy.
This whole issue came up on another thread a while back and UncaJimmy said I was being overly pedantic.
So be it.