Skeptical Thinking: Do Your Own Research

Posted At : June 3, 2008 3:49 PM | Posted By : Admin
Related Categories: Myth Buster

I was just reading the book "Blink" by Malcolm Gladwell. It is quite interesting in that it examines the human ability to make [relatively] accurate snap judgements with only a tiny "thin slice" of information. Our subconscious mind picks up on related bits of information and assists the conscious mind in making a decision even though we might not fully understand why. The result of this is the much articulated "gut feeling" about something.

A section within "Blink" that discusses "priming" caught my attention. Psychological researchers have fun putting subjects through tests that have secret agendas. For example, you may take a simple test that asks you to make sentences out of groups of words. What you don't know though, is that a certain tone is generated by key words sprinkled throughout the test.

Subjects coming away from these test have been primed for behavior according to the wishes of the researchers. For example, half the subjects may be exposed to words that evoke aggressive behavior while the other half deals with calming, peaceful words. When given a subsequent task, the two groups will manifest the behavior type indicated by their word groups. This effect has been witnessed and documented time and again.

The book highlights one particular priming study conducted by Clause Steele and Joshua Aronson in 1995 (*). Steele and Aronson found that African American college students performed worse than expected if they were told that the test would measure their cognitive abilities. Steele and Aronson labeled this phenomenon as "Stereotype Threat". Simply being asked to identify their race before the test cut their performance in half!

Sounds amazing right? Could this really be true? Malcolm Gladwell then proceeds to ask hypothetical questions about whether standardized tests really are reliable indicators of the test takers knowledge and intelligence. Could this explain the so-called race gap in standardized tests? These results seemed too amazing and too politically correct for me, so my skeptical red flag was raised. I proceeded to do additional research.

Right away I discovered a web site that apparently has been created as a direct result of the Steele, Aronson research: http://reducingstereotypethreat.org/. That makes sense. Certain groups would certainly welcome these results in lending ammunition to their cause.

Further investigation revealed a published paper called "On interpreting stereotype threat as accounting for African American-White differences on cognitive tests" by Sackett PR, Hardison CM, Cullen MJ. Here is the abstract of that paper:

C. M. Steele and J. Aronson (1995) showed that making race salient when taking a difficult test affected the performance of high-ability African American students, a phenomenon they termed stereotype threat. The authors document that this research is widely misinterpreted in both popular and scholarly publications as showing that eliminating stereotype threat eliminates the African American-White difference in test performance. In fact, scores were statistically adjusted for differences in students' prior SAT performance, and thus, Steele and Aronson's findings actually showed that absent stereotype threat, the two groups differ to the degree that would be expected based on differences in prior SAT scores. The authors caution against interpreting the Steele and Aronson experiment as evidence that stereotype threat is the primary cause of African American-White differences in test performance.

Ah ha! Isn't that interesting. I knew the Steele, Aronson results (as reported in "Blink") were too convenient.

In short order, I ran across the website http://www.debatingracialpreference.org that discusses the issue in depth. Included on that website are excerpts from popular media that widely and politically correctly reported the results of the Steele, Aronson research:

(From "Secrets of the SAT," written by M. Chandler, broadcast 10/4/99, Boston: WGBH.)

At Stanford University, psychology professor Claude Steele has spent several years investigating the 150-point score gap between Whites and Blacks on standardized tests. Was the cause class difference, lower incomes, poorer schools, or something else?... In research conducted at Stanford, Steele administered a difficult version of the Graduate Record Exam, a standardized test like the SAT. To one set of Black and White sophomores, he indicated that the test was an unimportant research tool, to other groups that the test was an accurate measure of their verbal and reasoning ability. Blacks who believed the test was merely a research tool did the same as Whites. But Blacks who believed the test measured their abilities did half as well. Steele calls the effect "stereotype threat."

(Newsweek, 11/6/95, p 82)

In another experiment, when Blacks were told that they were taking a test that would evaluate their intellectual skills, they scored below Whites. Blacks who were told that the test was a laboratory problem-solving task that was not diagnostic of ability scored about the same as Whites.

And even in scientific journals:

(Wolfe, C.T. & Spencer, S.J. (1996). "Stereotypes and Prejudice: Their Overt and Subtle Influence in the Classroom," American Behavioral Scientist 40, 176-185)

Steele and Aronson (1995) found that when African American and white college students were given a difficult test of verbal ability presented as a diagnostic test of intellectual ability, African Americans performed more poorly on the tests than Whites. However, in another condition, when the exact same test was presented as simply a laboratory problem-solving exercise, African Americans performed equally as well as Whites on the test. One simple adjustment to the situation (changing the description of the test) eliminated the performance differences between Whites and African Americans.

But is all of this really true? It turns out, no.

As reported in popular media, these cursory descriptions of the research findings leave out an extremely important piece of information. All three quotes above misinterpret the results in an incorrect way, and they are all completely erroneous.

In fact, the black students in the control group did NOT perform "the same as Whites". They did NOT perform "equally as well as Whites on the test". In fact, the black students not subjected to Stereotype Threat performed "the same", meaning "as expected" according to their previously recorded SAT scores. The previously recorded verbal SAT scores of the test subjects averaged 603 for the blacks and 655 for the whites. The gap remains.

Yes, Steele and Aronson demonstrated that Stereotype Threat can widen the performance gap, but the absence of that threat certainly does not eliminate the gap as was widely reported in the media.

The purpose of this blog post is not racial in nature. It is to illustrate the idea that what you read may not be as true as you think. If you care about real knowledge and real truth, it pays to activate your skeptical eye and do your own research as necessary.

(*) Steele, C. M., & Aronson, J. (1995). Stereotype threat and the intellectual test performance of African-Americans