Original Citation. Alter, A.A., & Oppenheimer, D.M. (2008). Effects of fluency on psychological distance and mental construal (or why New York is a large city, but New York is a civilized jungle). Psychological Science, 19, 161-167.
Target of replication. Initially, Study 3 (the last study reported in the paper) was targeted for replication, however, it was concluded that a faithful replication of Study 3 was not feasible. Study 3 analyzed archival data that accrued over a period of seven years. To replicate this study, the archival data would need to be recreated using a new set of participants. The length of time and costs associated with this replication effort were deemed impractical. The results of Study 2B were very similar to Study 3 in terms of their theoretical meaning and were attained using a simple experimental procedure. Therefore, the decision was made to try to replicate Study 2B.
In Study 2B, conceptual fluency was operationally defined as whether participants were primed with the word “Los Angeles” (conceptually fluent) or not (conceptually disfluent) during a written exercise performed at the beginning of the experiment. Participants who were primed with Los Angeles later reported that they preferred written descriptions of the city of Los Angeles that were more concrete (e.g., “a dry temperate city” versus “a tangle of freeways”) compared to participants who were not primed with Los Angeles. The present study is a direct replication of this finding using nearly identical methods.
A priori replication criteria. Participants primed with Los Angeles were compared with participants not primed with Los Angeles in terms of what proportion of them preferred concrete descriptions of Los Angeles. The original study found that 53% of primed and 41% of unprimed participants preferred concrete descriptions, chi-square (1, N = 236) = 3.83, p = .05, prep = .92, Φ = .13. A successful replication would observe a similarly positive and significant chi-square value.
Materials, Data, and Report. A full report containing images of the materials that were used can be found in the Study Materials node. This report is also contained in the Report node. A copy of the SPSS dataset is available in the Data node.
Materials consisted of a priming exercise and a description rating exercise. Both exercises were attained form the first author of the original study. The priming exercise presented participants with a short article that rated six city’s baseball stadiums in terms of “comfort,” “hot dogs,” and “field quality.” Participants were then asked which city’s stadium they thought the players enjoyed most, which city’s stadium they thought the fans enjoyed most, and how avid of a baseball fan they were. Two versions of this exercise were used. The versions were the same except that one version contained Los Angeles as one of the six cities and the other version contained Detroit instead. Thus, participants who received the first version of the exercise were primed with “Los Angeles” whereas participants who receive the other version were not. As this relates to conceptual fluency, participants who were primed with Los Angeles were more conceptually fluent with Los Angeles than participants who were not primed with Los Angeles.
The description rating exercise asked participants to rank order eight different descriptions of the city of Los Angeles (1 = best description, 8 = worst description). Four of the descriptions were relatively concrete (e.g., “a dry temperate city”) and the other four were relatively abstract (e.g., “a tangle of freeways”). Participants who on average ranked the concrete descriptions more superior than the abstract descriptions were classified as concrete-preferring. Participants who on average ranked the abstract description more superior than the concrete descriptions were classified as abstract-preferring. The original paper did not specify what should be done with participants who on average ranked the abstract and concrete descriptions equivalently. I contacted the first author about this and he said that a small number of participants did indeed show no preference in the original study and were put into a third category (i.e., no preference). This no preference category was merged with the abstract-preferring category for analysis (thus, the single degree of freedom Χ2 test reported in the original paper). In sum, participants were divided into two categories: those who preferred concrete descriptions and those who did not (i.e., they either preferred abstract descriptions or exhibited no preference).
The entire study was conducted over the internet using SurveyMonkey® web-based survey software. Participants first read an informed consent agreement and clicked a “Next” button if they agreed to participate in the study. Next, they were presented with one of the two versions of the priming exercise at random. Finally, they completed the description rating exercise. Before ending the study, participants completed a demographics questionnaire and answered some questions pertaining to the experiment. Specifically, participants were given a recall task to determine whether they could recall which cities were shown during the priming exercise. Participants were also asked what type of device they used to complete the study (e.g., laptop computer, smartphone) and whether there were any issues with the image that was shown during the priming task (e.g., whether it was blurry, whether it was too big to fit on the screen). These last questions were asked to control for differences in presentation quality that could have resulted from the devices that participants used to access the study.
Conclusions. Alter and Oppenheimer (2008) found that participants who were primed with Los Angeles (conceptually fluent) were more likely to prefer concrete descriptions of Los Angeles compared to participants who were not primed with Los Angeles (conceptually disfluent), fluent = 53%, disfluent = 41%, Χ2(1, N = 236) = 3.83, p = .05, Φ = .13. In the present study, the difference between fluent participants (39.6%) and non-fluent participants (37.8%) was non-significant, Χ2(1, N = 1146) = .387, p = .53, Φ = .02.
I next conducted the same analysis, this time limiting the sample to participants who reported no image problems (N = 1047). I added this limitation with the idea being that participants who experienced poor image quality might not have been properly exposed to the priming stimulus. Again, I observed a non-significant difference between the two fluency conditions, fluent = 40.1%, disfluent = 38.2%, Χ2(1, N = 1047) = .42, p = .52, Φ = .02.
I conducted the analysis a third time, this time limiting the sample to participants who both reported no image problems and who reported accessing the study using either a desktop or laptop computer (N = 1011). I added this second limitation with the idea being that participants who used small devices to access the study might not have been properly exposed to the priming stimulus (e.g., the word “Los Angeles” might have been cut off the screen). Again, I observed a non-significant difference between the two fluency conditions, fluent = 40.3%, disfluent = 37.9%, Χ2(1, N = 1011) = .62, p = .43, Φ = .03.
Finally, I limited the sample based on how long it took participants to complete the study (in addition to the two limitations discussed above). The median completion time was 294 seconds (approximately five minutes). I excluded participants whose completion times placed them in the lower 10% of the sample (i.e., less than 164 seconds). I made this exclusion with the idea being that participants who completed the study too quickly might not have been adequately exposed to the priming stimulus and/or might not have paid adequate attention to the description ranking exercise. Again, I observed a non-significant difference between the two fluency conditions, fluent = 40.3%, disfluent = 39.7%, Χ2(1, N = 912) = .03, p = .86, Φ = .00.
In contrast to Alter and Oppenheimer (2008), I did not observe a statistically significant difference between the two experimental conditions in my replication attempt. That is, participants who were primed with Los Angeles (fluent) did not report that they preferred concrete descriptions of Los Angeles more than participants who were not primed with Los Angeles (disfluent). I therefore concluded that my replication attempt failed to replicate the result reported by Alter and Oppenheimer (2008).
Although the difference I observed was non-significant, it was in the same direction as that reported in the original study (replication: 40% versus 38%; original study: 53% versus 41%). Additionally, my replication attempt differed from the original study in that I conducted my replication attempt over the internet, whereas the original study was conducted in person. It is therefore possible that the effect is either (1) not real, (2) real but very small, or (3) real and very small when the study is conducted over the internet and larger when the study is conducted in person.