Can Citizens Be Framed? How Information, Not Emphasis, Changes
Transcription
Can Citizens Be Framed? How Information, Not Emphasis, Changes
Can Citizens Be Framed? How Information, Not Emphasis, Changes Opinions Thomas J. Leeper & Rune Slothuus Department of Political Science Aarhus University Bartholins Alle 7 8000 Aarhus C, Denmark E-mail: [email protected] / [email protected] Web: www.thomasleeper.com / www.ps.au.dk/en/slothuus March 20, 2015 Abstract Framing refers to a communicator’s presentation of a political issue and is widely seen as an effective way for elites to influence citizens’ opinions and behavior. The attractiveness of framing as a political strategy stems from the perception that it is relatively easy to focus people’s attention on an aspect of the issue, beneficial to the communicator. We present three studies that question the power of framing to shape public opinion. Existing studies confound framing with the provision of new information and argumentation. Our novel experimental design allows us— for the first time—to disentangle the separate effects of framing and information. Despite adjusting the design of each of our experiments in favor of the framing hypothesis, we find strong and consistent effects of information on opinions, but we find no effects of framing. These results have serious normative and practical implications for understanding elite influence on public opinion. A basic premise of representative democracy is that citizens can form and express their preferences for which public policies they want government to pursue and that elected representatives, in turn, respond to these public preferences (e.g., Dahl 1971; 1989: 109-112). Yet there might be a tension between the democratic ideal and political reality, because, as Disch (2011: 101) notes, “citizens’ capacity to form preferences depends on the self-interested communications of elites.” Indeed, fifty years of research on the influence of mass communication on opinion formation has demonstrated that politicians and other elites can powerfully shape citizens’ political preferences (see Kinder 2003). At the forefront of this literature is the repeated finding that “framing”—understood to mean a communicator’s selective presentation and interpretation of an issue or event—can have a sizeable impact on citizens’ opinions. For example, if a hate group rally is framed as a question of free speech, most citizens will tend to support allowing the rally to be held. In contrast, if the rally is framed as a threat to public order, citizens will tend to oppose allowing the rally (Nelson, Clawson, and Oxley 1997). Kinder (2003: 359) summarizes the framing concept clearly: “frames supply no new information. Rather, by offering a particular perspective, frames organize—or better, reorganize—information that citizens already have in mind. Frames suggest how politics should be thought about, encouraging citizens to understand events and issues in particular ways.” However, despite a “dramatic growth in framing studies” (Weaver 2007: 143-144), it is far from clear what this body of research tells us about how easily elites can “frame” public opinion. The reason for this is a mismatch between the theoretical definition of framing as a communicator or message recipients’ emphasis on subsets of issue-relevant considerations and the empirical paradigm for studying framing effects. A defining feature of most, if not all, existing framing studies is that they confound testing the impact of the framing of an issue with 2 the impact of providing substantive information and arguments (i.e., traditional argument-based persuasion; Scheufele and Iyengar 2015: 2; de Vreese and Lecheler 2012: 299). Experimental studies of “framing effects” typically involve messages that both frame an issue along a particular dimension and supply information and arguments about the issue (e.g., Boudreau 2015; Chong and Druckman 2007b; Druckman and Leeper 2012; Jerit 2009; Slothuus and de Vreese 2010; Sniderman and Theriault 2004). The same problem extends to observational studies of framing which assess frames in news coverage that also contains issue-relevant information (e.g., Hopkins 2014; Schemer, Wirth, and Matthes 2012; Slothuus 2010; Schuck, Vliegenthart, and de Vreese 2015).1 The consequences of this mismatch between theory and empirical practice are far more than semantic. Indeed, framing research has serious normative and practical implications for understanding how interactions between elites and the public work. Evidence for framing effects invites debate about the malleability of public opinion and the ability of citizens to participate meaningfully in democratic politics: “framing effects suggest that distributions of public preferences are arbitrary, and that political elites can manipulate popular preferences to serve their own interests” (Chong and Druckman 2007a: 120). If all a politician must do to influence citizens’ policy preferences is mention a dimension of the issue (or, in practice, stimulate media to feature that chosen dimension), elite influence can be achieved with ease.2 In this instance, it would be “highly rational for elites to pursue strategies based upon issue framing” because “framing as a political strategy involves minimal costs, and it has the potential to provide sizable 1 Scheufele and Iyengar (2015) describe the framing literature as in a “state of conceptual confusion” where “any attribute of information is treated as a frame and any response from the audience is deemed a framing effect. From this perspective, framing cannot be distinguished from other forms of media or social influence such as agenda setting, learning or persuasion” (also see Chong and Druckman 2007a: 115; de Vreese and Lecheler 2012: 299). 2 Some have interpreted framing effects as a deliberative process where citizens consciously weight alternative considerations emphasized by frames (Brewer 2001; Druckman 2001; Nelson, Oxley, and Clawson 1997). However, given the inability of previous framing studies to separate the effects of emphasis from providing substantive information, this remains unclear. 3 benefits” (Jacoby 2000: 751). In essence, “It may indeed often be easier to change frames of reference than the beliefs underlying one’s attitude” (Chong 2000: 118). This distinction mirrors the long-standing debate about the size and type of media effects on citizens’ political thinking and behavior. It is widely believed that media can influence public opinion only indirectly by changing the importance of citizens’ existing beliefs through processes of framing, priming, and agenda-setting whereas the media have limited capacity to provide information that will change the content of those beliefs. Framing is seen as politically important precisely because direct media influence is thought to be so difficult, but to date no study has distinguished the effects of framing from information. To clarify the effects of framing and information, we present a series of three experimental studies (combined n=2,299) that enable us to tease apart the separate influences of framing and issue-relevant information on political opinions. Our results show a strong and consistent effect of information on opinions but we find no effect of framing. And these results are consistent across the three studies despite our repeated attempts to carefully adjust the design in favor of the framing hypothesis. Given the normative and practical implications of framing research, these results invite us to reconsider our understanding of opinion formation, elite influence, and citizen competence. The Theory and (Limited) Evidence for Framing Effects A frame is generally understood to be “an emphasis in salience of some aspects of a topic” (de Vreese 2003: 27), it “suggests what the controversy is about, the essence of the issue” (Gamson and Modigliani 1987: 143), and it stresses “specific elements or features of the broader controversy, reducing a usually complex issue to one or two central aspects” (Nelson, Clawson, and Oxley 1997: 568; see also Druckman 2001; Kinder 2003; de Vreese and Lecheler 2012). Thus, a frame is typically thought of as a message that provides an interpretation of an issue or 4 policy by emphasizing which aspect of the issue is relevant for evaluating it, without the frame itself providing any new substantive information about the issue (Price and Tewksbury 1997; Nelson, Oxley, and Clawson 1997). Even though framing effects are thought to work by merely highlighting and emphasizing information and beliefs already present in a debate or the public’s minds, theorizing suggests frames are powerfully impactful. The reason is that citizens often possess a mix of considerations that could be used to form an opinion on an issue. These considerations might point in different directions, each pushing the individual to support or oppose the issue (Chong 1993; Feldman and Zaller 1992), thus “leaving citizens often confused and conflicted about where to stand… Frames help to resolve this confusion by declaring which of the many considerations is relevant and important, and which should be given less attention” (Nelson and Kinder 1996: 1058). Frames are therefore thought to shift public opinion via “framing effects” where during “the course of describing an issue or event, a speaker’s emphasis on a subset of potentially relevant considerations causes individuals to focus on these considerations when constructing their opinions” (Druckman and Nelson 2003: 730; also see Jacoby 2000: 750, 751). More formally, the psychological theory underlying framing effects draws on the expectancy-value model of attitude formation. In this model, an opinion towards an object (e.g., a policy) reflects the weighted sum of a set of evaluative beliefs about that object. Thus, opinion = Σ beliefi × weighti, where beliefi is the evaluative belief on dimension i and weighti is the subjective weight or importance the individual attaches to that dimension (Chong and Druckman 2007a: 105-106; Nelson, Oxley, and Clawson 1997: 225-228). For example, an opinion towards a social welfare policy might be the result of a positive belief that the policy will fight poverty (i.e., a reason to support the policy) and a negative belief that it will increase taxes (i.e., a reason to oppose the policy). Depending on the relative weight or importance an individual attaches to each 5 of these beliefs, overall opinion towards the policy might be positive or negative. Thus the strategic appeal of framing is that it requires comparatively less effort than following a strategy of argument-based persuasion aimed at convincing the public to change their beliefs (e.g., that a tax increase is worth paying for). It is thought to be far easier to remind citizens of particular preexisting beliefs when forming their opinion than it is to change those beliefs (e.g., Chong 2000: 118; Jacoby 2000: 751). Despite the clarity of emphasis framing theory, there is a remarkable lack of clean empirical testing for framing effects. In particular, empirical tests of framing rarely, if ever, follow the strict definition of only changing the emphasis or importance of an issue dimension. Instead, studies have typically assessed the impact of experimental manipulations of both information (e.g., factual policy information and persuasive arguments) and emphasis.3 A paradigmatic example is Druckman and Leeper’s (2012) study of opinion toward the Patriot Act, where the policy was framed as either about “weakening the protection of citizens’ civil liberties” or a way “to identify terrorist plots on American soil and to prevent attacks before they occur.” In addition to emphasizing these alternative dimensions of the issue, the experimental manipulations also presented study participants with different information about policy content—either that under the Patriot Act, “the government has access to citizens’ confidential information from telephone and e-mail communications” or, in the alternative presentation, that “the government has more resources for counterterrorism, surveillance, border protection, and other security policies.” This combination of framing and information leaves it an open question to what extent opinions in the study were swayed by framing (that is, emphasizing civil liberties versus 3 Notable exceptions are Berinsky and Kinder’s (2006) study of journalistic storytelling and Druckman et al.’s (2010) study of candidate evaluations. 6 emphasizing fighting terrorism) or the substantive information that was also provided along each of those dimensions. In another well-known experiment, the framing of a government spending proposal was varied by either stating that it “means higher taxes” or gives people “a better chance of getting ahead in life” (Sniderman and Theriault 2004). While these statements refer to different policy dimensions, it is difficult to know if opinion changes were caused by the mere emphasis on these dimensions or by potential changes in respondents’ underlying beliefs about the different consequences of the proposed spending in each condition.4 Reviewing the literature as a whole, it appears to be challenging to avoid supplying different arguments and additional information to participants in different framing conditions (e.g., Aarøe 2011; Boudreau 2015; Chong and Druckman 2007b; Druckman and Nelson 2003; Hartman and Weber 2009; Lecheler and de Vreese 2013; Nelson, Clawson, and Oxley 1997; Slothuus 2008; Slothuus and de Vreese 2010). If extant framing experiments show opinion change in response to political communications, does it matter if that opinion change is due to information and arguments rather than framing? We do not argue that the empirical findings of previous studies are invalid, but rather that they entail a mismatch between theory and design. In light of the operational confounding, interpreting these studies through the lens of framing theory is problematic given that they do not test framing in the strict sense that “frames organize—or better, reorganize— information that citizens already have in mind” (Kinder 2003: 359). Consequently, the valueadded of the framing concept itself is unclear from extant research because “we cannot be sure that there is anything truly ‘unique’ about this phenomenon; that framing cannot be subsumed 4 This operational confounding of framing and information is further confused by the sometimes reference to frames and arguments interchangeably (e.g., Chong and Druckman 2007b: 641; Druckman et al. 2013: 57; Jerit 2009: 412). 7 under some other generally understood concept, such as persuasion” (Nelson, Oxley, and Clawson 1997: 223). Moreover, the normative implications of the current framing literature is unclear. While influence through emphasis alone (i.e., framing) need not imply that citizens are irrational or that they fail to deliberate over political issues, opinion change without substantive information or belief change means the public can be easily coopted by the slightest of rhetorical shifts (see discussion by Druckman 2014: 474-475). A public that instead requires evidence and argumentation before changing its views implies a democratic process more closely aligned to popular conceptualizations of the democratic ideal of citizens forming “enlightened” preferences (Dahl 1989: 108-112). Properly evaluating framing requires disentangling the experimental confound. We therefore lay out two distinct empirical alternatives about elite influence on opinions. One relates to the effects of information provision or persuasive argumentation on opinions (i.e., changes in the content of beliefs) and the other relates to the effects of emphasizing particular considerations (i.e., changing the weights applied to different issue-relevant beliefs). We call the first the information hypothesis, which expects that information and arguments will shape beliefs about the policy and influence policy opinion in the direction implied by the evaluative content of the information or argument (Chong and Druckman 2007a: 115-116; Kinder 2003: 359). In contrast, the framing hypothesis expects the mere emphasis on a dimension or aspect of a policy will influence the importance of beliefs and change opinion in the direction implied by the content (positive or negative) of the evaluative beliefs emphasized (Nelson, Oxley, and Clawson 1997). Testing these two hypotheses requires a novel paradigm for studying elite influence. It is of course possible that both forms of communication may be influential; ours is the first study to provide such a clean test of each. 8 A New Approach for Studying Framing Effects To advance our knowledge of whether framing is a viable and effective political strategy to influence public opinion distinct from providing new information, we propose a novel methodological approach to operationalize and test issue framing in a form true to its theoretical definition. We argue that to assess the value-added of the framing concept, we need to independently evaluate the impact on opinions of mere emphasis on an issue dimension (i.e., framing) by separating it experimentally from the impact of policy information or argumentation.5 Very little, if any, extant research does this.6 Our experimental design isolates the manipulation of beliefs and framing at two different points in time (see Figure 1). In the first stage (t1), we manipulate information using the paradigm of persuasion research. As Kinder (2003: 367) explains, “Political persuasion entails the supply of arguments and evidence through which people are induced to change their minds about some aspect of politics” (emphasis added; see Chong and Druckman 2007a: 115; Zaller 1992: 118). We provide study participants with information and arguments on a policy issue that should lead them to form positive or negative evaluative beliefs on the issue. Depending on the evaluative content (or direction) of those beliefs, participants are expected according to the expectancyvalue model of attitudes to support or oppose the policy (given the change in the content of beliefs in), hence testing the information hypothesis. 5 As an alternative strategy to avoid the confounds between framing and related concepts, Scheufele and Iyengar (2015) propose to confine framing research to “equivalence” framing and visual manipulations. Equivalence framing, however, is not how issues typically are framed in political discourse—“It is difficult to satisfy this requirement of interchangeability of alternatives outside a narrow range of choices” (Sniderman and Theriault 2004: 135; see de Vreese 2003: 27)—and visual manipulations constitute a limited set of real-world communications. 6 Some might see question wording experiments as the closest empirical approximation of framing true to its name (Schuman and Presser 1996). The concern with many question wording experiments, however, is that they either (1) supply different information, as in a paradigmatic “framing” experiment, or (2) effectively ask survey respondents to evaluate two distinct attitude objects, rather than framing a single issue in two distinct ways (e.g., the seminal experiment asking about support for “assistance to the poor” versus “welfare” in fact asks about two entirely different policies; Huber and Paris 2013). 9 In the second step of our experimental design (t2), we manipulate framing in a manner true to its theoretical definition. Specifically, we manipulate the emphasis placed on alternative dimensions of consideration in a political debate to adjust which dimension of the policy (i.e., which evaluative belief) study participants should most heavily rely upon when forming their opinion. Because participants hold beliefs formed during t1, the manipulation of framing can be implemented purely by emphasizing one dimension of the policy and without supplying any new information or arguments. This way, we avoid the risk of further changing belief contents and thus create a clean test of the framing hypothesis.7 Based on extant work, we expect both information provision and framing to be effective communication strategies and we thus except to find support for both of our hypotheses. Yet it is important to note that, given the problems with previous studies of framing effects discussed above, we believe we are the first to conduct an empirical test of the framing hypothesis in its pure form.8 Study 1: A Test of Pure Framing We recruited participants for our survey experiment online using the Amazon Mechanical Turk (MTurk) crowdsourcing platform and randomized respondents into experimental conditions using Qualtrics. A number of recent studies indicate that while MTurk is not a representative online panel, the pool of respondents is more diverse than traditional convenience samples 7 Importantly, this temporal separation of information supply and framing is critical for disentangling the information hypothesis and the framing hypothesis. The information stage alone allows us to test the information hypothesis but also serves to provide an exogenous shock to participants’ issue-relevant beliefs. If information (and thus beliefs) were not manipulated, then framing effects may not manifest due to participants’ lack of relevant beliefs or beliefs that are insufficiently heterogeneous to display framing effects. Framing theory requires an individual to have beliefs in memory that can be emphasized, hence our choice to manipulate information before manipulating framing. For the same reason, we opted to not separate our information and framing phases by a long time window where the effects of information on beliefs might have decayed. 8 Perhaps the study most closely related to what we will do is Druckman and Bolsen (2011) who investigated whether the effect of a frame is enhanced if the frame contains factual information. However, they did not manipulate information and framing separately and hence did not distinguish the information and framing hypotheses. 10 (Berinsky et al. 2012) and results from experiments performed on MTurk closely match those performed on other samples and in other settings (Leeper and Mullinix n.d.; Mullinix, Druckman, and Freese n.d.). Table A1 in the Online Appendix reports the demographic composition of the study sample. A total of 750 participants took part in the experiment. The survey took about 10 minutes to complete and respondents were compensated $0.75 for their time. We next describe our design in more detail, including the selection of policy issue, how we manipulated information and framing, and how we measured our key variables. Policy Issue Our first task was to select a suitable policy issue. Given that our goal is to test the information and framing hypotheses, we use a real issue of modest political salience on which it should be possible to observe both an influence of information and an influence of framing. We intentionally avoid studying an issue on which opinions are likely to be crystallized due to evidence that these types of opinions are resistant to communication effects (see Druckman and Leeper 2012). One strategy for circumventing the crystallization problem is to examine opinion dynamics surrounding a fictional issue. To avoid sacrificing ecological validity, however, we focus instead on a health care policy topic, like those used in many previous studies (Fowler and Gollust 2015; Leeper 2014). Specifically, we focus on citizens’ support for the implementation of electronic medical records, which are a digital replacement for paper records typically maintained by physicians, clinics, and hospitals. The issue provides an excellent case for studying the influence of information and framing for several reasons. First, the topic is a real and substantively important policy issue. In recent years the estimated number of physicians and hospitals using electronic medical records have varied from 50% to nearly 80%, meaning that their use is not yet widespread or uncontroversial (Miller and Sims 2004; Ford et al. 2006). Additionally, Title XIII of the American Recovery and 11 Reinvestment Act of 2009 created financial incentives for the adoption of electronic medical records through Medicaid reimbursements (Blumenthal and Tavenner 2010). Second, the issue—at the time of the study—had not received significant, recent media attention. Thus, we avoid studying a more salient issue where citizens would possess more crystalized opinions. Third, and most importantly, electronic medical records raise numerous considerations, including whether they increase or decrease the cost of health care and whether they have any impact on the quality of patient care. We focus on these important and commonly debated dimensions of the issue—the policy’s fiscal cost and the policy’s impact on a particular target group—that tend to also be the important dimensions by which many policies are evaluated (see Jerit 2009). Time 1 Information Manipulation and Measures As already stated, the purpose of the first phase of the experiment was to manipulate information about the issue.9 At the beginning of the survey, experimental participants were told that they would participate in a study “about the quality of journalism” and that they would “read a few short excerpts from a news article.” We randomly assigned participants to one of four treatment conditions or a control group. In each treatment group, participants then read a sequence of four excerpts of an article created by the study investigators that described (fictional) discussions by the American Medical Association about the proposed use of electronic medical records. We intentionally wrote articles in journalistic style and masked the final purpose of the study so that participants were led to believe that they were simply evaluating journalistic quality. Each condition contained a different version of the articles. In the High Cost information condition, participants read information about the high costs of electronic medical records, while in the Low Cost condition other participants read information about the low costs of electronic medical records. In the High 9 The full stimulus material and all question wordings for all studies are available in the Online Appendix. 12 Impact condition, participants read information about the high (positive) impact of electronic medical records on health care, while in the Low Impact condition participants read information about the low impact of electronic medical records. In the control condition, participants read about an unrelated topic (an album being produced by a musical group). After each excerpt, participants were asked a factual question about the text they had just read which serves as an attention check and a mechanism for increasing participants’ engagement with the stimulus materials. As a manipulation check of the information treatments, we asked participants two questions to gauge their beliefs about the policy. Specifically, the Cost Belief question read: “The article you read discussed a proposal to standardize electronic medical records. From what you read, will health care costs be more expensive or less expensive under the proposed changes?” and recorded their answers on a seven-point scale from “much less expensive” (coded -1) to “much more expensive” (coded 1). The Impact Belief question asked: “From what you read, will health care quality be improved or harmed under the proposed changes?” and recorded their answers on a seven-point scale from “Harmed a lot” (-1) to “Improved a lot” (1).10 Participants then answered some general questions about the written quality of the excerpts (consistent with our cover story), answered some demographic questions, and then participated in an unrelated study of preferences for political candidates’ faces (which served as a distractor activity). We intentionally did not measure policy opinion during the t1 information phase to avoid any consistency biases in opinion reports during the framing phase of the study. Time 2 Framing Manipulation and Measures After completing the intervening material, participants then participated in the final part of the survey where we exposed them to frames about the policy. This part of the survey was 10 Participants in the control condition were not asked these questions. 13 introduced by telling participants that “The third study asks you for your opinion about a recent political issue” and then asked them to read a text that emphasized either the costs or health impact of electronic medical records.11 This framing manipulation was carefully crafted to not supply any additional information about the issue, but rather only emphasizes one dimension or the other. This provides a clean test of whether emphasis alone can move opinions. Participants were randomly assigned to receive either the Cost Frame or the Impact Frame: Cost Frame: “Recently there has been some debate in Congress about a new proposal regarding electronic medical records. Some support the proposal. Others are opposed because they say that we should judge the proposal based on whether it is costly.” Impact Frame: “Recently there has been some debate in Congress about a new proposal regarding electronic medical records. Some support the proposal. Others are opposed because they say that we should judge the proposal based on whether it will affect the health of average Americans.” This manipulation is fully crossed with the t1 information conditions, yielding a 5x2 factorial design with 10 experimental conditions in total. After reading the framing treatment, respondents were asked for their opinion on the electronic medical records proposal: “Given this information, to what extent do you favor or oppose this proposal?” and could supply their response on a fully labeled seven-point scale from “strongly favor” to “strongly oppose” (with “neither favor nor oppose” as the middle category). In the analysis, this measure was recoded to scale from -1 to 1, with higher scores indicating support. We also included a secondary dependent variable as an additional way of gauging framing effects. This measure allows us to test whether participants in the Cost (Impact) Frame condition 11 We use one-sided frames because we want to test the impact of framing when it is theorized to be most powerful. Previous work has found that the effects of opposing or competing frames tend to cancel (Chong and Druckman 2007b; Sniderman and Theriault 2004). 14 said that costs (impact on patients’ health) were a more important consideration when thinking about the issue (Druckman and Nelson 2003; Nelson, Clawson, and Oxley 1997). Specifically, we provided participants with a list of six ideas (“improving technology,” “costs,” “errors in medical records,” “the health of average Americans,” “the opinions of doctors,” and “patient privacy”) and asked them to rate the importance of each idea on a five-point scale from “not at all important” (coded 0) to “extremely important (coded 1). As a manipulation check of the framing treatment, we asked all respondents the following: “Please think again about the proposal regarding electronic medical records. Do opponents say we should judge the proposal according to its costs, its impact on health, both, or neither?” We then measured the percentage of respondents correctly identifying the dimension emphasized by proposal opponents (which is what was manipulated in the framing vignettes). This enables us to test whether the frames were “received” by the participants, thus allowing us to rule out inattention as an alternative explanation in the event that we find weak framing effects. Results We begin by examining whether the information manipulation influenced participants’ beliefs about electronic medical records. The results are reported in Table 1, with column 1 showing results for the Cost Belief question and column 2 showing results for the Impact Belief question. As should be immediately clear, those who received High Cost information saw electronic medical records as much more costly than those in any of the other conditions, but differed only marginally from those in the Low Cost information condition when it comes to perceptions of the policy’s impact on health care quality.12 By contrast, individuals in the High Impact condition saw the policy as offering substantially improved health care quality compared to those in the Low Impact condition. While 12 For clarity of presentation, we place all statistical significance tests in Tables A2-A8 in the Online Appendix. 15 these two groups differed somewhat in their perceptions of the cost of electronic medical records, both conditions were much closer to neutral on average than either of the costs conditions.13 Yet the effects of the information manipulations are clear: Participants’ beliefs about the issue were substantially influenced by exposure to just a few short paragraphs of information. This result supports the information hypothesis. Table 1. Beliefs about Cost and Impact by Information Condition (Study 1) Information Condition Cost Beliefs Impact Beliefs High Cost 0.81 (0.02) 0.22 (0.03) Low Cost -0.65 (0.03) 0.33 (0.03) High Impact -0.25 (0.04) 0.88 (0.02) Low Impact -0.03 (0.03) 0.17 (0.03) We now turn to analyzing the second phase of the experiment. In particular, we are interested in the mean level of support for electronic medical records in each of our ten experimental conditions. If information affects opinions, then these groups should differ based on what information they received in the information treatment phase. If framing matters, then groups that received the same information treatment should differ based only on what dimension is emphasized in the framing treatment phase. We present the results in Figure 2, as pairs of points representing treatment group means with error bars representing one- and two-standard errors of the mean. In each pair, the gray bar represents the Cost Frame condition and the black bar the Impact Frame condition. The results are easy to interpret: Framing an issue by emphasizing a particular issue dimension (be it cost or impact) has no effect on participants’ opinions. Only in the control group (where respondents received no information treatment) do we see anything close to a framing 13 We acknowledge that the impact information manipulations produced a somewhat smaller difference in impact beliefs than did the cost information for cost beliefs. One possibility for this is that it is hard to imagine serious policy debate by health care experts (i.e., the AMA) about a policy that would have negative impacts on health, thus the Low Impact information simply made the policy seem inconsequential as opposed to detrimental to health. 16 effect and it is arguably in the opposite direction of what might be expected given those in the Cost Frame condition are more supportive. How should we interpret these results in light of the information and framing hypotheses? Looking again at Figure 2, we see that information clearly mattered. Individuals in the High Impact and Low Cost conditions were more supportive on average than those in the Low Impact, High Cost, or control conditions. When individuals have received favorable information about a policy, they are more supportive and when they have received unfavorable information about that policy, they are less so. This lends clear support to the information hypothesis and suggests that policy information is vital to shaping preferences. Framing, by contrast, seems to matter very little. Simply because the debate shifts to emphasize one particular feature of the policy does not mean that the public is easily swayed to change their opinions. Information, not emphasis, changes opinions. One could argue, however, that these results do not challenge the framing hypothesis; rather they simply indicate that the framing manipulation did not “work” experimentally and thus there is a null finding. It may simply be that participants did not notice that a particular dimension of the policy was being emphasized. Fortunately, our framing manipulation check (measuring whether participants correctly perceived the frame used by the proposal’s opposition) allows us to know whether participants picked up on the frame being emphasized. In the Cost Frame condition, 63% (s.e. = 2.50) of respondents correctly perceived the debate as emphasizing costs and only 6% (1.19) perceived the debate as emphasizing impact. Similarly, in the Impact Frame condition, 34% (2.46) of respondents correctly perceived the debate as emphasizing impact and only 24% (2.21) of respondents perceived the debate to be emphasizing costs. While this manipulation check indicates that participants were better able to identify the Cost Frame than the Impact Frame, the differences in responses indicate that the two framing conditions clearly 17 emphasized different dimensions. The framing treatments therefore seem to have “worked” as intended and were received by the study participants. We can further probe the possibility of finding framing effects with our secondary dependent variable on belief importance. Consistent with the results on opinion, however, we find that framing a given dimension did not sizably change the importance participants attached to that dimension. In the Cost Frame condition, respondents rated the importance of costs at 0.72 (0.01) (on the 0 to 1 scale) and rated the importance of health impact at 0.75 (0.01). Similarly, in the Impact Frame condition, respondents rated the importance of health impact at 0.79 (0.01) and the importance of impact at 0.76 (0.01). In other words, pure framing in the form of telling citizens that a dimension is important apparently is not enough for them to substantially value that dimension more and, consequently, may explain why framing alone did little to change opinions toward the issue. This is important because it means that framing does not work empirically as it is thought to work in theory. The result of our first study is clear: Information, not emphasis, changes opinions. Study 2: A Stronger Framing Manipulation Given Study 1 indicated that opinions moved in response to information but not in response to framing, we modified our experimental protocol to create more favorable conditions for finding support for the framing hypothesis. Specifically, we changed the framing manipulations to be longer and more clear, while still not introducing any additional information. The text now read: Cost Frame: “Recently there has been some debate in Congress about a new proposal regarding electronic medical records. Some support the proposal. Others are opposed because they say that we should judge the proposal based on whether it is costly. Indeed, much of the debate over the proposal now revolves around the question of costs.” 18 Impact Frame: “Recently there has been some debate in Congress about a new proposal regarding electronic medical records. Some support the proposal. Others are opposed because they say that we should judge the proposal based on whether it will affect the health of average Americans. Indeed, much of the debate over the proposal now revolves around the question of the proposal’s impact on patients’ health.” Unlike in Study 1, where the frames were owned by particular sides in the debate (opponents owning one side and supporters the other), this manipulation additionally highlights that the emphasized frame is defining public debate. In other words, the emphasized frame has won.14 Furthermore, we changed the question wording for our main outcome measure. Specifically, those in the Cost Frame condition were asked: “Based on whether you think the proposal will mean higher or lower costs, to what extent do you favor or oppose this proposal?” while those in the Impact Frame condition were asked: “Based on whether you think the proposal will have a large or small impact on patients' health, to what extent do you favor or oppose this proposal?” Response categories for both questions were the same as in Study 1. This manipulation of question wording reinforces the framing manipulations by asking participants to explicitly weight particular beliefs while, again, not supplying any new information. Otherwise the protocol for Study 2 closely matched that for Study 1. Indeed, the remaining questionnaire was identical. We again recruited participants from Amazon Mechanical Turk, excluding those who had participated in Study 1. This time, however, we engaged in quota sampling based on self-reported party identification. We used a Qualification Test to identify workers who were self-described Democrats, Republicans, and Independents and then capped the 14 We also made small changes to the wording of our information treatments (see the Online Appendix). 19 number of participants from each of these groups.15 Given the liberal-Democratic leanings of MTurk participants, we hoped this quota sampling procedure would balance our sample ideologically (lest that affected the Study 1 results) and indeed it succeeded (see Table A1 in the Online Appendix). While the demographics of our Study 2 sample closely mirror those for Study 1 in terms of race, age, and education, the Study 2 sample of 743 participants is notably more conservative, Republican, and male, thus providing us with a more face-representative sample. Results As in Study 1, we begin by first examining whether our information manipulations at t1 changed participants’ beliefs about the cost and impact of the policy. We report these results in Table 2. Consistent with Study 1, we see that those in the High Cost condition saw electronic medical records as more costly than those in the Low Cost condition, but they differed only modestly in their perceptions of the health impact of electronic medical records. Similarly, those in the High Impact condition saw electronic medical records as more beneficial for patients’ health than those in the Low Impact condition, while these groups differed only modestly in their perceptions of the policy’s costs. In other words, the information manipulations influenced beliefs as expected. Table 2. Beliefs about Cost and Impact by Information Condition (Study 2) Information Condition Cost Beliefs Impact Beliefs High Cost 0.81 (0.02) 0.23 (0.03) Low Cost -0.70 (0.03) 0.41 (0.03) High Impact -0.23 (0.04) 0.89 (0.02) Low Impact 0.03 (0.02) 0.14 (0.02) We can then examine the mean level of policy support in each of our experimental conditions. As in Study 1, we report these results visually (see Figure 3). As in Study 1, we find 15 We accomplished this using separate HITs for each group, though workers were naïve to why they qualified for only one of the three HITs. This was achieved using MTurkR (Leeper, 2012). 20 that the information treatments have large and intuitive effects on opinions: Those exposed to favorable information (High Impact or Low Cost) are more supportive than those who are told the policy has a low impact on health or those in the control group. Participants in the High Cost condition are particularly unfavorable. Information matters, yet Figure 3 also makes clear that— despite our efforts to strengthen the framing manipulation in order to create favorable conditions for the framing hypothesis—we once again find no support for framing. Emphasis alone did not change opinions. As in Study 1, we can verify if the framing manipulation “worked.” It did. In the Cost Frame condition, 64% (2.50) of respondents correctly perceived the debate as emphasizing costs and only 7% (1.35) perceived the debate as emphasizing impact. Similarly, in the Impact Frame condition, 34% (2.46) of respondents correctly perceived the debate as emphasizing impact and 29% (2.35) of respondents perceived the debate to be emphasizing costs. While it may be that the Impact Frame did not adequately steer participants’ attention to impact, the Cost Frame clearly did so, and still, we did not find support for the framing hypothesis even in the Cost Frame conditions. Consistent with these results, we also find that the framing treatment did not substantially affect the importance participants attached to issue-relevant beliefs. In the Cost Frame condition, respondents rated the importance of costs at 0.71 (0.01) (on the 0 to 1 scale) and rated the importance of health impact at 0.74 (0.01). Similarly, in the Impact Frame condition, respondents rated the importance of health impact at 0.78 (0.01) and the importance of impact at 0.75 (0.01). In sum, Study 2 aimed to build on the results from Study 1 while intentionally modifying the design to favor the framing hypothesis. Thus, we modified the framing manipulations to make the emphasized dimensions clearer and to more explicitly emphasize that debate about the policy was universally seen as either about costs or impact on health. Despite these modifications, the 21 results of Study 2 closely mirror those of Study 1. Information (the change in belief content), not framing (the changing importance of beliefs), led to opinion changes. Study 3: Inducing Multiple, Competing Beliefs Given that Study 2, with its strengthened framing manipulation, again failed to find support for the framing hypothesis, we decided to pursue a third and final study. It might have been the case that Studies 1 and 2 created unrealistic conditions for finding a framing effect. Specifically, all participants were exposed to only one-sided information at t1. Thus, when participants who at t1 were presented with High Cost information about electronic medical records, subsequently (at t2) received the Impact Frame, it is possible that they had no relevant beliefs about impacts on patients’ health that could be weighted by the frame. In other words, without having a mix of beliefs available in memory about the dimension being emphasized, it may be that we unintentionally neutered our framing manipulation. Indeed, the perception in the framing literature is that frames are particularly influential when they play at different, opposing considerations (Nelson and Kinder 1996: 1058; Sniderman and Theriault 2004: 138). In response, for Study 3 we modified our information treatments. Specifically, rather than exposing each participant to information about only one issue dimension, all respondents were exposed to both dimensions such that every participant received information about the cost of electronic medical records and information about the impact on patients’ health. The design therefore consisted of five conditions at t1: (1) Low Cost/Low Impact, (2) High Cost/Low Impact, (3) Low Cost/High Impact, and (4) High Cost/High Impact, and (5) the usual control condition from Studies 1 and 2. Two of these conditions (1 and 4) should induce high degrees of belief ambivalence and thus create ripe conditions for framing effects because on one dimension the policy scores well while on the other it scores poorly. Weighting one dimension versus the other should therefore produce a change in opinion. Similarly, in condition (2) the 22 policy should be disliked regardless of emphasis because it scores poorly on both dimensions and in condition (3) the policy should be liked regardless of framing because it scores well on both dimensions. Otherwise, the protocol for Study 3 closely mirrored that of Studies 1 and 2. Aside from changes to the t1 information manipulation, the remainder of the questionnaire (including the framing manipulation) was unchanged. We again recruited participants from Amazon Mechanical Turk, excluding those who had participated in either Study 1 or Study 2, and we again relied on the quota sampling procedure used in Study 2 to create an ideologically balanced sample of 806 participants (for demographics, see the Online Appendix). Results Given that Study 3 used substantially different information manipulations than Studies 1 and 2, it is worth reading these results carefully. Our concern in the previous experiments was that we only supplied participants with information about a single dimension of consideration during the information phase, thus preventing them from being framed by the subsequent mere emphasis of a particular dimension of the issue. It is therefore important that the information treatments induced changes in beliefs about both cost and impact. As is clear from Table 3, our information treatments indeed worked as expected. In the High Cost/High Impact condition, participants report that electronic medical records are both costly and impactful while those in the Low Cost/Low Impact condition see the proposal as inexpensive but also as having little impact on health. Therefore, depending on the dimension by which one considers the issue, participants in every condition should hold both favorable and unfavorable beliefs, creating a situation ripe for finding support for the framing hypothesis. For these two conditions, depending on which dimension is framed (i.e., which evaluative belief is weighted), policy opinions should differ. 23 Table 3. Beliefs about Cost and Impact by Information Condition (Study 3) Information Condition Cost Beliefs Impact Beliefs High Cost/High Impact 0.76 (0.03) 0.89 (0.02) Low Cost/High Impact -0.71 (0.02) 0.77 (0.02) High Cost/Low Impact 0.65 (0.03) 0.13 (0.02) Low Cost/Low Impact -0.47 (0.03) 0.10 (0.02) In contrast, in the two remaining information conditions, policy beliefs are consistently positive or negative. Thus, as expected, participants in the Low Cost/High Impact condition believe the proposal is both very inexpensive and highly impactful whereas those in the High Cost/Low Impact condition see the proposal as costly and ineffectual. In these information conditions, we should expect little impact of framing because their evaluative beliefs are positive or negative, respectively, regardless of which dimension is emphasized. With the information hypothesis again supported, we can now turn to our results on policy support which we report visually in Figure 4. First we note that the control group reports opinions consistent with the control groups in our previous studies. Without information about the policy, this group is neutral toward the policy regardless of frame. We are particularly interested in whether framing mattered in the conditions where the information treatments induced conflicting beliefs among participants to make them ripe for framing effects. Consider first the High Cost/High Impact information condition. If framing matters, we would expect those in the Impact Frame condition (black bar) to be more supportive than those in the Cost Frame condition (gray bar), yet these groups are indistinguishable from one another. Similarly, in the Low Cost/Low Impact information condition, we would again expect participants’ opinions to vary depending on what dimension is framed. Framing participants in this information condition to think about costs should make them more favorable (given they see the policy as low cost) while inducing them to think about impact should make them less 24 favorable (given they see little impact on patients’ health). We find, however, that framing does not influence opinions as policy support is not significantly different across the two framing conditions. Moving to the last two information conditions—High Cost/Low Impact and Low Cost/High Impact—we would expect minimal framing effects because each evaluative belief points in the same direction within the conditions. As expected, we find participants in the High Cost/Low Impact information conditions to be the least supportive of the policy, regardless of framing. Looking at the Low Cost/High Impact information conditions, we see that they as expected are among the most positive toward the policy. Here we do find a difference in opinion across framing conditions where policy support is higher in the Cost Frame (0.58) than in the Impact Frame (0.46) condition. However this is only marginally significant (p<0.07) and, ironically, it occurs at an instance where it should not be expected according to the framing hypothesis. As we found in Studies 1 and 2, this lack of evidence for the framing hypothesis is unlikely to be explained by a failure to grasp the changing emphasis in the debate. Thus, our manipulation check showed that our frames were successfully received by participants. In the Cost Frame condition, 61% (2.43) of respondents correctly perceived the debate as emphasizing costs and only 9% (1.44) perceived the debate as emphasizing impact. Similarly, in the Impact Frame condition, 35% (2.39) of respondents correctly perceived the debate as emphasizing impact whereas 28% (2.24) of respondents perceived the debate to be emphasizing costs. As in our previous studies, we recognize the Cost Frame appears to be more easily recognized than the Impact Frame, but we yet again found no support for the framing hypothesis. Consistent with the previous studies, we find substantially tiny framing effects on belief importance. The alternative considerations were perceived as essentially equally important regardless of frame. Specifically, in the Cost Frame condition, respondents rated the importance 25 of costs at 0.71 (0.01) and the importance of health impact at 0.77 (0.01). Similarly, in the Impact Frame condition, respondents rated the importance of health impact at 0.74 (0.01) and the importance of impact at 0.75 (0.01). What does this mean? Our concern in Studies 1 and 2 was that we had unfairly disadvantaged the framing hypothesis by manipulating participants’ beliefs about only one dimension of the policy. Our response was to cross the two information dimensions (cost and impact) in an effort to more fairly test the framing hypothesis. When we disentangle the impact of information from the impact of framing in this way, the results suggest it is information, not emphasis, that explains the opinion changes consistently observed in previous studies of “framing.” Discussion A framing effect—the result of placing emphasis on a particular dimension of consideration—is widely thought to be among politicians’ most important tools for influencing public opinions. Yet we have argued that most extant framing research does not disentangle the influence of emphasizing a particular dimension of an issue (the framing hypothesis) from the provision of information and arguments (the information hypothesis). We approached this problem with a novel experimental design that physically and temporally separated the provision of information and the communication of frames. Across three studies, each carefully crafted to provide an easier and easier opportunity for framing effects to emerge, we find no evidence for framing effects. Information, not emphasis, shifts citizens’ opinions. These results invite us to reconsider the meaning of the sizeable extant literature on framing effects. We do not mean to question the validity or importance of the findings in this literature. However, our results suggest that extant studies demonstrate the influence of information and arguments, rather than emphasis alone. Opinions in previous experiments were likely moved by 26 the information and arguments contained in the treatments not the way the information was framed. This means that framing can rightfully be thought of as a phenomenon separate from argument-based persuasion or information effects, but we see little evidence here for the impact of framing (true to its name) on public opinions. This finding has important political, theoretical, and normative implications. Politically, the limited empirical support for pure framing implies that politicians, political parties, and other actors must do far more than frame (or reframe) issues to change public attitudes. They must provide arguments and evidence in support of their positions. As noted by Sniderman and Carmines (1997: 129), “the politics of public policy is [not] merely a matter of marketing. It is a matter of argument. The public’s reaction to public policy hinges in part on what is proposed by way of public opinion, and on what arguments are made both for and against what is being proposed.” This more demanding political strategy should lead political scientists to rethink the scope and size of elite influence, and might hold promising normative implications for citizens forming opinions grounded in substantive arguments to a larger extant than previously thought (see Druckman 2014). Theoretically, our results challenge the meaning and relevance of the framing concept and the viability of the expectancy-value model of attitudes that underlies it. The literature on media effects already discusses how to distinguish between framing and other types of effects and psychological processes (e.g., de Vreese and Lecheler 2012; Kinder 2003; Scheufele and Iyengar 2015). Notably, Chong and Druckman (2012: 310-311) suggest to “view framing, priming and agenda setting as equivalent processes that involve alterations of the weight component of an attitude … until definitive evidence reveals mediational or moderating differences, the concepts should be treated as the same, meaning research on each should be merged and redundancy avoided,” and they add: “Along similar lines, the extent to which persuasion and the other 27 concepts differ remains unclear” (p. 321). If framing, priming and agenda setting are related, or perhaps equivalent, concepts, our findings should also raise concern for the distinctiveness of these other concepts. Lenz (2009) already showed that the processes underlying “priming effects” are more often learning followed by rationalization, and Togeby (2007) showed that in order for priming to occur, an argument was needed to make clear why a given issue would be relevant for evaluating government performance (also see Druckman, Kuklinski, and Sigelman 2009; Miller and Krosnick 2000; Mutz 1998: 72-73).16 That evidence, in tandem with our findings, suggests that “direct” communication effects might play a much larger role than the “indirect” influences of agenda setting, priming, and framing has suggested (Kinder 2003). We need future studies that probe the potential and meaning of framing further. It is clear that citizens are able to view an issue from different perspectives, emphasizing different set of considerations (Chong 1993; Feldman and Zaller 1992). Yet it has been difficult to illuminate empirically what it takes for citizens to come to rely on different dimensions and, furthermore, what those different frames in thought actually mean for opinion dynamics. A growing body of evidence suggests that citizens can approach a judgment or decision-making task from different perspectives: elite arguments shape how citizens choose information (Druckman, Fein, and Leeper 2012) and how they evaluate political debates (Druckman et al. 2010). Perhaps frames provide a lens for understanding new information, even if they do little to reorganize beliefs already in memory. Finally, democracy as a form of government is premised upon the representation of public preferences by government. The consistent finding of a reverse pattern of influence whereby 16 Mutz (1998: 72-73) notes that evidence of agenda setting and priming is “sparse”: “evidence to date suggests that what is happening in many cases may be a directional effect instead of, or perhaps in addition to, an effect on the accessibility of certain information … much of the evidence brought to bear on these hypotheses suggests that media are capable of altering more than simply the salience attached to a given issue or the standards people use to evaluate political leaders.” 28 elites shape public preferences problematizes this democratic foundation. Our findings suggest that elites cannot easily sway public preferences without communicating policy-relevant information and develop persuasive arguments to support their preferred position. We find that emphasis alone is an insufficient political strategy. Citizens are not so easily swayed. 29 Figure 1: Experimental Design for Separating Information and Framing Effects 30 Figure 2. Mean Opinion by Information and Framing Conditions (Study 1) Note: Points are mean levels of policy support, by information and framing conditions. Gray points represent Cost Frame conditions and black points represent Impact Frame conditions. Bars represent one and two standard errors of the treatment group mean. 31 Figure 3. Mean Opinion by Information and Framing Conditions (Study 2) Note: Points are mean levels of policy support, by information and framing conditions. Gray points represent Cost Frame conditions and black points represent Impact Frame conditions. Bars represent one and two standard errors of the treatment group mean. 32 Figure 4. Mean Opinion by Information and Framing Conditions (Study 3) Note: Points are mean levels of policy support, by information and framing conditions. Gray points represent Cost Frame conditions and black points represent Impact Frame conditions. Bars represent one and two standard errors of the treatment group mean. 33 References Aarøe, Lene. 2011. “Investigating Frame Strength: The Case of Episodic and Thematic Frames.” Political Communication 28(2): 207-226. Berinsky, Adam J., Gregory A. Huber, and Gabriel S. Lenz. 2012. “Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk.” Political Analysis 20(3): 351-368. Berinsky, Adam J., and Donald R. Kinder. 2006. “Making Sense of Issues Through Media Frames: Understanding the Kosovo Crisis.” Journal of Politics 68(3): 640–656. Blumenthal, David, and Tavenner, Marilyn. 2010. "The 'Meaningful Use' Regulation for Electronic Health Records.” New England Journal of Medicine 363: 501-504. Boudreau, Cheryl. 2015. “Party versus Principle: How Competing Parties and Frames Affect the Consistency between Citizens’ Values and Policy Views.” Typiscript. University of California, Davis. Brewer, Paul R. 2001. “Value Words and Lizard Brains: Do Citizens Deliberate About Appeals to Their Core Values?” Political Psychology 22(1): 45-64. Chong, Dennis. 1993. “How People Think, Reason, and Feel about Rights and Liberties.” American Journal of Political Science 37(3): 867. Chong, Dennis. 2000. Rational Lives: Norms and Values in Politics and Society. Chicago: University of Chicago Press. Chong, Dennis, and James N. Druckman. 2007a. “Framing Theory.” Annual Review of Political Science 10(1): 103-126. Chong, Dennis, and James N. Druckman. 2007b. “Framing Public Opinion in Competitive Democracies.” American Political Science Review 101(4): 637-655. Chong, Dennis, and James N. Druckman. 2012. “Dynamics in Mass Communication Effects Research.” In The Sage Handbook of Political Communication, eds. Holli Semetko, and Maggie Scammell. Thousand Oaks, CA: Sage Publications. Dahl, Robert A. 1971. Polyarchy. Chicago: University of Chicago Press. Dahl, Robert A. 1989. Democracy and Its Critics. New Haven, CT: Yale University Press. de Vreese, Claes H. 2003. Framing Europe: Television News and European Integration. Amsterdam: Aksant. de Vreese, Claes H., and Sophie K. Lecheler. 2012. “News Framing Research: An Overview and New Developments.” In The SAGE Handbook of Political Communication, eds. Holli Semetko, and Margaret Scammell. SAGE Publications. 34 Disch, Lisa. 2011. “Toward a Mobilization Conception of Democratic Representation.” American Political Science Review 105(1): 100-114. Druckman, James N. 2001. “The Implications of Framing Effects for Citizen Competence.” Political Behavior 23(3): 225-256. Druckman, James N. 2014. “Pathologies of Studying Public Opinion, Political Communication, and Democratic Responsiveness.” Political Communication 31(3): 467-492. Druckman, James N., and Toby Bolsen. 2011. “Framing, Motivated Reasoning, and Opinions About Emergent Technologies.” Journal of Communication 61(4): 659–688. Druckman, James N., Cari Lynn Hennessya, Kristi St. Charlesa, and Jonathan Webber. 2010. “Competing Rhetoric Over Time: Frames Versus Cues.” The Journal of Politics 72(1): 136-148. Druckman, James N., James H. Kuklinski, and Lee Sigelman. 2009. “The Unmet Potential of Interdisciplinary Research: Political Psychological Approaches to Voting and Public Opinion.” Political Behavior 31(4): 485-510. Druckman, James N., and Thomas J. Leeper. 2012. “Learning More from Political Communication Experiments: Pretreatment and Its Effects.” American Journal of Political Science 56(4): 875-896. Druckman, James N., and Kjersten R. Nelson. 2003. “Framing and Deliberation: How Citizens' Conversations Limit Elite Influence.” American Journal of Political Science 47(4): 729745. Druckman, James N., Erik Peterson, and Rune Slothuus. 2013. “How Elite Partisan Polarization Affects Public Opinion Formation.” American Political Science Review 107 (1): 57-79. Feldman, Stanley, and John Zaller. 1992. “The Political Culture of Ambivalence: Ideological Responses to the Welfare State.” American Journal of Political Science 36(1): 268-307. Ford, Eric W., Menachemi, Nir, and Phillips, M. Thad. 2006. “Predicting the Adoption of Electronic Health Records by Physicians: When Will Health Care Be Paperless?” Journal of the American Medical Informatics Association 13: 106-112. Fowler, Erika F., and Sarah E. Gollust. 2015. “The Content and Effect of Politicized Health Controversies.” The ANNALS of the American Academy of Political and Social Science 658(1): 155-171. Gamson, William A., and Andre Modigliani. 1987. “The Changing Culture of Affirmative Action.” Research in Political Sociology 3: 137-177. Hartman, Todd K., and Christopher R. Weber. 2009. “Who Said What? The Effects of Source Cues in Issue Frames.” Political Behavior 31(4): 537-558. 35 Hopkins, Daniel J. 2014. “The Exaggerated Life of Death Panels: The Limits of Framing Effects in the 2009‐2012 Health Care Debate.” Typescript. Georgetown University. Huber, Gregory A., and Celia Paris. 2013. “Assessing the Programmatic Equivalence Assumption in Question Wording Experiments: Understanding Why Americans Like Assistance to the Poor More Than Welfare.” Public Opinion Quarterly 77: 385-397. Jacoby, William G. 2000. “Opinion and Public Issue Framing Spending on Government.” American Journal of Political Science 44(4): 750-767. Jerit, Jennifer. 2009. “How Predictive Appeals Affect Policy Opinions.” American Journal of Political Science 53(2): 411-426. Kinder, Donald R. 2003. “Communication and Politics in the Age of Information.” In Oxford Handboook of Political Psychology, eds. David O. Sears, Leonie Huddy, and Robert Jervis. New York: Oxford University Press. Lecheler, Sophie K., and Claes H. de Vreese. 2013. “What a Difference a Day Makes? The Effects of Repetitive and Competitive News Framing Over Time.” Communication Research 40(2): 147-175. Leeper, Thomas J. 2012. “MTurkR: R Client for the MTurk Requester API.” Version 0.6. Leeper, Thomas J. 2014. “The Informational Basis for Mass Polarization.” Public Opinion Quarterly 78(1): 27-46. Leeper, Thomas J., and Kevin J. Mullinix. N.d. “What If You Had Done Things Differently? Testing the Generalizability of Framing Effects with Parallel Experiments.” Typescript. Aarhus University. Lenz, Gabriel S. 2009. “Learning and Opinion Change, Not Priming: Reconsidering the Priming Hypothesis.” American Journal of Political Science 53(4): 821-837. Miller, Joanne M., and Jon A. Krosnick. 2000. “News Media Impact on the Ingredients of Presidential Evaluations: Politically Knowledgeable Citizens are Guided by a Trusted Source.” American Journal of Political Science 44(2): 301-315. Miller, Robert H., and Ida Sim. 2004. “Pursuit Of Quality: Physicians' Use Of Electronic Medical Records: Barriers And Solutions.” Health Affairs 23: 2116-2126. Mullinix, Kevin J., James N. Druckman, and Jeremy Freese. N.d. “When Convenience Samples Yield Generalizable Estimates in Social Science Experiments.” Typescript. Northwestern University. Mutz, Diana C. 1998. Impersonal Influence. New York: Cambridge University Press. Nelson, Thomas E., and Donald R. Kinder. 1996. “Issue Frames and Group-Centrism in American Public Opinion.” The Journal of Politics 58(4): 1055. 36 Nelson, Thomas E., Rosalee A. Clawson, and Zoe M. Oxley. 1997. “Media Framing of a Civil Liberties Conflict and Its Effect on Tolerance.” American Political Science Review 91(3): 567-583. Nelson, Thomas E., Zoe M. Oxley, and Rosalee A. Clawson. 1997. “Toward a Psychology of Framing Effects.” Political Behavior 19(3): 221-246. Price, Vincent, and David Tewksbury. 1997. “News Values and Public Opinion: A Theoretical Account of Media Priming and Framing.” In Progress in Communication Sciences. Norwood, N.J.: Ablex Publishing Corporation, 173-212. Schemer, Christian, Werner Wirth, and J• org Matthes. 2012. “Value Resonance and Value Framing Effects on Voting Intentions in Direct-Democratic Campaigns.” American Behavioral Scientist 56(3): 334-352. Scheufele, Dietram A., and Shanto Iyengar. 2015. “The State of Framing Research: A Call for New Directions.” In The Oxford Handbook of Political Communication, eds. Kate Kenski and Kathleen Hall Jamieson. New York: Oxford University Press. Forthcoming. Schuck, Andreas R.T., Rens Vliegenthart, and Claes H. de Vreese. 2015. “Who's Afraid of Conflict? The Mobilizing Effect of Conflict Framing in Campaign News.” British Journal of Political Science. Forthcoming. Schuman, Howard, and Stan Presser. 1981. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. Thousand Oaks, CA: SAGE Publications. Slothuus, Rune. 2008. “More Than Weighting Cognitive Importance: A Dual-Process Model of Issue Framing Effects.” Political Psychology 29(1): 1-28. Slothuus, Rune. 2010. “When Can Political Parties Lead Public Opinion? Evidence from a Natural Experiment.” Political Communication 27(2): 158-177. Slothuus, Rune, and Claes H. de Vreese. 2010. “Political Parties, Motivated Reasoning, and Issue Framing Effects.” The Journal of Politics 72(3): 630-645. Sniderman, Paul M., and Sean M. Theriault. 2004. “The Structure of Political Argument and the Logic of Issue Framing.” In Studies in Public Opinion: Attitudes, Nonattitudes, Measurement Error, and Change, eds. William E. Saris, and Paul M. Sniderman. Princeton, NJ: Princeton University Press, 133-165. Togeby, Lise. 2007. “The Context of Priming.” Scandinavian Political Studies 30(3): 345-376. Weaver, David H. 2007. “Thoughts on Agenda Setting, Framing, and Priming.” Journal of Communication 57(1): 142-147. Zaller, John. 1992. The Nature and Origins of Mass Opinion. New York: Cambridge University Press. 37 SUPPORTING INFORMATION ONLINE APPENDIX Appendix 1: Supplemental Statistical Results Table A1. Study Demographics % Female % White Mean Age % With University Degree % Democrat % Republican Mean Ideology Sample size Dates of Data Collection Study 1 0.56 (0.25) 0.77 (0.18) 36.72 (12.84) 0.50 (0.25) 0.51 (0.25) 0.26 (0.19) 0.16 (0.57) Study 3 0.44 (0.25) 0.80 (0.16) 33.27 (10.78) 0.48 (0.25) 0.41 (0.24) 0.34 (0.23) 0.10 (0.54) 750 743 806 Jun. 19-21, 2014 Oct. 16-Nov. 5, 2014 Jan 5-28, 2015 38 Study 2 0.40 (0.24) 0.79 (0.16) 31.60 (9.71) 0.52 (0.25) 0.40 (0.24) 0.34 (0.22) 0.11 (0.54) Table A2. Mean Opinion and Framing Effect by Information Condition (Study 1) Information Control High Cost Low Cost High Impact Low Impact F-test Impact Frame Cost Frame t-statistic (p-value) 0.13 0.31 -2.53 (p<0.01) 0.12 0.14 -0.19 (p<0.85) 0.39 0.37 0.29 (p<0.77) 0.48 0.51 -0.34 (p<0.74) 0.27 0.23 0.42 (p<0.67) F(4,368) = 8.73 (p<0.00) F(4,371) = 7.08 (p<0.00) Table A3. Mean Opinion and Framing Effect by Information Condition (Study 2) Information Impact Frame Cost Frame t-statistic (p-value) Control 0.14 0.22 -1.04 (p<0.30) High Cost 0.02 0.07 -0.64 (p<0.52) Low Cost 0.47 0.51 -0.58 (p<0.57) High Impact 0.50 0.47 0.53 (p<0.59) Low Impact 0.36 0.32 0.71 (p<0.48) F-test F(4,366) = 15.74 (p<0.00) F(4,364) = 12.23 (p<0.00) Table A4. Mean Opinion and Framing Effect by Information Condition (Study 3) Information Control High Cost/High Impact Low Cost/High Impact High Cost/Low Impact Low Cost/Low Impact F-test Impact Frame Cost Frame t-statistic (p-value) 0.14 0.19 -0.69 (p<0.49) 0.33 0.30 0.41 (p<0.68) 0.46 0.58 -1.81 (p<0.07) 0.03 0.06 -0.44 (p<0.66) 0.37 0.45 -1.34 (p<0.18) F(4,397) = 12.14 F(4,398) = (p<0.00) 16.76 (p<0.00) Table A5. Belief Importance by Framing Condition (All Studies) Health Beliefs Study 1 Study 2 Study 3 Cost Beliefs Study 1 Study 2 Study 3 Impact Frame Cost Frame t-statistic (p-value) 0.79 0.78 0.74 0.75 0.74 0.77 2.28 (p<0.02) 2.22 (p<0.03) -1.49 (p<0.14) 0.76 0.75 0.75 0.72 0.71 0.71 2.32 (p<0.02) 2.24 (p<0.03) 2.36 (p<0.02) 39 Table A6. Proportion Perceiving Debate Frame by Framing Condition (Study 1) Perceive Cost Perceive Impact Emphasized Emphasized Difference in Proportions Impact Frame 0.24 0.34 χ2 = 8.86 (p<0.00) Cost Frame 0.63 0.06 χ2 = 270.71 (p<0.00) Difference in χ2 = 112.72 (p<0.00) χ2 = 94.81 (p<0.00) Proportions Table A7. Proportion Perceiving Debate Frame by Framing Condition (Study 2) Perceive Cost Perceive Impact Emphasized Emphasized Difference in Proportions Impact Frame 0.29 0.34 χ2 = 2.25 (p<0.13) Cost Frame 0.64 0.07 χ2 = 256.84 (p<0.00) Difference in χ2 = 90.73 (p<0.00) χ2 = 79.95 (p<0.00) Proportions Table A8. Proportion Perceiving Debate Frame by Framing Condition (Study 3) Perceive Cost Perceive Impact Emphasized Emphasized Difference in Proportions Impact Frame 0.28 0.35 χ2 = 5.15 (p<0.02) Cost Frame 0.61 0.09 χ2 = 235.98 (p<0.00) Difference in χ2 = 87.56 (p<0.00) χ2 = 80.37 (p<0.00) Proportions 40 Appendix 2: Study 1 Treatments and Questionnaire Introductory Text Today we ask you to participate in three short studies. First you will read some newspaper excerpts and answer some questions about what you read. Then, you will complete another study and we will ask you some background questions about yourself. Finally, we will ask you for your opinions about some political issues. Information Manipulation <CONTROL GROUP> The first study you will participate in is about the quality of journalism. We are interested in how well journalists write about the news. You will read a few short excerpts from a news article and then we will ask you some questions to see how well you understood what was written. We are asking different people to read articles on different topics. You are being asked to read excerpts from an article about music. <Excerpt1> Here is the first part of the article. Please read it carefully. LOS ANGELES -- Most albums aren't recorded overnight; just ask Steely Dan or Bruce Springsteen. However, No Doubt's Push and Shove, is one of the longest-gestating pop albums in recent memory. According to bassist Tony Kanal, it's a triumph that the album – the band's first since 2001's Rock Steady – was completed at all. </Excerpt1> <Excerpt2> Here is the next part of the article. Please read it carefully. Gwen Stefani hates it when people point out that No Doubt haven't released an album in 10 years. "A lot of stuff happened during that time period," she says. "Marriages, babies and, for me, two records and two clothing lines. So if you really worked out the math, you'd be like, 'Wow, you guys are going fast.'" </Excerpt2> <Excerpt3> Here is the next part of the article. Please read it carefully. The winding road to Push and Shove began in 2007 while lead singer Gwen Stefani was touring behind her second solo album, The Sweet Escape. In June, during the encore for one of her shows in Irvine, California, the other members of No Doubt joined her onstage for a four-song set that included their hits "Just a Girl" and "Hella Good." "The response was so amazing that we said, 'We have to start working,'" explains Kanal. </Excerpt3> <Excerpt4> Here is the last part of the article. Please read it carefully. Not long after the show, the band convened to start writing new material, but Stefani was still 41 exhausted from touring and was expecting her second child. The band was also hit with collective writers' block. Those songwriting sessions and further ones in 2008 yielded little, and the band decided to give it a rest. "We know when it's right and when it isn't," Kanal says. "We had to get that feeling again." </Excerpt4> <PAGEBREAK/> What is the name of the band discussed in the article? Bruce Springsteen and the E Street Band Bad Religion No Doubt Red Hot Chili Peppers Don’t Know <PAGEBREAK/> When did the band first start work on their current album? 1996 2001 2007 2014 Don’t know </CONTROL GROUP> <TREATMENT CONDITIONS> The first study you will participate in is about the quality of journalism. We are interested in how well journalists write about the news. You will read a few short excerpts from a news article and then we will ask you some questions to see how well you understood what was written. We are asking different people to read articles on different topics. You are being asked to read excerpts from an article about electronic medical records. <PAGEBREAK/> <Exercept1> Here is the first part of the article. Please read it carefully. DENVER, Colo. – Leaders of the American Medical Association have been busy at work this week in an annual conference held here in Denver. On Tuesday, the AMA’s policy advisory committee began debate on a proposal that would change rules regarding how electronic medical records are handled by the nation’s hospitals and clinics. The proposal would standardize medical record technology. If implemented, the proposal would mean <CostHigh> significantly higher costs for doctors and clinics. A typical clinic could see expenses rise by as much as $5000 per year under the proposed changes. Nationwide, administrative costs related to medical records could rise by as much as $17 million over the next two years. 42 </CostHigh> <CostLow> significant cost savings for doctors and clinics. A typical clinic could see expenses reduced by as much as $5000 per year under the proposed changes. Nationwide, administrative costs related to medical records could be lowered by as much as $17 million over the next two years. </CostLow> <BenefitsHigh> that doctors, clinics, and hospitals would be able to share records more easily and be able to better serve patient needs and more quickly retrieve vital health information. This would improve the quality of health care for most Americans. </BenefitsHigh> <BenefitsLow> that doctors, clinics, and hospitals would use standardized software so electronic records are interoperable – meaning computer systems would rely on compatible formats. This would have little impact on the quality of health care for most Americans. </BenefitsLow> <PAGEBREAK/> <Comprehension> What organization was mentioned in the text? American Hospital Association American Medical Association Association of Nurse Practitioners Doctors Without Borders Don’t know </Comprehension> </Excerpt1> <Excerpt2> Here is the next part of the article . Please read it carefully. Electronic medical records have been the subject of debate within the medical field for more than ten years. These systems would replace paper records kept by individual doctors and clinics with sophisticated in-house computer systems. Numerous companies currently produce competing software. <CostHigh> The push for electronic medical records is seen by many experts as an expensive alternative to paper records. Startup costs for a new electronic medical records system can cost a five-physician clinic $32,000 plus $8000-$9000 in annual maintenance costs. </CostHigh> <CostLow> The push for electronic medical records is seen by many experts as an cost-saving alternative to paper records. A new electronic medical records system can save a five-physician clinic $8000$9000 per year in operating costs. </CostLow> <BenefitsHigh> 43 The push for electronic medical records is seen by many experts as a clear way to improve the quality of Americans’ health care. With these systems, doctors can more easily track patient health over time, and more easily catch risk factors for chronic and life-threatening diseases. </BenefitsHigh> <BenefitsLow> The push for electronic medical records is seen by many experts as a clear way to keep pace with changing technology. With these systems, medical coding is streamlined, simplifying recordkeeping processes for clinic staff, even if it will have no impact on patients’ health. </BenefitsLow> <PAGEBREAK/> <Comprehension> According to the article, how long has the issue of electronic medical records been debated? Less than 1 year 1-5 years 5-10 years More than 10 years Don’t know </Comprehension> </Excerpt2> <Excerpt3> Here is the next part of the article. Please read it carefully. The Brookings Institution, an independent policy think tank based in Washington, DC, released a report this morning describing an in-depth analysis of electronic medical records in the United States. At a press conference, Brookings spokesman Will Richardson said that the report details how proposed rule changes currently being debated by the nation’s leading hospitals, clinics, and doctors would <CostHigh> lead to significantly higher costs for doctors and patients. Clinics have been slow to adopt electronic medical records precisely because the costs are so high. </CostHigh> <CostLow> lead to significantly lower costs for doctors and patients. Clinics have been slow to adopt electronic medical records precisely because they have been waiting for these kinds of savings to become available. </CostLow> <BenefitsHigh> substantially improve the flexibility of the U.S. health care system, allowing patients and doctors to quickly share vital health information. The changes would also mean simplified handling of insurance claims, reducing hassle for American families. </BenefitsHigh> <BenefitsLow> 44 likely lead to modifications of existing clinical software and the future development of uniform formatting requirements for electronic records. The changes would also mean modified rules for software developers to comply with the new standards. </BenefitsLow> <PAGEBREAK/> <Comprehension> In what city is the Brookings Institution located? New York City, New York Washington, DC Seattle, Washington Denver, Colorado Don’t know </Comprehension> </Excerpt3> <Excerpt4> Here is the last part of the article. Please read it carefully. Back in Denver, the AMA conference continues through the end of the week where electronic medical records seem to be the big topic of debate among conference attendees. Both practitioners and technology developers are eager to see the policy advisory committee’s recommendations, which are due by the end of the week. <CostHigh> One doctor, Jan Miller of Oakland, CA, remained concerned about her clinic’s bottom line. “Electronic medical records are a substantial change in clinical practice and my practice cannot afford such a system without passing costs on to patients.” </CostHigh> <CostLow> One doctor, Jan Miller of Oakland, CA, was confident that electronic medical records would reduce costs her in practice. “Electronic medical records are a substantial change in clinical practice and my patients are asking for this system because they know it will save them money.” </CostLow> <BenefitsHigh> One doctor, Jan Miller of Oakland, CA, was confident that electronic medical records would improve her patients’ health. “Electronic medical records are a substantial change in clinical practice and patients will see the benefits of these systems right away. Electronic records help me make fewer errors and know my patients better.” </BenefitsHigh> <BenefitsLow> One doctor, Jan Miller of Oakland, CA, downplayed the importance of this week’s debates. “Electronic medical records mean little for my practice. It’s really same old, same old. I don’t see how patients would be affected really.” </BenefitsLow> <PAGEBREAK/> <Comprehension> 45 What was the profession of Jan Miller, the individual quoted in the article? Software developer Nurse Doctor Insurance agent Don’t know </Comprehension> </Excerpt4> Information Manipulation Check The article you read discussed a proposal to standardize electronic medical records. From what you read, will health care costs be more expensive or less expensive under the proposed changes? Much more expensive Somewhat more expensive A little more expensive No change A little less expensive Somewhat less expensive Much less expensive From what you read, will health care quality be improved or harmed under the proposed changes? Improved a lot Improved somewhat Improved a little No change Harmed a little Harmed somewhat Harmed a lot How easy or difficult was this article for you to understand? Very easy Moderately easy Somewhat easy Neither easy nor difficult Somewhat difficult Moderately difficult Very difficult How engaging did you find this article to be? Extremely engaging Moderately engaging Somewhat engaging Not very engaging 46 Not at all engaging How similar is this article to news articles you typically see in newspapers or on the internet? Extremely similar Moderately similar Somewhat similar Not very similar Not at all similar Distractor Activity Demographics Are you male or female? Male Female Which of the following do you consider to be your primary racial or ethnic group? White or Caucasian Black or African American Asian American Hispanic Native American Other What is your age in years? [Free Response] What is the last grade or class that you completed in school? None, or grade 1-8 Some high school High school (Grade 12 or GED) Technical, trade, or vocational school after high school Some college College graduate Post-graduate training or professional schooling after college Generally speaking, do you consider yourself a Democrat, Independent, or Republican? Strong Democrat Weak Democrat Independent leans Democrat Independent Independent leans Republican Weak Republican 47 Strong Republican Framing Manipulation The third study asks you for your opinion about a recent political issue. Recently there has been some debate in Congress about a new proposal regarding electronic medical records. [RANDOMIZE INTO ONE OF TWO FRAME CONDITIONS] <CostFrame> Some support the proposal. Others are opposed because they say that we should judge the proposal based on whether it is costly. </CostFrame> <BenefitsFrame> Some support the proposal. Others are opposed because they say that we should judge the proposal based on whether it will affect the health of average Americans. </BenefitsFrame> Outcome Measures Given this information, to what extent do you favor or oppose this proposal? Strongly favor Moderately favor Somewhat favor Neither favor nor oppose Somewhat oppose Moderately oppose Strongly oppose How certain are you of your opinion on the proposal regarding electronic medical records? Not at all certain Somewhat certain Moderately certain Very certain Extremely certain How important to you personally is your opinion about the proposal regarding electronic medical records? Not at all important Somewhat important Moderately important Very important Extremely important 48 Framing Manipulation Checks We are now going to list a few ideas that individuals have expressed when describing their opinions about the proposal regarding electronic medical records. Some of these ideas may seem important to you, while others may seem less important. Please tell us how important each of these ideas is to you when thinking about your overall opinion of this proposal. [GRID] Improving technology Costs Errors in medical records The health of average Americans The opinions of doctors Patient privacy [GRID] Not at all important Somewhat important Moderately important Very important Extremely important Please think again about the proposal regarding electronic medical records. Do opponents say we should judge the proposal according to… Its costs Its impact on health Both Neither Don’t know Appendix 3: Study 2 Treatments and Questionnaire Today we ask you to participate in four short studies. First you will read some newspaper excerpts and answer some questions about what you read. Second, you will complete another study and we will ask you some background questions about yourself. Third, we will ask you for your opinions about some political issues. Finally, we will ask you for your opinions about some different groups in society. Information Manipulation [RANDOMIZE INTO ONE OF FIVE CONDITIONS] <CONTROL GROUP> [SAME AS STUDY 1] 49 </CONTROL GROUP> <TREATMENT CONDITIONS> The first study you will participate in is about the quality of journalism. We are interested in how well journalists write about the news. You will read a few short excerpts from a news article and then we will ask you some questions to see how well you understood what was written. We are asking different people to read articles on different topics. You are being asked to read excerpts from an article about electronic medical records. <PAGEBREAK/> <Exercept1> Here is the first part of the article. Please read it carefully. DENVER, Colo. – Leaders of the American Medical Association have been busy at work this week in an annual conference held here in Denver. On Tuesday, the AMA's policy advisory committee began debate on a proposal that would change rules regarding how electronic medical records are handled by the nation's hospitals and clinics. The proposal would standardize medical record technology. If implemented, the proposal would mean <CostHigh> significantly higher costs for doctors and clinics. A typical clinic could see expenses rise by as much as $5000 per year under the proposed changes. Nationwide, administrative costs related to medical records could rise by as much as $17 million over the next two years. </CostHigh> <CostLow> significant cost savings for doctors and clinics. A typical clinic could see expenses reduced by as much as $5000 per year under the proposed changes. Nationwide, administrative costs related to medical records could be lowered by as much as $17 million over the next two years. </CostLow> <BenefitsHigh> that doctors, clinics, and hospitals would be able to share records more easily and be able to better serve patient needs and more quickly retrieve vital health information. This would improve the quality of health care for most Americans. </BenefitsHigh> <BenefitsLow> that doctors, clinics, and hospitals would use standardized software so electronic records are interoperable – meaning computer systems would rely on compatible formats. This would have little impact on the quality of health care for most Americans. </BenefitsLow> <PAGEBREAK/> <Comprehension> What organization was mentioned in the text? American Hospital Association 50 American Medical Association Association of Nurse Practitioners Doctors Without Borders Don't know </Comprehension> </Excerpt1> <Excerpt2> Here is the next part of the article . Please read it carefully. Electronic medical records have been the subject of debate within the medical field for more than ten years. These systems would replace paper records kept by individual doctors and clinics with sophisticated in-house computer systems. Numerous companies currently produce competing software. <CostHigh> The push for electronic medical records is seen by many experts as an expensive alternative to paper records. Startup costs for a new electronic medical records system can cost a five-physician clinic $32,000 plus $8000-$9000 in annual maintenance costs. </CostHigh> <CostLow> The push for electronic medical records is seen by many experts as an cost-saving alternative to paper records. A new electronic medical records system can save a five-physician clinic $8000$9000 per year in operating costs. </CostLow> <BenefitsHigh> The push for electronic medical records is seen by many experts as a clear way to improve the quality of Americans' health care. With these systems, doctors can more easily track patient health over time, and more easily catch risk factors for chronic and life-threatening diseases. </BenefitsHigh> <BenefitsLow> The push for electronic medical records is seen by many experts as a clear way to keep pace with changing technology. With these systems, medical coding is streamlined, simplifying recordkeeping processes for clinic staff, even if it will have no impact on patients' health. </BenefitsLow> <PAGEBREAK/> <Comprehension> According to the article, how long has the issue of electronic medical records been debated? Less than 1 year 1-5 years 5-10 years More than 10 years Don't know </Comprehension> </Excerpt2> 51 <Excerpt3> Here is the next part of the article. Please read it carefully. The Brookings Institution, an independent policy think tank based in Washington, DC, released a report this morning describing an in-depth analysis of electronic medical records in the United States. At a press conference, Brookings spokesman Will Richardson said that the report details how proposed rule changes currently being debated by the nation's leading hospitals, clinics, and doctors would <CostHigh> lead to significantly higher costs for doctors and patients. Clinics have been slow to adopt electronic medical records precisely because the costs are so high. </CostHigh> <CostLow> lead to significantly lower costs for doctors and patients. Clinics have been slow to adopt electronic medical records precisely because they have been waiting for these kinds of savings to become available. </CostLow> <BenefitsHigh> substantially improve the flexibility of the U.S. health care system, allowing patients and doctors to quickly share vital health information. The changes would also mean simplified handling of insurance claims, reducing hassle for American families. </BenefitsHigh> <BenefitsLow> likely lead to modifications of existing clinical software and the future development of uniform formatting requirements for electronic records. The changes would also mean modified rules for software developers to comply with the new standards. </BenefitsLow> <PAGEBREAK/> <Comprehension> In what city is the Brookings Institution located? New York City, New York Washington, DC Seattle, Washington Denver, Colorado Don't know </Comprehension> </Excerpt3> <Excerpt4> Here is the last part of the article. Please read it carefully. Back in Denver, the AMA conference continues through the end of the week where electronic medical records seem to be the big topic of debate among conference attendees. Both practitioners and technology developers are eager to see the policy advisory committee's recommendations, which are due by the end of the week. <CostHigh> 52 One doctor, Jan Miller of Oakland, CA, remained concerned about her clinic's bottom line. “Electronic medical records are a substantial change in clinical practice and my practice cannot afford such a system without passing costs on to patients.” </CostHigh> <CostLow> One doctor, Jan Miller of Oakland, CA, was confident that electronic medical records would reduce costs her in practice. “Electronic medical records are a substantial change in clinical practice and my patients are asking for this system because they know it will save them money.” </CostLow> <BenefitsHigh> One doctor, Jan Miller of Oakland, CA, was confident that electronic medical records would improve her patients' health. “Electronic medical records are a substantial change in clinical practice and patients will see the benefits of these systems right away. Electronic records help me make fewer errors and know my patients better.” </BenefitsHigh> <BenefitsLow> One doctor, Jan Miller of Oakland, CA, downplayed the importance of this week's debates. “Electronic medical records mean little for my practice. It's really same old, same old. I don't see how patients would be affected really.” </BenefitsLow> <PAGEBREAK/> <Comprehension> What was the profession of Jan Miller, the individual quoted in the article? Software developer Nurse Doctor Insurance agent Don't know </Comprehension> </Excerpt4> Information Manipulation Check [SAME AS STUDY 1] Distractor Activity [SAME AS STUDY 1] Demographics [SAME AS STUDY 1] Framing Manipulation The third study asks you for your opinion about a recent political issue. 53 Recently there has been some debate in Congress about a new proposal regarding electronic medical records. [RANDOMIZE INTO ONE OF TWO FRAME CONDITIONS] <CostFrame> Some support the proposal. Others are opposed because they say that we should judge the proposal based on whether it is costly. Indeed, much of the debate over the proposal now revolves around the question of costs. </CostFrame> <BenefitsFrame> Some support the proposal. Others are opposed because they say that we should judge the proposal based on whether it will affect the health of average Americans. Indeed, much of the debate over the proposal now revolves around the question of the proposal's impact on patients' health. </BenefitsFrame> Outcome Measures <CostFrame> Based on whether you think the proposal will mean higher or lower costs, to what extent do you favor or oppose this proposal? Strongly favor Moderately favor Somewhat favor Neither favor nor oppose Somewhat oppose Moderately oppose Strongly oppose </CostFrame> <BenefitsFrame> Based on whether you think the proposal will have a large or small impact on patients' health, to what extent do you favor or oppose this proposal? Strongly favor Moderately favor Somewhat favor Neither favor nor oppose Somewhat oppose Moderately oppose Strongly oppose </BenefitsFrame> How certain are you of your opinion on the proposal regarding electronic medical records? Not at all certain 54 Somewhat certain Moderately certain Very certain Extremely certain How important to you personally is your opinion about the proposal regarding electronic medical records? Not at all important Somewhat important Moderately important Very important Extremely important Framing Manipulation Check We are now going to list a few ideas that individuals have expressed when describing their opinions about the proposal regarding electronic medical records. Some of these ideas may seem important to you, while others may seem less important. Please tell us how important each of these ideas is to you when thinking about your overall opinion of this proposal. [GRID] Improving technology Costs Errors in medical records The health of average Americans The opinions of doctors Patient privacy [GRID] Not at all important Somewhat important Moderately important Very important Extremely important Do opponents say we should judge the proposal according to... Its costs Its impact on health Both Neither Don't know Appendix 4: Study 3 Treatments and Questionnaire Today we ask you to participate in four short studies. First you will read some newspaper excerpts and answer some questions about what you read. Second, you will complete another study and we will ask you some background questions about yourself. Third, we will ask you for 55 your opinions about some political issues. Finally, we will ask you for your opinions about some different groups in society. Information Manipulation [RANDOMIZE INTO ONE OF FIVE CONDITIONS] <CONTROL GROUP> [SAME AS STUDY 1] </CONTROL GROUP> <TREATMENT CONDITIONS> The first study you will participate in is about the quality of journalism. We are interested in how well journalists write about the news. You will read a few short excerpts from a news article and then we will ask you some questions to see how well you understood what was written. We are asking different people to read articles on different topics. You are being asked to read excerpts from an article about electronic medical records. <PAGEBREAK/> <Exercept1> Here is the first part of the article. Please read it carefully. DENVER, Colo. – Leaders of the American Medical Association have been busy at work this week in an annual conference held here in Denver. On Tuesday, the AMA's policy advisory committee began debate on a proposal that would change rules regarding how electronic medical records are handled by the nation's hospitals and clinics. The proposal would standardize medical record technology. If implemented, the proposal would mean <BenefitsHigh> that doctors, clinics, and hospitals would be able to share records more easily and be able to better serve patient needs and more quickly retrieve vital health information. This would improve the quality of health care for most Americans. </BenefitsHigh> <BenefitsLow> that doctors, clinics, and hospitals would use standardized software so electronic records are interoperable – meaning computer systems would rely on compatible formats. This would have little impact on the quality of health care for most Americans. </BenefitsLow> It would also mean <CostHigh> significantly higher costs for doctors and clinics. A typical clinic could see expenses rise by as much as $5000 per year under the proposed changes. Nationwide, administrative costs related to medical records could rise by as much as $17 million over the next two years. </CostHigh> 56 <CostLow> significant cost savings for doctors and clinics. A typical clinic could see expenses reduced by as much as $5000 per year under the proposed changes. Nationwide, administrative costs related to medical records could be lowered by as much as $17 million over the next two years. </CostLow> <PAGEBREAK/> <Comprehension> What organization was mentioned in the text? American Hospital Association American Medical Association Association of Nurse Practitioners Doctors Without Borders Don't know </Comprehension> </Excerpt1> <Excerpt2> Here is the next part of the article. Please read it carefully. Electronic medical records have been the subject of debate within the medical field for more than ten years. These systems would replace paper records kept by individual doctors and clinics with sophisticated in-house computer systems. Numerous companies currently produce competing software. <CostHigh> The push for electronic medical records is seen by experts as an expensive alternative to paper records. Startup costs for a new electronic medical records system can cost a five-physician clinic $32,000 plus $8000-$9000 in annual maintenance costs. </CostHigh> <CostLow> The push for electronic medical records is seen by experts as a cost-saving alternative to paper records. A new electronic medical records system can save a five-physician clinic $8000-$9000 per year in operating costs. </CostLow> <BenefitsHigh> Many also see it as a clear way to improve the quality of Americans' health care. With these systems, doctors can more easily track patient health over time, and more easily catch risk factors for chronic and life-threatening diseases. </BenefitsHigh> <BenefitsLow> Many also see it as a clear way to keep pace with changing technology. With these systems, medical coding is streamlined, simplifying record-keeping processes for clinic staff, even if it will have no impact on patients' health. </BenefitsLow> 57 <PAGEBREAK/> <Comprehension> According to the article, how long has the issue of electronic medical records been debated? Less than 1 year 1-5 years 5-10 years More than 10 years Don't know </Comprehension> </Excerpt2> <Excerpt3> Here is the next part of the article. Please read it carefully. The Brookings Institution, an independent policy think tank based in Washington, DC, released a report this morning describing an in-depth analysis of electronic medical records in the United States. At a press conference, Brookings spokesman Will Richardson said that the report details how proposed rule changes currently being debated by the nation's leading hospitals, clinics, and doctors would <BenefitsHigh> substantially improve the flexibility of the U.S. health care system, allowing patients and doctors to quickly share vital health information. The changes would also mean simplified handling of insurance claims, reducing hassle for American families. </BenefitsHigh> <BenefitsLow> likely lead to modifications of existing clinical software and the future development of uniform formatting requirements for electronic records. The changes would also mean modified rules for software developers to comply with the new standards. </BenefitsLow> The report also expects the change to <CostHigh> lead to significantly higher costs for doctors and patients. Clinics have been slow to adopt electronic medical records precisely because the costs are so high. </CostHigh> <CostLow> lead to significantly lower costs for doctors and patients. Clinics have been slow to adopt electronic medical records precisely because they have been waiting for these kinds of savings to become available. </CostLow> <PAGEBREAK/> <Comprehension> In what city is the Brookings Institution located? 58 New York City, New York Washington, DC Seattle, Washington Denver, Colorado Don't know </Comprehension> </Excerpt3> <Excerpt4> Here is the last part of the article. Please read it carefully. Back in Denver, the AMA conference continues through the end of the week where electronic medical records seem to be the big topic of debate among conference attendees. Both practitioners and technology developers are eager to see the policy advisory committee's recommendations, which are due by the end of the week. <CostHigh> One doctor, Jan Miller of Oakland, CA, remained concerned about her clinic's bottom line. "Electronic medical records are a substantial change in clinical practice and my practice cannot afford such a system without passing costs on to patients." </CostHigh> <CostLow> One doctor, Jan Miller of Oakland, CA, was confident that electronic medical records would reduce costs her in practice. "Electronic medical records are a substantial change in clinical practice and my patients are asking for this system because they know it will save them money." </CostLow> <BenefitsHigh> But when asked about how patients' health would be impacted, Dr. Miller said that "patients will see the benefits of these systems right away. Electronic records help me make fewer errors and know my patients better." </BenefitsHigh> <BenefitsLow> But when asked about how patients' health would be impacted, Dr. Miller said that "patients will see limited benefits with these systems. It's really same old, same old. I don't see how patients would be affected really." </BenefitsLow> <PAGEBREAK/> <Comprehension> What was the profession of Jan Miller, the individual quoted in the article? Software developer Nurse Doctor Insurance agent Don't know </Comprehension> </Excerpt4> 59 <PAGEBREAK/> Information Manipulation Checks [SAME AS STUDIES 1 AND 2] Distractor Activity [SAME AS STUDIES 1 AND 2] Demographics [SAME AS STUDIES 1 AND 2] Framing Manipulation [SAME AS STUDY 2] Framing Manipulation Check [SAME AS STUDIES 1 AND 2] 60