You are in the MOST Phase I website (1994-2003). The MOST Phase II website is available at: www.unesco.org/shs/most.

(also available in French)

The Definition of the Problem
Key Research Questions
Conceptual Framework
Research Methods
Relation of the Study to Social Transformation
Intended Outputs
Participating Institutions

Æ See also: Call for proposals, Federal Office for Scientific, Technical and Cultural Affairs, Belgium (French)

The Definition of the Problem The problem that the study addresses is the frequent disregard of the findings of sound research and evaluation in the making of social policy. Much social research is conducted for the avowed purpose of influencing policy decisions. Government agencies, international organizations, foundations, and professional associations sponsor research that is intended to provide guidance for wiser policies. The research is conducted in university departments, university research centers, institutes of science, not-for-profit and for-profit research organizations, and within operating service organizations. Much of the research (although by no means all) is thoughtfully done with careful attention to appropriate social science theories and valid research methods. Yet a sizable fraction of it never influences the audiences for which it was meant. The proposed research will examine the question of which kinds of research, under which conditions, have a better chance of making a difference in policy councils. The study, and the series of collateral studies in other nations for which it will provide a model, will help research sponsors and researchers to conceptualize, conduct, and disseminate research in ways that increase the likelihood that policy audiences will pay attention. That is the long-range objective: to help improve the use of evaluation evidence in the making of policy. A great deal is known from more than twenty-five years of research on "knowledge utilization" (KU), which is the term by which this phenomenon is generally considered. Many studies have investigated the characteristics that are associated with greater use of research findings: characteristics of the studies, of the dissemination mechanisms, of the researchers, or of the users. Scores of studies of knowledge utilization (KU) have been published, providing a cafeteria of answers to the question of what kinds of research are most apt to be used. Unfortunately, the answers have not converged. For example, some studies find that research quality is important for use; other find that the quality of the research is unrelated to how much influence it has. Some studies find that policy actors turn to research findings when they face a crisis situation; other studies find that crisis is not an environment favorable to research use. Much of the reason for the discrepancies in research results on KU has to do with differences in definition and in methodology. Regarding definition, researchers have taken different approaches to the meaning of "utilization." What does it mean that a particular study has been "used"? Some researchers expect that the findings of the research will determine the course of policy, i.e. it will change a decision from what it would have been in the absence of that research. Other researchers take a wider frame: they consider "use" as any serious consideration of the findings of research, whether or not they are actually followed. These latter researchers recognize that many elements go into the making of policy and that it is naïve to expect that research results will overpower all other interests, ideological commitments, and previous information in the issue-arena. If research findings are given a serious hearing, they have a chance of altering policy makers' understanding of the issues, their priorities and agendas, and even their subsequent actions further down the road. On the other hand, if one takes a liberal definition of "use," how can the KU researcher truly know whether the research results have received serious consideration? Regarding differences in methodology of KU studies, a variety of methods have been used. Studies on KU have been conducted through quantitative surveys, qualitative interviewing, analysis of the fate of particular research studies by the researchers who conducted them, analysis of documents, and through prospective, retrospective, and simulated time frames. The studies have dealt with such varying issue-arenas as housing, energy, mental health, and education. The research on KU has been conducted at different periods of time, much of it in the late 1970s and early 1980s, with another upsurge in the last few years. Most of it has been conducted in the U.S., although studies have also been conducted in Western Europe and occasionally other places as well. It is small wonder that the findings on KU do not cohere into a tidy package of accepted truths. It is the aim of the proposed study to develop a better understanding of the factors that enhance wise use of research results in the making of policy. Key Research Questions The key questions have to do with factors that advance the utilization of research results and the obstacles that thwart the use of research results in the policy arena. Specifically, which characteristics promote research use and which block research use? We will consider such characteristics in four main categories: Characteristics of the research and the researchers who conduct it Characteristics of modes of dissemination or linkage between researchers and the policy arena Characteristics of potential user groups Characteristics of the political domain that the research enters Let me give examples of the kinds of features in each category. Regarding research and researchers, it is possible that research undertaken by well-known and respected researchers who work in respected research institutions will be better attended to than research from novices in provincial institutions. Other hypotheses are that research of better methodological quality is more often heeded than research of lower quality, or that quantitative research receives a better hearing than qualitative research. In similar vein, it may be that results of meta-analyses or syntheses of a body of literature are more influential than single studies. Regarding modes of dissemination, particular attention will be devoted to four strategies for making research findings known. One is the ordinary mode of academic publication. Research results are reported in professional journals and in reports to the agency that sponsors the research. A second type of dissemination is inclusion of stakeholders in the research process, so that they have a say in shaping the study from the start and remain close to findings as they emerge. This is currently one of the most widely advocated mechanisms for disseminating research results to prospective users. By engaging stakeholders in the research, it tends to make the research more relevant to users' needs and to keep them apprised of emerging findings during the research so as to avoid surprises. However, when the intent is to engage policy makers, this strategy assumes a willingness to participate in the research process and a continuity of attention that few policy makers appear willing to sustain. A third strategy of dissemination that will be studied is networking. This is a "political" strategy, akin to lobbying, that relies on face-to-face personal contacts with influential elites. Some U.S. research groups that aim to reach members of the Congress contact staff and members of important Congressional committees to convince them of the relevance of the research results and their implications for policy. Finally, one may come across cases where research was picked up and disseminated by others (i.e. not the researcher or research organization), such as the mass media, think tanks, a research "champion" who takes it upon him/herself to spread the word, or an "issue network" of issue activists from different institutional bases who stay in contact with each other and share information. Characteristics of potential user groups are also likely to make a difference in what kind of hearing the research receives. For example, in Germany and France, many high officials in state bureaucracies have been educated in the law and relatively few have any background in the social sciences. There is some evidence from the U.S. and Canada that policy actors trained in law are less responsive to social research and evaluation than officials with different kinds of training. Another characteristic of policy actors that appears relevant from some KU research is how long they have worked in the issue- arena. Relative newcomers often seem to be more receptive to research findings than those who have long experience and believe themselves well informed about the issues. Other such characteristics will receive consideration. Characteristics of the issue-arena are clearly important, and yet these are the factors that have received least attention in the KU literature. It seems important to investigate whether some policy fields (e.g., macroeconomics, criminal justice) are more amenable to research and evaluation than others. Similarly, the degree of polarization in the politics of the issue would seem to make a difference, as would the intensity of lobbying from interested parties. The extent to which research results run counter to the politics of the arena can be expected to affect the degree of influence that research and evaluation findings will have. Conceptual Framework It is proposed to develop case studies of a series of research studies (or syntheses of studies). Many earlier studies have used quantitative methods, attempting to quantify the effects of different factors on the utilization of research findings. Studies based on surveys, content analyses, and other quantitative approaches have provided useful understanding. However, each quantitative study tends to define "utilization" in somewhat different terms. They also use somewhat different variables and create the variables from different questions. Therefore, they develop different models of factors affecting the use of research and different estimates of the effects of various features on the extent of research utilization. A point has now been reached in the KU field where further findings of this sort are not likely to contribute substantially to further knowledge. We now need to take context into account and to do so systematically. We also need to use common theoretical frameworks, common concepts, and common variables in the investigations. That is why a series of case studies, each of them employing the same conceptual framework, makes the best sense at this point. Our plan for the case studies involves deep and thorough investigation into: (1) the initiation of the study (Who initiated it? For what reasons? For what purposes? Did the initiators evince concern for the application of results of the research to practical action? Was there a formal proposal or Request for Proposals? How was the research performer chosen? Who funded the study? (2) review of the proposal for the study Who judged its merits? Were there conflicting opinions about the worth of the study? Was the expected usefulness of results a criterion for approval? Did reviewers consider the methodological quality of the proposed study? Did they consider the academic reputation of the researcher(s) and/or their organization? (3) the researcher(s) What was their reputation as researchers? What was the reputation of their research organization? Had they done research of this kind before? What was their disciplinary affiliation? (4) conduct of the study How long did the study take? How many people worked on it, in what capacities? Were stakeholders, including policy makers, involved in the conduct of the study during its course? Did the research have an explicit theoretical basis? What research methods were used? How were data collected? How were the data analyzed? Did the methodology adhere to sound technical principles? What kind(s) of report(s) were written? (5) findings What did the findings show? Did they provide clear direction for action? Were the implications of the research a need to make changes in current policy or programming? Big changes or small changes? Did the findings run counter to established policy or agency interest? Would implementation of the findings be expensive in financial terms? Would implementation of findings require changes in agency structure or standard operating procedures? (6) dissemination of results How were results of the research communicated to research audiences? to policy making audiences? Did the researchers speak at conferences, or training sessions for professionals, or in public forums? Did researchers attempt to reach policy makers with their findings? Was there a "research champion" outside of the research team who publicized the findings? Did the media report any of the findings? Did other "intermediaries," such as think tanks or interest groups, communicate the findings to policy audiences? (7) political context What was the history of the issue-arena, e.g. decentralization of educational administration, privatization of day care provision? Had the topic been debated before? Were there clear supporters and opponents of the direction that the research supported? Were divisions between supporters and opponents long-standing and firm? Did the issue-arena respect research evidence? What other research findings, if any, were in currency? What other information was being supplied to decision makers? What groups were lobbying decision makers in support of which causes? Did any of the lobbying interests use research findings in making their case? Was their time pressure for reaching a decision? Were decision makers facing a crisis in which existing solutions were unworkable? (8) prospective users Who were the most appropriate users of the research? What positions do they hold? What is their disciplinary background? How long have they worked in the issue-arena? Did they learn of the findings of the research? Through what channels? If they heard about findings, did they believe them? Did they believe that the findings were relevant to their issues and the conditions they faced? Did they interpret the findings fairly, or did they misinterpret or distort the findings? (9) utilization Did decision makers, in fact, use the findings to reconsider what they were doing or planning to do? Did they make changes? If so, were they big changes or small changes? Which findings were ignored? How much time elapsed before they implemented changes related to the research? What factors promoted consideration of the research? What factors inhibited consideration of the findings? How satisfied were decision makers with the extent to which the research served their needs? Research Methods Documents and records will be used for data collection wherever possible. For example, information on the length of the study and its funding will be sought in records, as will characterization of the history and configuration of the issue-arena. We will also use records to learn the names and current locations of the individuals involved in the case, as initiators, funders, researchers, reviewers, disseminators, and prospective users. These people will be the subjects for interviews. Where records of participants are incomplete, we will begin interviewing and ask respondents for the names of other people who were involved. Most of the information will not be available in records or documents, and we will need to interview participants in the research and utilization processes. Interviews can be conducted with multiple actors in the case, in order to hear about many different perspectives. Multiple interviews will also allow identifying and disregarding any erroneous information supplied, either through ignorance or for self-serving purposes. It is anticipated that respondents will offer different interpretations of events not so much because of outright falsehoods as because they see events differently. A serious hearing will be given to the various perspectives, that will then be used to make sense of the case. Before each interview, a review will be undertaken of whatever materials that can be found about the individual and his/her role in the case. Each interview pursues questions that are appropriate to that role. A master list of questions exists, drawn from sections (1) – (9) above, and use will be made of questions suitable for each individual. To arrange interviews, it has been found in other work that the best approach is a phone call, followed, if the respondent desires, by a letter explaining the purposes of the study and its methods of inquiry. Shortly thereafter a second call is made to set up the interview. Each case will be analyzed as a whole. Quantitative data will be combined with the narrative accounts from the interviews and attempt to reconstruct the "story" of the confrontation of the research with the debate and negotiation in the issue-arena. Through the information collected, it will be able to test various hypotheses about conditions that promote careful attention to research findings. For example, it will be possible to analyze whether involvement of prospective users in the research process enhances KU, whether policy makers with social science backgrounds are more likely to take research results seriously, whether long-standing contention between factions thwarts the use of research. If the series of case studies in different countries follows similar frameworks of inquiry, this will provide the MOST Programme and its constituents with a great deal of knowledge about the use (and non-use) of research. Our intent is to draw conclusions, even if tentative, about the extent to which various mechanisms advance the use of research in social transformations. Relation of the Study to Social Transformation International agencies and national governments are seeking to reduce poverty and develop national capacities to cope with the social problems of the 21st century. Many of them sponsor social science research to help them chart appropriate and feasible courses of action. Research, it is assumed, will provide direction for fruitful responses to the social ills that affect the planet. An example of this research strategy is MOST's Comparative Research Networks (CRNs). The international agencies and national governments that fund research expect that policy makers and officials in both governmental and non-governmental locations will heed the findings of research and thereupon make more efficacious and cost-effective policies. However, experience shows that many policy making bodies take actions that ignore, or are even diametrically opposed to, the directions recommended by high-quality research. This proposal is premised on the belief that the world's extensive investment in research is paying fewer dividends than it should because we do not yet understand all the conditions that would encourage policy makers to attend to research results. The problem is endemic to all areas of the world. It is being analyzed, to more or less extent, in the developed world and in the developing world, in democracies and in authoritarian regimes. The research proposed here is intended to remedy the shortfall between the knowledge embodied in first-quality social science research and the actions taken by international agencies, governments, and NGOs in pursuit of social transformation. If the findings of this study can help to improve the translation of research results into appropriate policy, an important contribution will have been made. Intended Outputs Findings will be widely reported in research journals and in policy journals, in an attempt to reach researchers who undertake studies intended to have a policy impact and research funders who support such studies. Thus we plan to submit papers reporting our findings to such journals as the International Social Science Journal, Policy Studies Journal, Journal of Policy Analysis and Management, Evaluation: the International Journal of Theory, Research and Practice, Educational Evaluation and Policy Analysis, Knowledge and Policy, American Sociological Review, and similar journals in countries around the world. Furthermore, consideration will be given toward developing training programs for specific groups of research users, such as upper-level bureaucrats, legislators, program managers, and journalists. If the findings point to fruitful lessons for such audiences, collaborative work with the MOST staff will be undertaken to develop training that appeals to the particular interests of each group and conveys the major messages of our case studies. Participating Institutions LIST OF COUNTRIES AND INSTITUTIONS PARTICIPATING AS OF MAY 2000 One case study is proposed in the United States. It is included here as a model for information for other countries who may wish to participate in this venture. The MOST secretariat hopes to be able to recruit researchers and research institutions in other countries to do case studies of research on their terrain. This proposal is being sent to Member countries of the MOST Intergovernmental Council to solicit interest in participating through a case study. The Case study as developed for the United States is included below as a model.   First Case Study by Professor Carol Weiss, Graduate School of Education, Harvard University The first case I propose to do in the U.S. is a follow-up of the Comprehensive Child Development Program (CCDP) evaluation done by Robert St. Pierre, Jean Layzer, and others at Abt Associates. It is a five-year randomized experiment evaluating a nation-wide program of comprehensive services to young children and families in troubled homes. The evaluation is highly sophisticated, based on long-term follow-up, standardized measures, interviews and observations, and sophisticated statistical analysis. The study was requested by the Administration for Children, Youth, and Families in the U. S. Department of Health and Human Services. Abt Associates was selected as the research performer through a competitive process. The report on the study was issued last year. Basically, the evaluation found that while the young children and families receiving services improved on a number of dimensions, they did not improve any more than did a randomly selected control group. The obvious implication is that the extensive services provided to families, which were mediated by a case manager for each family, did not help the families; they would have done as well on their own. The CCDP program has expired. In its place Congress has legislated the Early Head Start program. The Early Head Start legislation was adopted prior to the release of the Abt report, and many grants were made to local agencies to carry out Early Head Start in very much the same style as CCDP. Many of the local groups that formerly conducted Comprehensive Child Development Programs now receive grants from Early Head Start, and they are reportedly providing much the same kinds of services as they did under CCDP. Early Head Start mushroomed before the evaluation that revealed its limitations was available to decision makers. Immediate users for the study findings should be the administrators and staff of the Early Head Start program. The researchers at Abt Associates say that these federal officials' first reaction was to criticize the study and hold up the release of the report, but that now they are giving consideration to some of the findings. It is possible that Early Head Start is being altered because of the nature of the findings, that for example, more latitude will be given to local grantees to vary the activities from the set formula of CCDP.. We will investigate what the federal officials in the CCDP office expected from the evaluation initially, and what they think of the report, as well as reactions from Early Head Start administrators and staff, and local Early Head Start programs. We will also talk to staff members of Congressional committees that have cognizance over the program. We want to find out whether they have heard about the findings, how much credence they give them, and whether the findings have raised questions or modified their agenda for future legislation. Furthermore, we will visit at least three local programs, probably now funded under the Early Head Start legislation. Two sites will have previously implemented CCDP and at least one will be newly funded as an Early Head Start site. We want to know whether local administrators pay attention to evaluation findings, how they interpret the findings for their own program, and whether they are thinking about changing activities, clients, record-keeping, or other facets of their program. We intend to pursue the range of questions adumbrated in the outline of methodology above. Work Plan There are three parts to the work plan. The first is the conduct of the case on the comprehensive child development program. The second is the creation and maintenance of the comparative research network that will undertake parallel case studies in other countries. The third is the comparative analysis of the whole set of case studies to draw generalizations. This section outlines the schedule for the first task. I foresee that the case study will take about 18 months.

To MOST Clearing House Homepage