» 150 readers of the UNESCO Science Report air their views
14.03.2018 - Natural Sciences Sector

150 readers of the UNESCO Science Report air their views

© stockfour / Shutterstock.com.

In early 2017, UNESCO contracted the independent Technopolis Group in Paris to conduct an Evaluation of the UNESCO Science Report published in late 2015. The evaluation was presented to UNESCO’s Executive Board in October 2017. Here, we compare the findings of a stakeholder survey conducted as part of this evaluation with those of a reader satisfaction survey available on the UNESCO Science Report portal.

In the Evaluation’s survey of 99 users of the UNESCO Science Report, almost half of respondents came from academia (21) and National Commissions for UNESCO (24). The latter are government relays in Member States for the implementation of UNESCO’s programme. A further 13 respondents identified themselves as policy-makers in government and four as employees in the private for profit or non-profit sector. Eleven were employed by UNESCO field offices or by the UNESCO Institute for Statistics. Two-thirds of respondents were men.

Respondents said they used the report most for conducting research (56%), for learning purposes (51%), for monitoring and benchmarking (48%) and for policy advocacy (47%). Only 16% of respondents declared using it for fundraising purposes.

Report ‘fully in line’ with mandate and 2030 Agenda

Drawing on the survey’s findings, the Evaluation concluded that ‘the production of the UNESCO Science Report is fully in line with UNESCO’s mandate’… This was confirmed by all of the members of UNESCO staff and Member State Permanent Delegations interviewed as part of this evaluation’. Users considered the UNESCO Science Report ‘to be relevant and coherent vis à vis other UNESCO initiatives and tools in the field of science, technology and innovation (STI)’

In particular, the Evaluation found the UNESCO Science Report to be ‘in line with the ambitions of the 2030 Agenda for Sustainable Development. Specifically, the UNESCO Science Report is compatible with Sustainable Development Goal 9 (SDG 9), the pledge by countries to “build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation”. Target 9.5 calls upon countries to encourage innovation and substantially increase the number of researchers, as well as public and private spending on research and experimental development (R&D).

The evaluators found that ‘the case for continuing to support the UNESCO Science Report is strengthened by its high degree of relevance and the overall positive appreciation expressed by readers, in particular with a view to the significant potential for the UNESCO Science Report in contributing to influencing and monitoring progress towards the SDG target 9.5’. The Evaluation found that ‘the UNESCO Science Report could also potentially contribute to monitoring the contribution of science, technology and innovation to reaching other SDGs’.

Report provides baseline indicators

Despite the fact that OECD members accounted for 66% of global research spending in 2013, compared to 5% for low income and lower middle-income countries, two-thirds of geographical coverage in the UNESCO Science Report concerns the developing world, as information and data for these countries tends to be less readily available. The Evaluation found that the report ‘plays a distinctive role in providing evidence, data and information on the state of STI at the global level’ but that, ‘compared to other comparable data sources, the level of USR data is considered as basic’.

In order to ensure that as many countries as possible are able to contribute data, the UNESCO Institute for Statistics limits its range to basic R&D indicators, such as researchers per million inhabitants and gross domestic expenditure on R&D as a percentage of GDP, the two baseline indicators for Sustainable Development Goal 9.5.

‘UNESCO, the UNESCO Institute for Statistics and the UNESCO Science Report are seen as legitimate sources of information to monitor this target’, noted the Evaluation. The UNESCO Institute for Statistics ‘is generally considered as a complementary source of “raw data” on STI at the global level, rather than as a competing source of information’.

Report useful ‘bundler’ of information

For the Evaluation, stakeholders and users have clearly identified the uniqueness of the UNESCO Science Report vis à vis external information and data sources. Just over ‘half the evaluation survey respondents (53%) indicate that the information contained in the UNESCO Science Report is not available elsewhere, while a remaining 32% consider it to be only partially available elsewhere. In this sense, the USR appears to be considered by many as a “bundler” of information which may or may not be available elsewhere, which in itself represents an added value to the reader.

One survey respondent stated that ‘I mostly look for country/regional trends. These data are available but in disparate sources. The UNESCO Science Report can act as a one-stop-shop for such information’. A second respondent commented that ‘as it is public information, it can be obtained from other sources too but in the [UNESCO Science Report] it is presented in a convenient form for a researcher”.

The Evaluation observed that, ‘while other international organizations producing similar content (e.g. OECD) are also highly valued and recognized for the quality and value of their work, these tend to focus less on the social and economic implications of STI in countries outside of their constituencies (e.g. OECD member states, EU member states) and are thus to be viewed as less “global” in terms of data availability’.

Geographical balance and thematic relevance ‘good or very good’

‘Over 80% of [survey] respondents considered the geographical balance of the report to be good or very good. Despite this, ‘a number of interviewees indicated that, rather than seeking to achieve a global coverage, the UNESCO Science Report should focus on countries – particularly developing ones – where the lack of reliable and up-to-date data on STI is still considered to represent a major roadblock to the promotion of STI policies and support’. In addition to the survey, the evaluators interviewed 33 people familiar with the report, including 10 UNESCO staff members and two authors.

‘Around 90% of respondents considered the relevance of the themes covered by the report to be good or very good’. ‘In addition to this, the level of use and relevance of the different components and sections of the UNESCO Science Report appears to be roughly the same… The frequency of use of the different parts of the UNESCO Science Report appears to be spread evenly, according to survey respondents. The Executive Summary does appear however to be the most frequently consulted section of the report, but only by a slight margin’.

‘In spite of this, the qualitative interviews did reveal that there are specific chapters and sections of the 2015 edition of the UNESCO Science Report which are more frequently cited, when it comes to describing the value and use of the report. For instance, Chapter 3 of the report on the gender gap in science and engineering was frequently cited by interviewees as one of the pieces contained in the report of particular interest. In addition, readers tend to cite the country or regional chapters of their home countries and regions as the sections of the report they most often consult, or have read in detail.

Yet, according to the website metrics analysis, the number of visitors to the individual chapters’ webpages differed significantly. Chapter 15 on Iran was visited the most, with 1,479 visitors [by early 2017], followed by Chapter 3 on ‘Tracking trends in innovation and mobility’, with 986 visitors, and by Chapter 1 (Executive Summary) entitled “A world in search of an effective growth strategy”, with 821 visitors.

This compares with 8,855 views for the same English-language document over the 18 months to May 2017 when it was presented in booklet form as the Executive Summary rather than as Chapter 1. This corresponds to an average of 492 visitors per month. The full report in English was downloaded 56,223 times over the same 18-month period, an average of 3,124 times per month.

Survey participants also expressed a high level of satisfaction with the presentation and visual style of the report and its format: over 80% of respondents considered these aspects of the report to be good or very good.

Some respondents were critical of UNESCO’s communications on the report, the quality of the website and the frequency of release of the report. Many felt that a report that had grown to 800 pages, including the Statistical Annex, was now too voluminous a volume, even it did ensure global coverage.

Six out of ten users (62%) preferred the online version of the report, one in five (21%) the hard copy and 18% alternated between the two formats.

Internaut survey mirrors findings of Evaluation

Readers surveyed on the UNESCO Science Report portal in 2016 and 2017 in English and French also rated the report highly. Three-quarters of respondents rated it ‘excellent’ (27%) or ‘very good’ (47%). The remainder rated it ‘good’ (20%) or ‘adequate’ (6%). None considered it ‘poor’.

Thirty respondents to the online survey identified themselves as men and 21 as women. Originating from countries on every continent, they spanned all age groups: 5 were less than 30 years old, 10 were aged 30–39 years, 9 were aged 40–49 years, 7 were aged 50–59 years and 20 were 60 years of age or more. Two had retired from professional life.

The majority of respondents were researchers or policy analysts in the natural or social sciences and engineering. Many indicated more than one affiliation. For instance, some academics also worked for a private research institute, non-governmental organization or academy of science. One respondent described himself as a researcher, policy analyst, entrepreneur and journalist.

Only two respondents indicated that they worked for a government department or ministry. A third was a school teacher. Six stated that they were university students and eight that they worked in the private non-profit or for profit sector.

One-quarter of respondents (13) had heard about the report by word of mouth. Others had discovered it in the workplace (3), traditional press (2) on social media (3) and while surfing internet (3). Seven respondents said they followed the series, two had discovered the report at a library, six at a conference and one at an event presenting the report’s findings. Six had learned about the report through an academic article, one through a research project and another via a university course. (Tallies may not add up to 51, owing to some respondents not answering all questions.)

Almost half of respondents (47%) said they had only read selected chapters from the report. A further 27% said they had read most of the report and 12% claimed to have read the full report. Some 8% had only read the chapter on their country or region, 2% had searched for keywords in the report and 4% had read only the webpages.

When asked to identify those parts of the report they found most useful (multiple answers were possible), 45% of respondents highlighted the socio-economic and geopolitical background to trends in STI, 41% the country profiles, 27% trends in innovation and 26% data trends.

The next most popular categories of information were the policy recommendations in each chapter (24% of responses), the development of a national policy framework (24% and national scientific infrastructure (24%), scientific mobility (22%) and science diplomacy (20%), emerging priority areas for R&D (20%), trends in international scientific collaboration (18%) and sustainable development (16%), gender-related issues (12%) the quantifiable targets for trade, R&D and sustainable development (10%) business R&D (10%) and short essays on emerging issues (8%). A further 12% of respondents ticked the option ‘all of the above’.

Some 55% of respondents planned to use the report for policy- or other decision-making, 47% for an academic research paper or to prepare a report, 45% for their personal knowledge, 26% to inform project development or international cooperation, or for a presentation to a meeting, 22% for advocacy, 8% for market research or for a journalistic article or blog and 6% as input to course teaching. Three respondents were exclusively interested in the data.

One in ten respondents said they found the report hard to find with a search engine and a similar proportion had trouble opening the full report in PDF format (82 MB). Chapters may also be downloaded individually from the portal, an option which offers the same material in a lighter format.

Not too late to share your opinion

It is still possible for internauts to make their opinion of the report known. All people need to do is answer the multichoice questions in the online reader satisfaction survey.

Source: Reader satisfaction survey and Evaluation of the UNESCO Science Report.

<- Back to: All news
Back to top