Global Co-ordination > Working Group on Education for All >
First meeting / Document 14
EFA Observatory
based in the UNESCO Institute for Statistics
Denise Lievesley

Background

There was clear agreement based on the experience of the first ten years of the EFA programme that regular monitoring of the state of education throughout the world must be an essential aspect of the follow-up to Dakar. A rush to acquire data towards the end of the evaluation period is wasteful of resources, inefficient and of limited value. The aim for the next stage in EFA assessment must be to establish reporting systems which inform the development of policies in addition to providing a monitoring role. Early warning systems must be implemented to indicate that policies should be amended when it is estimated that the targets will not be achieved.

Objectives

The objective of the EFA Observatory is to collect, analyse and disseminate up-to-date information on the state of education required by countries, regions and the international community monitoring progress towards the goals of Education for All. Regular assessment must be focussed on helping Governments develop, review and amend national policies as necessary to ensure that the EFA goals are achieved in all countries and as soon as possible.

Principles

It is important that the work of the EFA Observatory be founded on a set of basic principles. These need to be debated and agreed by the relevant parties but initial suggestions include:

  • There should be no duplication of data collection

  • International agencies should collaborate in their statistical programmes and should share data

  • Data collection should reflect national needs with national statisticians being consulted about the international database

  • National statistical priorities should not be distorted by international demands

  • Wherever possible, data collection should utilise existing sources

  • A key element of the data strategy should be the building of technical capacity in data collection and analysis including international comparison

  • The data should be returned to the countries and assistance given to ensure their effective exploitation

  • Data analysis and interpretation should be culturally sensitive

    • A possible additional principle for adoption is that data should be owned by countries themselves. This could imply that countries would have control over what is published. However, this would conflict with the need for the EFA Observatory to assert its independence by refusing to publish, or publishing with caveats, certain data. In particular the EFA Observatory must have the right to question the integrity of any data it receives. The issue requires discussion with representatives of countries in a way that is sensitive to their situations.

    Data Problems

    Experience of the EFA Assessment to date has highlighted many problems with the data quality including:

  • The inability of some countries to provide any data

  • Incomplete or inconsistent data over time · Incomplete data within a country - with data relating to particular sectors such as private education being particularly difficult to obtain

  • Inconsistencies of data within a country - especially where the data have been supplied by different ministries

  • Inadequate implementation of the international standards and classifications leading to data which are not comparable across countries· The loss of experienced staff from statistical work leading to continuity problems and resulting in poor quality data

  • An over-reliance on data from administrative sources and the lack of other data with which to validate information

  • Long time lags before data are processed and available

  • The absence of information on quality of the data

  • Poor IT infrastructure and weak analytic skills leading to under-exploitation of the data within a country

  • Some countries do not have data systems of high integrity and the data have been politically manipulated

    • Overcoming these data problems

    There are a number of ways in which the EFA Observatory might try to tackle these problems.

    Supportive networks

    One of the main ways is to forge strong and continuing links with the relevant statisticians, consulting them about the content and form of data collection, involving them in the design of data collection instruments and ensuring that they are supported through a network of people who understand the problems they face. This can be best achieved by integrating the EFA data collection into the UIS regular data collection, as far as possible. so that the resources of the Institute in working with national statisticians can be used most effectively. The UIS has already conducted a series of nine regional workshops this year in order to lay the groundwork for this. The workshops will be reconvened in a few months in order maintain the momentum. The experience of the World Education Indicators project (funded by the World Bank and carried out jointly by the UIS and the OECD) is proving very valuable in this respect. Similarly the example of the Indicators project of the Summit of the Americas is very enlightening.

    Statistical Capacity building

    As indicated earlier, it is also vital that there is recognition of the ongoing resources and expertise needed to collect timely and policy relevant data. The importance of integrating data collection and statistical capacity building cannot be over-emphasised. However this is neither a cheap option nor a fast solution. A difficult balance has to be struck between the need to obtain an immediate snapshot of the situation in countries which have hitherto not supplied adequate information and the responsibilities we all share to build sustainable statistical systems. It will be essential that the EFA Observatory collaborates fully with other statistical capacity building initiatives in order to stretch our resources as far as we can.

    A particularly relevant initiative is PARIS 21 (PARtnerships In Statistics for the 21st Century), a programme launched at a joint UN, OECD, IMF and OECD meeting in Paris in November 1999. This has as an aim the sharing of resources for statistical support and ensuring that capacity building is integrated into national statistical plans which serve the policy needs of countries. A role of the EFA Observatory is to give statisticians from ministries of education, and other line ministries, a voice in the development of such plans. It is vital that even the most de-centralised statistical systems take account of the career and resource needs of these statisticians. Having said this, the interest of the EFA Observatory extends also to national statistical agencies and finance ministries which supply critical population and educational finance data especially since many of the problems of the recent EFA assessment were due to internal inconsistencies in population data and gaps in finance data.

    Mechanisms need to be devised to identify countries with particular training or infrastructural requirements, and to match these to available resources.

    Advocacy

    A major activity of the EFA programme is advocacy and this should include advocacy for statistics: raising the public and political profile of official statistics is vital in order to win support for the continued allocation of resources to data collection programmes and to ensure that relevant groups (such as head teachers and education administrators) have a sustained commitment to supplying the information. There are a number of ways in which such advocacy can be developed. The communication skills of statisticians need to enhanced in order to promote the cause with the media and the policy makers. The role of the EFA Observatory could be to organise a series of regional workshops on this topic and to share relevant materials across countries.

    Promoting data use

    Support for data collection is contingent upon data use. The more that data are used for national purposes, so that policy makers see a need for them, the more they will ensure that resources are available for data collection. It is important that any new data collections are driven by policy needs. This is not simply a matter of filling every conceivable data gap especially since developing new data sources is resource intensive.

    It is important to develop an environment of openness with regard to data access in countries for purposes of accountability and to permit open discussions of data quality. There are two imperatives to help researchers within countries to have access to their own data:
    a. to ensure local commitment to the activity and

    b. because ideally data should be analysed by those who understand their provenance. Too often data are analysed only at an international level.

    So there must be mobilisation of efforts to establish national data resource centres which preserve and provide access to data and support users. The EFA Observatory could administer a programme of research scholarships and run data confrontation summer schools.

    Following the UIS's move to Montreal in August/September 2001, there will also be opportunities (provided by the Institute's location in a university setting) to have PhD students attached to the EFA Observatory. Young researchers from all over the world would be able to work on their countries' data in collaboration with the UIS staff and gain a respected qualification at the same time.

    Data Quality

    The most significant ways of improving data quality can only take place at the country level where as indicated above a culture of data use must be encouraged. However it is also essential that the EFA Observatory develop and make explicit quality assurance processes to identify weak data and to help countries take remedial action. The development of quality assurance processes capable of such filtering is facilitated by the availability of multiple sources of data so that inconsistencies can be identified. This process is sometimes known as 'triangulation'. Thus in developing the indicators the EFA Observatory will need to explore alternative data sources.

    Determining the set of indicators

    A first priority for the EFA Observatory is to widen consultations with users and producers of the EFA data in order to identify

  • Which indicators should be retained in the regular monitoring programme

  • Which indicators need to be amended in order to improve their efficacy for monitoring purposes

  • Gaps in the set of indicators

    • This process will build on the initiatives already taken. The UIS has begun the consultation with data producers in order to take advantage of the regional workshops conducted in June and July 2000 and to ensure that the interim data collection launched for 2000 and 2001 will yield some of the data needed for EFA monitoring.

    Consultations about the gaps in the indicators, as well as weaknesses in current indicators, have already suggested that the EFA data should be greatly strengthened in relation to

  • The quality and outcomes of formal education

  • Educational inequity, within-country analysis, and information on children not in the educational system

  • Measures of literacy, both in the school and the adult populations

  • Non-formal education

  • Early childhood education

  • Efficiency - the ways in which countries ensure that they achieve value for the investment

  • Financing of education - public and private partnerships and other strategies for generating resources

    • The EFA Observatory is keen to consult widely to ensure that it takes account of all perspectives on this issue. Part of the consultation exercise must be to identify partner agencies who are already collecting some of these data or who are willing to extend their current data collection methods to incorporate the needs- it is important to appreciate that there is no intention that the EFA Observatory should be the lead agency for all data collection.

    The outcomes of these consultation exercises will be 'data maps' at the international level. Painstaking work is then required with country representatives to translate these into programmes of data collection.

    Developing new indicators

    It is important to be realistic about the EFA Observatory's work. New indicators cannot and should not be implemented fast. For each new area of work methodological development may need to take place. Wherever possible advantage will be taken of existing expertise in an area - for example any programme developing literacy indicators would take account of the work of the OECD, Statistics Canada and the International Literacy Institute in Philadelphia, USA. But experience gained in one environment cannot necessarily be applied elsewhere. Thus it may be necessary to carry out feasibility and then pilot studies to develop the appropriate methodology for the collection of new data. A balance must be achieved between the need for speedy monitoring and the collection of quality data in a way which can be sustained over time.

    Task forces of experts (both users and producers of data) will be established with respect to each main category of data development. Wherever possible these task forces will be consulted electronically but occasionally it may be necessary to hold meetings. An example of such a meeting is the workshop on identifying gaps in the data on financing of education systems which the UIS is running in early December 2000. This is part of a scoping exercise and the outcome is expected to be a programme of methodological development on the collection of finance data.

    One of the most difficult aspects of the task of the EFA Observatory is to determine priorities for new developments. There will be advocates for every area of educational data but the reporting burden on countries must be borne in mind. There are already pressures within the ECOSOC system to curb what is seen as a proliferation of indicators. Thus the UIS's role as a bureau member of the ACC Sub-Committee on the Co-ordination of Statistical Activities is important since major data developments need to be discussed within this committee. Similarly consultation will take place at the UN Statistical Commission in March 2001 in New York.

    Reporting on the indicators

    A priority will be to disseminate the raw data together with the indicators as widely as possible. They must be underpinned by full descriptions of the methodology (and other associated metadata) to enable users to judge fitness for purpose. The EFA Observatory's aim is to promote the informed use of these data. Web-based access systems will be developed to deliver the data as cheaply as possible and to give a wide coverage to the EFA results. However it is essential that the EFA Observatory is also sensitive to the fact that not all users have access to the internet and so other methods of electronic delivery will be available together with the services of a user support officer who will be able to provide customised data delivery and support. We will deliver data in geographical information systems and will explore other developments in data delivery.

    The UIS, in its new location at the University of Montreal, will have a resource centre with facilities for visitors. This means that users will be able to work with staff who have direct experience of the data. It also gives the opportunity for partnerships to be forged with University staff and students in the analysis of the EFA data. As already indicated not all of the data will necessarily have been collected by the EFA Observatory. The EFA Observatory will however have responsibility for gathering together the data from different sources. In addition the quantitative data will be merged with relevant qualitative, contextual and policy information in order to facilitate appropriate interpretation of the data. This will necessitate the EFA Observatory collaborating with other UNESCO Institutes, UNESCO Sectors and other agencies who gather different pieces of this complex web of information.

    Although it is important to deliver the data in a timely (probably continually updated) way to users, this is not sufficient. Regular reports will be needed which include an interpretation of the data and link it to the interim milestones referred to earlier. These reports must speak in a language which is understood by international and national policy makers. Thus a key activity of the EFA Observatory will be the preparation of regular EFA monitors.

    In addition to the dissemination referred to above and the regular policy monitors the EFA Observatory will prepare, often in collaboration with policy experts in countries, regional organisations and international agencies in-depth analyses of particular topics. These may be themed reports for example focussing on children out of school or on the parental contribution to school finances. These may draw heavily on ad hoc sources of data in addition to the regular reporting systems.

    Regionalisation

    It is recognised that regional expertise can be invaluable in building appropriate statistical systems and that regional networks can be effective and relevant mechanisms for supporting national initiatives. The EFA experience to date has been mixed in this regard - some of the regional initiatives in data collection and support were extremely effective whereas others were less so. The EFA Observatory will build on the experience of EFA assessment to date. Regional networks will be utilised wherever possible and regional expertise harnessed, in order to adhere to the principles of both relevance and subsidiarity.

    One issue for discussion is whether the EFA Observatory should concentrate efforts on the international database or should develop regionally-specific databases each containing harmonised core international data.