Contents - Previous - Next
7. Evaluation and change
7.2 Evaluation: specific examples
7.1 Evaluating effectiveness
Evaluating the effectiveness of a library: a
theoretical and methodological framework
On evaluating the effectiveness of school libraries
Concepts of library goodness
Evaluating the effectiveness of a library: a theoretical and methodological framework
Collège de Trois-Rivières Library
The author proposes a theoretical and methodological. framework for studies to evaluate the effectiveness of libraries. Systems analysis, the principles of which are explained, is the most frequently adopted approach in studies of this kind in the information sciences. There follows a definition of microevaluation, which employs indicators of effectiveness as the basis for a rigorous self-evaluation by a library of its results and the quality of its products and services. The author concludes with a critical analysis of the philosophies of the 'library-college' and 'media centre', which do not identify the teaching library with a documentation system.
One of the current priorities of organization management is to evaluate and control the quality of the products and services supplied to customers. In the education sector particularly, evaluating the various components of the education system is a major concern. Librarians and other documentation specialists have not therefore been bypassed by the universal phenomenon of evaluation. Over the past 15 years or so a considerable body of theoretical studies and empirical research work on the evaluation of libraries and documentation centres (information retrieval systems) has been produced, making this one of the most highly developed sub-branches of the information sciences.
De Prospo notes that libraries are the most advanced of all public-sector institutions in the evaluation and measurement field and that all such institutions would do well to follow the models developed by documentation-science specialists with regard to controlling the quality of their services. The recent work by Lancaster, which examines a fair number of evaluation projects undertaken in libraries up to 1974, includes a substantial biography testifying to the wide range of studies in the library-science field on the measurement of effectiveness.
An evaluation project presupposes a conception of the nature of the library in question and of the procedure to be employed. Since any conception of reality implies a particular framework of interpretation, an evaluation study must necessarily be based on a more or less systematic scheme of thought. Before evaluating a documentation service the evaluators should adopt and make explicit a frame of reference. The purpose of this article is to propose such a framework by putting forward a logical set of basic concepts and methodological principles that could serve as the theoretical foundation for future evaluations of the effectiveness of documentation centres in Quebec. We believe that the development of a meaningful theoretical and methodological framework is a vital prerequisite for the application of particular evaluation models.
Before undertaking an evaluation project librarians must be convinced of the need for such an enterprise and of its importance. It is therefore logical to begin by asking why documentation-science specialists should evaluate libraries.
The first reason is that, since quality control Is one of the four major functions (planning, organization, administration and control) of sound management, librarians, who nearly all occupy posts to some degree administrative in nature, are professionally responsible for evaluating the quality of the services provided to the public.
To check whether the library is achieving its objectives or, in other words, to make sure that it adequately meets the needs of its users, the librarian must assemble relevant and objective information on the quality of the services provided. The purpose of an evaluation project is to provide the administrator with the systematic information he requires to assess the quality of services objectively and take a rational decision.
When the library staff has adequate knowledge of a situation they are then in a position to decide on judicious corrective measures to improve the quality of services. The ultimate aim of any evaluation project is not., then, to monitor with the purpose of rewarding or punishing but rather to improve the performance of a documentation service. The essential aim of a library-evaluation project should therefore be to evaluate in order to evolve.
Viewed from this standpoint, an evaluation study becomes a management tool enabling the staff of a particular library to determine how far it is meeting user requirements and to identify the shortcomings and gaps in its services with a view to making the necessary improvements.
Another reason that should prompt librarians to evaluate documentation services systematically is the restricted nature of financial resources. Given the ever-rising costs of public services and the scarcity of financial resources, all public institutions - including teaching establishments and libraries - have become accountable to taxpayers and their representatives. Librarians must justify financial allocations to documentation services in terms of target results and must show that the products and services financed by these allocations match up to the expected results (objectives set). Libraries in Quebec and elsewhere will increasingly live in such a world. Library authorities will have to prove the effectiveness of library activities and will have to demonstrate, by means of meticulous studies, the quality of documentation services provided to the public.
Types of evaluation
There are three distinct types of evaluation a documentation service: evaluation of its effectiveness, evaluation of its cost-effectiveness ratio, and evaluation of its value.
Evaluation of effectiveness involves determining the extent to which the objectives pursued by a library are achieved. The results of a documentation system are analysed to see how far they match the objectives set (expected results). An evaluation of effectiveness is designed to monitor the quality of a library's results and to check how far its products and services effectively satisfy user needs.
A cost-effectiveness evaluation seeks to calculate what it costs a library in terms of resources to achieve its objectives. A documentation service is 'efficient' when it achieves its objectives at the lowest possible cost. A valid indicator in this area is the unit cost of each volume loaned or of each relevant biographical reference supplied.
Measuring value or cost benefit involves determining the consequences of a library's effectiveness and defining its impact on user behaviour. The value of a documentation centre is equal to its contribution to the achievement of the general objectives of the organization to which it belongs. For example, the value of a school or university library corresponds to its contribution to the achievement of the general educational goals of the teaching establishment it serves. The value of a multimedia documentation centre should be worked out objectively by measuring its real impact on the outcome of the learning process.
The value of a library should be determined not solely on the basis of economic factors but also with reference to intangible benefits, in the educational and cultural spheres for example. It goes without saying that assessing the value of a library is an extremely complex operation. At the present time there is no model enabling us to determine objectively the value or benefits of a documentation centre.
A conceptual framework for the evaluation of effectiveness
We shall confine ourselves to discussing evaluation of a library's effectiveness, since we believe that the first need is for the profession to prove the quality of the documentation services provided to the public. As stated above, evaluating effectiveness involves determining the extent to which the objectives pursued by a library are achieved. Since, under a valid planning system, the objectives should reflect the needs of the environment (users) in terms of meaningful results to be achieved, it follows that evaluating the effectiveness of a library involves checking how far it satisfies the requirements of its users.
To be valid, evaluation of a library's effectiveness should be user-oriented; it should therefore consider the library in relation to its environment. There is at the present time a consensus among evaluation specialists to the effect that a genuine evaluation project should be focused on the user and should employ user requirements as the criteria for assessing the quality of the services provided.
Macroevaluation and microevaluation
Lancaster establishes an important distinction between two types of evaluation of the effectiveness of a library: macroevaluation and microevaluation. Macroevaluation seeks to determine how far the objectives of a documentation service are achieved. It simply measures the performance level, without specifying why such results have been achieved and without indicating what could be done to improve performance in the future.
In contrast, microevaluation is a much more detailed study of the effectiveness of a documentation centre. It analyses the performance level and identifies the factors affecting it. Microevaluation is, in fact, a diagnostic tool that relies on analytical models to distinguish high performance from mediocre performance, to identify the main factors that affect performance for good or ill and to pinpoint the gaps in the services provided and the causes of them.
By pinpointing the major factors that diminish the effectiveness of a given library a microevaluation provides the librarian with the objective information required to remedy current shortcomings and improve future results. As the ultimate goal of the evaluation project is to improve the quality of services it is clear that a microevaluation is, in the end, the only true evaluation of a library's effectiveness.
Measures of the quality of services
In order to arrive at a valid diagnosis of the results achieved by a particular library, a microevaluation should undertake an objective analysis of its performance. A study of this kind seeks to measure the results of a documentation service so as to determine how far the objectives have been achieved. In order to check whether the quality of services matches the objectives, there is a prior requirement: the library's objectives must be operational and measurable.
Unfortunately, few of those in charge of documentation systems currently frame service objectives in operational terms, which makes it impossible to evaluate performance in terms of how far the chosen objectives have been realized. Once they have worked out the goals (philosophy) of their service, librarians should therefore translate these goals into measurable operational objectives. The effectiveness of the library, i.e. the degree to which it achieves its objectives, can subsequently be demonstrated through the use of measures of quality.
Measuring the results of a library implies the use of quantitative methods. By the use of statistical techniques it is possible to analyse objectively the quality of a library's services. Unless it is backed up by statistics, an evaluation study amounts to no more than a statement of opinion and is devoid of any scientific value. A methodical analysis of the results of a documentation service calls for valid models and instruments of measurement. The use of analytic models in library science will make the scientific management of libraries a possibility.
Systems analysis is currently one of the preferred methods used by administrators to measure the performance of their organizations. In the information sciences likewise, most models for evaluating the effectiveness of libraries or documentation centres are based on systems-analysis techniques. Before examining the systems approach applied to librarianship we shall attempt to define the nature of systems analysis.
A system is a network of organically interrelated parts (objects, men, processes) geared to the achievement of a common ultimate goal through the intermediary of specialized functions. These specialized functions are fulfilled by functional services (subsystems) of the organization. Each functional service or subsystem therefore has a specific mission to accomplish, a particular function to perform.
The purpose of systems analysis is to investigate systematically the meaningful interactions between the various components of a system. It is based on the principle that altering one of the components in an organized system influences the effectiveness of all the other components and consequently has an impact on the overall performance of the organization. The systems approach measures the overall quality of the system and analyses the contribution of each of the components to the effectiveness of the organization as a whole.
Systems analysis uses mathematical models incorporating the main elements (variables) of an organization to measure their combined effect on the overall effectiveness of the system. To undertake an analysis of this kind, the main variables of the subsystem or system concerned must be properly quantified. Systems analysis always yields objective data - reproducible by other researchers - and makes it possible to predict the effectiveness of a given system.
Indicators of a service's quality
With systems analysis, the quality of a service or a system is not determined by the impressions of the person in charge as to the results achieved, nor by the subjective opinions of users, but by indicators of quality. An indicator of quality is a quantifiable datum obtained through observation and systematic analysis of a system's results, products and services (output). This quantifiable datum indicates how far the results achieved match up to the objectives (expected results) and thereby satisfy the needs of the environment.
An indicator of quality is a sign that shows the quality of a system's products and services (results); it also represents an indicator of effectiveness, since it shows how far an operational objective is being achieved.
Indicators of quality are indicators of results (output) rather than resources (input). Indeed, measurement of resources tells us nothing about the effectiveness of a system. The material, financial and human resources (input) of a system express no more than its capacity to produce results and not the results, products and services (output) actually achieved. To evaluate the effectiveness of a service, the systems approach uses indicators to measure the results achieved and to check how far they conform to the objectives set (expected results). It will be seen that operational objectives and indicators of results are essential elements in systems analysis.
As Paul-Emile Gingras rightly points out, indicators are used nowadays in most activities involving planning, analysis and evaluation. Today, the credibility of an evaluator is no longer based on subjective opinions or perceptions but on objective evidence derived from indicators. Administrators who employ techniques of management by objectives often use quality indicators to compare the results achieved with the objectives set. Similarly, a department head who bases his planning on systems analysis will frequently use the technique of management by objectives to obtain adequate financial resources by justifying them in terms of operational objectives to be achieved and products and services (results) to be provided.
Systems analysis in libraries
The principles of systems analysis briefly outlined above are obviously applicable to a library, which clearly constitutes an organized system. To evaluate a library's effectiveness, quality indicators must be used to interpret in rigorous and systematic fashion its results, products and services (output) in order to establish whether its performance corresponds to the predetermined operational objectives. Indicators of a library's results reveal the quality of services provided and show to what extent those services are meeting users' varied needs. As models for evaluating the effectiveness of documentation services are almost all based on relatively complex techniques of systems analysis, we propose to describe the theoretical principles underlying this approach as adopted in most library-evaluation studies.
Orr has expounded more clearly than anyone the implications of systems analysis as applied to libraries. The systems approach in library science is essentially user-oriented, since it sets out to measure the effectiveness of a library by means of criteria based on user requirements. For the user, the library is a black box. What goes on inside the box - the operations that take place there and the methods by which they are carried out - are of no interest to him. He is concerned solely with the results, products and services (output) he gets from the library.
It is for this reason that systems analysis measures only the results of library operations. Irrespective of whether the operations are performed by humans or by machines, the effectiveness of a library needs to be evaluated in terms of the results achieved by means of a rigorous and systematic analysis of the products and services provided to users.
Are we to conclude that systems analysis is not concerned with resources and with technical service operations (input)? By no means. Since the documentation chain consists of a logical succession of activities, technical service operations (input) have an impact on the quality of library products and services (output). In a complex documentation system, such as a library, there is a causal relationship between technical service policies and operations and the results achieved, between input and output.
Direct evaluation of technical service policies and operations is not meaningful since it is not focused on user needs and does not reveal the extent to which they satisfy those needs. It is only through a microevaluation of a library's products and services that the impact of technical-service policies and operations on the quality of its results can be measured. Poor performance by a documentation centre is due largely to shortcomings in technical-service policies or operations. A diagnostic evaluation of the indifferent results achieved by a particular library is the only way of identifying the causes of its weaknesses and of detecting the flaws in its technical services.
The indicators of quality used in the information sciences to measure the results achieved by a library and to assess the quality of its services are user-oriented indicators. They are a means of assessing the quality of documentation services in quantitative terms. The systems approach applied to librarianship is firmly scientific, since it seeks to quantify library products and services, i.e. to express their quality in terms of quantities by means of mathematical units.
The systems approach to the evaluation of a library's effectiveness consists in using scientifically valid models to obtain an objective and rigorous measure of its results (output) and of the major variables significantly affecting its performance. An evaluation project of this kind calls for a valid conceptual framework and a rigorous methodology. Although the methodology - which determines how measurement will be made - is important, the conceptual framework remains the key element, since it specifies what will be measured and why.
The conceptual framework defines the variables relevant to the evaluation project and specifies the relationship between them. As well as identifying the problem area, the conceptual framework specifies the objectives of the evaluation and the approach to be adopted, in the present case systems analysis.
Orr recommends that the method of measurement should be capable of yielding valid and objective data. The method chosen should be cheap and easy to apply so as not to interfere seriously with the library's regular operations. It should also be applicable in all types of libraries. Finally, the method selected should be applied by the staff of the library being evaluated rather than by outside 'experts' and should be capable of measuring the effects of changes in policies and operations on the quality of services.
However, the essential characteristic of an adequate methodology is that it should be rigorous. Those responsible for an evaluation project should take particular care to select samples that are representatives of the library's clientele.
A rigorous methodology will ensure that the results of the study are valid. An evaluation project that strictly measures up to all the requirements of validity and soundness will be able to stand up to critical examination based on the criteria of scientific research and will thereby gain the approval of users, administrators and other members of the profession.
On the other hand, an evaluation project that embodies many methodological shortcomings will cast doubt on the validity of the results obtained. The authors of an evaluation study should supply methodological details, which are vital to those who wish to analyse critically the procedure employed.
Fortunately, librarians and documentalists have become aware of the need for methodological rigour in studies on the effectiveness of libraries and documentation centres. Nowadays, the authors of empirical research on the evaluation of the effectiveness of information systems are careful to control the main variables so as to measure their real effects on the quality of the service concerned.
Swanson has defined as follows the essential components of a valid methodology as applied to an evaluation of the effectiveness of an information system:
This first stage consists in identifying the elements in the library that are to be measured.
This involves defining in what way the elements to be considered will be measured, identifying the standards against which the results (indicators) of the measurement will be compared and clarifying the presuppositions underlying the procedure used.
These variables are the technical-service policies and operations (input) together with elements of the service provided to the public (output) and the interaction between the user and the information system.
The variables incorporated in the analytic model need to be properly conceptualized and quantified.
This last stage entails analysing and interpreting the results of the study, determining the reasons for successes and failures in specific sectors and proposing, in the light of the results of the evaluation, changes to the system to improve its future quality.
To avoid any confusion among readers, those responsible for an evaluation project should be careful to define in terms comprehensible to other specialists in the information sciences the concepts employed in the study. Saracevic points out that there is a certain ambiguity at the present time in the use of concepts relating to measurements and criteria of quality. He draws a distinction between a measurement (e.g. time), a unit of measurement (e.g. an hour), an instrument of measurement (e.g. a watch) and an indicator of effectiveness (e.g. the time taken to carry out an operation: six hours). A criterion of effectiveness corresponds to a standard (e.g. a maximum time of five hours). An evaluation model is a set of logical, formalized and explicit relationships between the variables affecting the performance of the information system under consideration.
The systems approach in library management sets out to express the quality of a library's services by means of quantitative methods. To accomplish this very complex task, care must be taken to establish clearly the relationship between the quantitative data collected and the quality of the library services. As Daniel notes, it is easy to reach agreement on the result of a measurement, but the significance to be ascribed to that measurement in terms of quality of service may be the subject of heated discussion. A quantifiable datum has no significance by itself, yields no direct information regarding the quality of information products and services. It must, then, be interpreted by means of explicit criteria. Before carrying out measurements of quality, it is necessary to clarify the criteria that render those measurements meaningful and to specify which quantitative data correspond to a higher quality of service.
Critique of other types of evaluation
A microevaluation is an objective analysis of a particular library's results, products and services to see how far they match up to the objectives pursued (expected results). It measures the results achieved by a documentation system and identifies the factors affecting those results; it is a diagnostic tool that employs the systems approach to evaluate quantitatively the quality of products and services through measurements of effectiveness.
As a diagnostic analysis, microevaluation is the most valid type of evaluation for identifying a library's shortcomings and improving its future performance. There exist other types of evaluation, which we shall examine briefly.
Evaluation by traditional standards and statistics
Library standards are essentially quantitative rules defining the minimal resources in terms of documents, staff and material facilities necessary for the functioning of a library. The main disadvantage of using standards to evaluate the effectiveness of an information service lies in the fact that standards are not user-oriented but are focused mainly on resources (input) rather than on results, products and services (output).
Moreover, the general propositions that currently serve as standards for libraries are not genuine standards since, contrary to the standards applicable in engineering for example, they are not the outcome of scientific research and do not measure factors directly related to the objectives (expected results) of documentation centres. As pointed out by Lancaster, current standards are too vague and imprecise to serve as the basis for a meaningful evaluation of a library's services.
The traditional statistics compiled by libraries are likewise inadequate as a way of evaluating the effectiveness of a documentation service because, just like the standards, they focus almost exclusively on resources (input) rather than on results (output). Library statistics deal mainly with such matters as the budget, the number of documents in a collection, the number of new accessions annually and staff numbers. They consequently measure resources, which do no more than express the potential capacity of the library to provide a certain quality of service, but which do not evaluate directly the quality of service currently provided to users.
Subjective evaluation by means of a questionnaire
A subjective evaluation of the quality of a library's services involves asking the users by means of a questionnaire or interview how far they are satisfied with the products and services provided. This purely subjective evaluation procedure has limited validity since it merely reveals how users perceive the quality of the library's services. As pointed out by Kantor, an opinion survey analyses customers' perceptions of the service provided rather than analysing objectively the service itself.
An opinion survey is not very useful in diagnosing shortcomings in the products and services provided by a particular library. Users are not always capable of identifying gaps in the services provided, and still less of identifying their causes (e.g. indexing errors). A subjective procedure such as a questionnaire can be valid in identifying the attitudes or perceptions of library users but remains much too superficial to serve as a basis for a valid diagnosis of the effectiveness of a documentary centre or to lead to significant improvements.
A user survey or non-user survey of a library should not be confused with an evaluation study. Such surveys, very popular at the present time, employ a questionnaire to establish a profile of users or non-users and to discover the reasons why they use or do not use a documentation centre. The purpose of this kind of research, conducted mainly in the education sector, is to identify the perceptions and attitudes of teachers and students with regard to the library.
However, a temptation lies in wait for the authors of opinion surveys that of identifying such surveys with evaluation studies. In asking teachers who teach a specific subject and use a particular learning method the reasons why they do not urge their students to use the library, it may be tempting to make a judgement about the 'value' of the documentation centre and to conclude that it has no role to play in achieving the learning objectives in a given subject or in the context of a given teaching method.
To prevent the drawing of such inferences it is necessary to specify from the start that teachers who do not encourage use of the library are expressing a merely subjective view when they say that the consultation of library documents by their students would have no impact on the learning process. Teachers' replies are thus no more than the reflection of their opinions and should not be identified with objective truth. The value of a teaching library should be determined objectively rather than subjectively, by measuring its real impact (not its perceived or imagined impact) on the achievement of learning objectives.
Evaluation by means of a check-list
The check-list is a procedure for evaluating a library's effectiveness used mainly in school libraries, although Orr has devised a detailed checklist of the services provided to users of a medical library enabling the' policies adopted in each of its services to be evaluated. The check-list most frequently used to evaluate a school library's programmes is that devised by Mary Gaver. This evaluation tool makes it possible to measure the quantity of services provided to the users of a school library.
The check-list as a method of evaluation has been strongly criticized by Daniel, who argues that the check-list lacks rigour and tends to be a trivial evaluation procedure since it focuses exclusively on the variety and quantity of services provided to users without concerning itself with their scope or quantity.
The popularity of check-lists is explained by the fact that they are easy to draw up and use. The analytic models used in microevaluation are more difficult to construct and make operational, but they permit a much more thorough evaluation of the effectiveness of a library. Whereas the check-list simply reveals the quantity of services provided to users, the microevaluation identifies the quality of services through statistical techniques and quantitative methods of measurement.
Our colleagues responsible for audiovisual media in the colleges of general and professional education (cégeps) recently developed An instrument for self-evaluation of their department.' This instrument is in fact a checklist of the activities related to the audiovisual sector's major functions. It enables the head of an audiovisual department to compare the policies relating to its various tasks with the policies of an ideal sector defined by the authors of the evaluation instrument.
Some librarians might contemplate drawing up a similar check-list to evaluate their library. Unfortunately, this type of instrument does not seem to us a very adequate way of assessing the effectiveness of a documentation service. Contrary to the check-lists devised in the school library context, which are focused on user needs and take into account the policies governing the services provided to customers (output), this type of instrument is almost exclusively concerned with evaluating policies for the activities and operations (input) of the service under consideration.
Moreover an ideal service is a concept devoid of meaning. As Orr has shown, an ideal service is not a meaningful frame of reference. A documentation service that did not have to cope with any constraints of finance, staffing or space could easily achieve perfection; but such an ideal situation has never existed and probably never will. This is why a much more meaningful frame of reference for evaluating a library is the criterion of 'optimal service'. A service is 'optimal' when it achieves a maximum level of effectiveness, having regard to all the constraints (financial and other) with which it has to contend.
Definition of a library
Any evaluation project presupposes a view of the nature and objectives of a library. This author sees the library as a documentation system that gathers, organizes, retrieves and disseminates recorded information with the aim of communicating knowledge to its users. Such a system pursues objectives in the documentation field proper corresponding to its customers' priority information needs.
Any library has two distinct aspects: a passive or reactive aspect centred on satisfying requests for information; and an active or dynamic aspect centred on promoting and extending the use of information resources.
At the present time, librarians place more stress on the dynamic than on the passive aspect of their service. A number of theoreticians belonging to the school, college or university library field have developed new philosophies (the 'library-college' and 'teaching resource centre' philosophies) essentially centred on a dynamic interaction between the library and the world of education.
Although these new conceptions initially played a useful role by drawing attention to the important and indispensable contribution - previously underestimated or disregarded - of the teaching library to the achievement of educational aims, the time has come for a critical analysis of the-basic theses being advanced. These philosophies, which from the very start were not devoid of ambiguity, have been transformed by exegetes indifferent to fine distinctions into theories that are not merely debatable but, in their most extreme form, frankly erroneous.
The two philosophies currently in fashion assert as a basic principle that the library is the centre of a school, college or university. This principle seems to us unacceptable. It is not the library (one of a number of functional services) that is the centre of a teaching institution but rather the main agents of education, i.e. the teachers and students. We agree entirely with the great American librarian Robert S. Taylor, who says that such a principle is simply a metaphoric platitude. Together with the student-learner, the most important agent in a teaching establishment is the teacher. Taylor's comments on the library-college approach are also worth quoting:
One recent and growing idea, the library college, though bold and challenging, is not the answer at this time(...). It represents a basically naive and early - too early - attempt to solve a very large and complicated problem. It is a rhetorical rather than empirical approach.
One of the contemporary theoreticians of the library-college movement, E. Youngblood, argues strongly in an interview published in his movement's official review that the application of the library-college approach implies the abolition of existing distinctions between the roles of the librarian, the teacher, the audiovisual specialist and the administrator. I hope this extreme stance will temper the enthusiasm of certain Quebecois documentation - science specialists who maintain that the library-college theory does not imply any change in the role of the librarian and does not therefore jeopardize in the immediate future the documentation function of librarians working in teaching establishments.
A teaching library obviously represents a teaching resource centre, but what is unacceptable in the 'media centre' philosophy is the interpretation placed upon it by the many theoreticians who incline towards a totalitarian view of education. They regard educational tasks as the only truly professional tasks carried out in a teaching establishment and therefore seek to replace the documentation function by more 'prestigious' functions.
To take one example from hundreds of similar ones, Leslie Gottardi states categorically that the school librarian is not a reference librarian but a master teacher and that the effectiveness of a library's programmes - the library being designated a 'learning centre' - should be evaluated on the basis of the teaching effectiveness of such centres.
The presuppositions at the root of the philosophies of the 'library-college' and the 'teaching resource centre', seen as an 'educational centre' rather than as a documentation service, should logically lead to the statement that the effectiveness of a teaching library should be evaluated on the basis of educational criteria. If that statement is true it means that all the evaluation models constructed by documentation-science specialists are invalid, that Lancaster's volume on the evaluation of libraries should be passed over because the author refers only to documentation criteria and that, similarly, this article should be ignored since it analyses models for evaluating the effectiveness of a library based on measures of performance in the documentation rather than the educational sphere. Clearly, these educational conceptions of a teaching library have far-reaching implications when one examines the logic of their postulates.
There is an urgent need in our view to debunk the more extreme theses of certain theoreticians who deny documentation-science specialists a place in the education sector. Such a debunking exercise has become a priority task for the profession since certain administrators and teachers in academic establishments in Quebec have espoused these theses completely and wish to transform librarians into educational experts whose function would be to 'help individual teachers, groups of teachers or departments to shape the objectives and possible content of a course, syllabus or curriculum'.
The essence of the problem, in our view, lies in demonstrating that the teaching library does not need to transform itself in order to fulfil a dynamic role in the educational environment, since, by its very nature as a documentation service, it already interacts dynamically with the educational environment. In order to demonstrate this, we will use the systems-analysis approach to identify the organic links between a library and the educational environment.
The dynamic interaction between the library and the education environment is illustrated in the following diagram.
Systems-based diagram, showing the dynamic interaction between the library and educational practices
* The symbols P.O.M.C. correspond to the main management functions: planning, organization, administration and control.
This diagram shows the way in which a documentation service is effectively integrated in the teaching environment. A school or academic library is a functional service, a subsystem concerned within recorded information within a teaching establishment. This functional service requires human, financial and material resources to produce, through a variety of processes (documentation-science methods and techniques) library products and services (results) that are subsequently used as (teaching) resources in the 'education' system and represent the library's specific and original contribution to the fulfilment of educational objectives.
We hope to have gone some way towards demolishing those philosophies that do not identify the teaching library with a system concerned with recorded information or - depending on the standpoint adopted - with a subsystem pursuing objectives in the documentation sphere. The subject-matter of library science is recorded information, and the professional and social responsibility of librarians and other documentation-science specialists is to acquire an objective and ever more adequate knowledge of the very complex phenomena of recorded information so as to optimize decision-making within their functional service (subsystem) and thereby improve the quality of documentation services provided to the public.
The philosophy of the library-college and that of the 'teaching resource centre', identified with a 'learning centre' rather than a 'system of recorded information', presupposes the reductive assimilation of library science to the education sciences. This presupposition does not correspond to reality. Library science (the science of recorded information) is essentially irreducible to the educational sciences (the science of learning processes).
These two theories rest on the fundamental postulate that the replacement of the library's objectives in the documentation sphere of educational objectives and, consequently, the replacement of its functions in the information chain by educational tasks will enable teaching establishments to become more effective. Such a hypothesis has not been validated by any empirical scientific research. We therefore feel it is wise to set aside these purely speculative theories, which have never passed the test of empirical verification, in the interests of the students, who should not have to suffer as guinea-pigs for 'theories' whose relevance has never been strictly verified. The systems approach - not to say simple common sense - leads me to conclude that documentation-science specialists will always be needed to perform the complex operations that make up the documentation chain, irrespective of whether the library serves an educational, medical or some other kind of environment.
Some go-ahead teachers in colleges of general and professional education employ avant-garde teaching methods that make maximum use of library materials in learning activities. These experiments are sometimes described as partial applications of the philosophy of the library-college. But is the library-college approach an educational method? I think not. It rather involves a philosophy of education that divests teachers of their fundamental responsibility as planners of the learning process which becomes the concern of the college community as a whole.
I am very much in favour of the educational methods employed by certain teachers (unfortunately too few and far between) who encourage their students to make the maximum possible use of the library, but I do not believe that to speak of the library-college educational method in that context can do anything but spread confusion in people's minds. What is involved here is rather an individualized form of education using either the tutorial or the self-instruction teaching method. Readers wishing to familiarize themselves with the essentials of these two methods would do well to consult the work by R.M. Gagné, one of the few learning theory specialists to have developed a theory of teaching.
A systems analysis of the interaction between a library (documentation system) and educational practices (teaching system) has enabled us to show the specific role of libraries as documentation centres in the education sector. The latter naturally have a part to play in shaping the general educational aims of teaching establishments; but forcing them to become involved in the educational field, to devote their energy at the expense of their specific documentation functions to defining the educational objectives of specific courses is to charge them with a task that far exceeds their professional responsibilities.
As rightly stated by Lancaster, the assimilation of a document by a user, once it has been supplied, is outside the library's sphere of influence: libraries exist essentially to communicate documents to users. The responsibility of librarians is to ensure that the user gains access to publications that are pertinent to his interests and comprehensible to him (i.e. written in a language he can read and at a level he can understand). A library can be evaluated only on its capacity to provide the materials sought by users at the time they are needed. What the user actually does with these materials is completely outside the library's control (and, some users might say, none of the library's business).
In a forthcoming article we shall analyse the evaluation models currently available for measuring the capacity of libraries to achieve their objectives in the documentation sphere. Most of these models are based on the systems-analysis approach and seek to evaluate a library's performance in terms of the information needs of users. Such a systems approach has been adopted by the members of the Committee of the Guide d'auto-évaluation des bibliothèques de collège set up by the Committee of Library Heads of the Fédération des cégeps. The Committee has received a generous grant from the Direction générale de l'enseignement collégial (DGEC) to carry out its research project. We are convinced that systems-analysis techniques will enable the Committee to work out one or more scientifically valid evaluation models. These models will be able to be used subsequently by the professional staff of college libraries to undertake, by means of indicators of effectiveness, microevaluations and diagnostic analyses of the quality of products and services provided to users.
A microevaluation of the products and services (output) of a college library or another kind of library corresponds to a scientific self-evaluation of its results. It is therefore akin to an institution analysis, which is one of the current priorities of the college education sector. An institution analysis seeks, employing a specifically local approach, to investigate the life of a college rigorously and systematically through scientifically valid models and instruments.
Contents - Previous - Next