Previous - Next
The value of personal information records
Problems with personal information records
Appraisal:
no easy answers
Purpose
of this study
Limitations
of this study
Notes
The value of personal information records
1. A leading Canadian archivist once said that "of all national assets archives are the most precious; they are the gift of one generation to another and the extent of our care of them marks the extent of our civilization." 1 During this century, the size of that "gift" has increased dramatically, and in direct proportion to the increased interaction of the citizen with the modern state. As a result of the growth of government, information about citizens, whether recorded by or about them, is everywhere apparent in the modern records. The history of our "civilization" cannot be told without these records containing personal information, and it therefore becomes essential for archivists to preserve the most important of them as our gift to future generations. In addition to their primary (or original) administrative use in the agency which created the records, such records have value to archives in four major ways.
2. Certain categories of records containing personal information protect the rights of citizens. Archives were first collected millennia ago to ensure the rights of the sovereign, but now it is the people who are sovereign in democratic societies. Examples abound of the use of archival records containing personal information to support such people's legal and fundamental rights: land claims of indigenous peoples, compensation for victims of wartime or other government excesses, exposing illegal or unethical intrusions of the powerful modern state into citizens' lives (secret brainwashing experiments, exposure of unknowing soldiers or citizens to nuclear or chemical health risks, unacceptable police or spy intelligence methods, and so on). Records containing personal information also uncover tyranny or illegal activities of leaders; both Kurt Waldheim and Ferdinand Marcos' activities during the Second World War were unearthed in their personnel files. By providing a valuable source by which governments can be held accountable for their actions and their processes, such records are essential for the democratic process. As an obvious corollary, there are also administrative uses of such records, long after the primary, original use for which the records were first created has ceased to exist. Records created for one reason may be needed later by the government itself for quite another, usually unforeseen reason. To cite but one example, the same records created in Canada to control the forced relocation of citizens of Japanese ancestry during the Second World War are being use} forty years later to pay compensation to the people involved.2
3. Records containing personal information are of course the central underpinning of-the new social history based on the insights of the Annales school.3 So far this work by historians, and increasingly scholars in other disciplines, has concentrated on past societies and used older (and far less voluminous) records containing personal information. Yet the scholars of tomorrow will do the same kind of research using the personal information records being created today. The patterns and themes uncovered by such research not only enrich our understanding of the past, but inform us of the important dynamics and mechanisms in the society in which we live. It is essential that archivists preserve sufficient similar records to permit future generations to reclaim their heritage. Without these kinds of records, the story of governments can be told, but not that of people. The historical research potential for certain categories of records containing personal information is, in short, extraordinarily high and forms an important part of our collective memory in a democratic era.
4. Records containing personal information can also be important to the development and evolution of public policy. Case files collected over time provide the longitudinal and demographic data necessary to assess the validity of and the need for change in accepted policies, programmes, and attitudes. Treatment by the state of women, juveniles, immigrants, prisoners, indigenous peoples, the poor, and different ethnic, tribal, and religious groups, for example, has varied greatly over the past few decades. Sociological research into these variations could help improve treatment and services to such groups, reduce discrimination and bias, and indeed is needed to test basic academic hypotheses upon which such treatment and services were based in the first place.4 On a broader level, the impact of state taxation policies, economic subsidies, and research grants may often be assessed through analysis of records containing personal information.
5. Such records are of course also the lifeblood of genealogical research. As people search more and more for their "roots" in an increasingly rootless world, where a sense of personal connection to the past assumes for many a larger importance in their lives, archives will be under pressure to retain more personal information records to respond to this need.
6. In summary, records containing personal information are valuable to society in many ways. Yet, "traditionally, case files have not been retained by government archivists; policy and operational files, with a token sample of case records, have usually been deemed sufficient documentation for any agency." The research value of these records, combined with new ways of manipulating the information in them with the computer, "challenge archivists to define anew their acquisition and selection criteria."6
Problems with personal information records
7. The principal problem in defining anew such appraisal criteria is the enormous bulk of records containing personal information. There has been an explosion in the quantity of all kinds of modern records, whether personal information records or not. The Archives rationales in France has retained an equal quantity of records for the years 1945-60 as it has for those from the end of the Middle Ages to 1945: four centuries of earlier archives now equals fifteen years of modern records. In the United States and Canada, the amount of government records approximately doubles once every decade. In the United Kingdom, fifteen major departments in 1979 held 4.5 million linear feet of shelf space of individual case files, with a growth rate of 200,000 feet annually. It is estimated that the Canadian government held, in the mid 1980s, 2.5 million linear metres of active and dormant textual records, or some 20 billion pages of information. Again in Canada, all post-1945 immigration case files alone amount to triple the archival holdings of the National Archives of Canada for all government departments for over a century. Even the relative increases are large: the records of nineteenth-century governors of the state of Illinois average ten cubic feet, whereas recent governors amassed more than seventy-five times that amount.7 These totals far outpace the resources of modern archives to ever think of keeping them all. And even if they could, few researchers would appreciate the result: a paper mountain impossible to describe and control adequately, thus rendering the location of valuable information for any research purpose either very difficult or impossible.
8. With overwhelming problems of quantity go those relating to quality. The interaction of the citizen with the modern state is very complex, fragmented throughout the records of numerous agencies, perhaps falling under different archival jurisdictions (national, provincial or state, municipal, university, and so on). Indeed, the growth in the number, size, and range of government agencies themselves has in this century been as awesome as the growth in the bulk and complexity of the records which they produce. Even for the records of a single programme or agency under one jurisdiction, most individual case records are created in hundreds of local or field offices, not maintained centrally in the agency's headquarters. As a result, local variations of practice and procedure enter both the business and records-keeping operations, undermining the records' homogeneity.8 For this reason, and because in such decentralized systems records are often misplaced, statistically valid sampling is very difficult. Furthermore, modern personal information records are most often aggregated in the form of case files, and these files are filled with photocopies, duplicated circulars, and forms and printouts of information captured elsewhere, usually in computer data bases. Unlike policy files which suffer from under-documentation because important decisions are made by telephone, personal conversation, or in other ways,9 case files often have a surfeit of forms. In this context, there are unlikely to be many forms missing from a case file and the essential decisions made concerning the citizen will be recorded, but the "real" reason behind the decision -- to deny an immigrant entry into the country, for example -- may well be unrecorded.10 From these factors come two, ironical results: too much redundant information and too much missing (essential) information.
9. Another significant problem is the fundamental tension between the archival retention, and especially public use, of records containing personal information and growing concerns about the protection of personal privacy.11 It is important to remember that personal information is provided by citizens about themselves; it is their information and its use by the state is increasingly being circumscribed. More and more countries have privacy laws which severely restrict the collection, let alone use, of certain types of personal information by the state, and by implication by its archives. There are also significant concerns about and even prohibitions against the linkage of information from discrete series or data bases to create larger profiles of individual citizens. As well, there are provisions in many privacy acts prohibiting the use of personal information for any administrative purpose other than that for which it was originally collected. As if these issues do not complicate archivists' lives enough, even where the creation of certain types of personal information records is permitted, the transfer of such records to an archives is sometimes explicitly prohibited by law.
10. To a varying degree, all archives face political pressure, whether explicitly from legislators or implicitly through lobbying groups, to retain more personal information records than they would if archival concerns alone were involved. In the United States, for example, with a strong military and patriotic tradition, it is very difficult, and in some cases politically impossible, for the National Archives and Records Administration to destroy military personnel files, even if they are voluminous and repetitive. Again in the United States, a strong civil libertarian presence was mainly responsible for launching the action that led to the celebrated FBI records appraisal case. In Canada, archivists were attacked in the press and called before a royal commission established by Parliament when it was suspected (wrongly, as it turned out) that sensitive immigration records had been destroyed.12 If there is a strong genealogical interest or tradition in a country, it is often difficult to destroy records that contain significant personal information on individuals, even if, using other archival appraisal criteria, the series would have marginal value at best. Archivists in all countries and jurisdictions must naturally be sensitive to these "politics of disposal" and attempt to balance such pressures with sound archival theory and practice.13 However, they must remember that, in an age of resource restraint, each unnecessary record retained by an archives in all likelihood eliminates the acquisition and custodial care of one with genuine archival value.
11. Archivists face these problems without the traditional support of many of their user communities. While few social scientists still urge the retention of all relevant records containing any possible value (on the basis that no archivist can anticipate all future research uses), many still express regret that practical resource factors are conspiring to produce a less complete archival record. While keeping all personal information records may have been feasible and even desirable for records from earlier centuries, when there were far fewer records created and the citizen-state interaction was much simpler, society would now "regard such broadness of spirit as profligacy, if not outright idiocy. Instead, archivists -- like most residents of the real world -- must pick and choose." 14
12. Such picking and choosing is not easy. Appraisal has been termed "the greatest professional challenge to the archivist" by the author of a leading manual on the subject.15 Yet appraisal has often been done in a random, fragmented, uncoordinated, even accidental manner, producing a biased, distorted archival record.16 The reason for this is the primitive state, at least in the English-speaking world, of archival theory on appraisal. To date this has rarely advanced beyond the "taxonomic" stage, that is to say, a descriptive categorizing of various values of records (such as evidential and informational, primary and secondary, and so on). It has rarely approached the level where research into concepts of societal dynamics has led to a theoretical model for appraisal. Appraisal, therefore, generally occurs in isolation, where various of these "taxonomic" values are applied to each series one by one. (An alternative, which forms the core of this study, is a global or comprehensive approach, where series are compared in a "macro" way first to determine the most important, before each is appraised by itself.) As a result of this conceptual vacuum, a recent study of archival theory in the United States deliberately omitted appraisal from its coverage and asserted that the development of "a body of appraisal theory is perhaps the most pressing need in the archival field today." 17 If this is true for all archival records, it is especially so for records containing personal information, given the difficult problems they present. While general approaches to appraisal apply to all records, it has been rightly recognized that personal information, especially when aggregated in modern, voluminous case files, is a "separate problem" and requires special treatment.18
13. The archivist's task of appraising modern records containing personal information is made considerably easier, however, by the existence of sound records management in the agency creating the records. Where this does not exist, it should be encouraged, for the records manager is the natural ally of the archivist. Both look after the same records at different stages of the life cycle. Both have a vital interest in ensuring that records of continuing administrative value are preserved. Both have a need for reliable surveys and inventories of records, for careful centralized control of records, for effective forms design and forms management, and for the clear, logical, and consistent classification of individual records on to the correct files. And both are keenly concerned with the timely and economical disposal of records, whether through transfer to the archives or by destruction (in either case, perhaps after a period of dormant storage in a records centre). It is true that the records manager and archivist have significant differences in emphasis and orientation. The former is concerned with all records created while the latter ultimately only in perhaps 5 per cent of that total. The former is also accountable only to his or her parent organization and is driven by business and economic values while the latter must also, while not ignoring these factors, take a much longer and larger view encompassing heritage and cultural factors. Nevertheless, the archivist can obviously use many of the tools of the records manager to aid in the identification and transfer of a better archival record. 19
14. The purpose of this report is to offer guidelines for archivists for the appraisal of records containing personal information. Guidelines are to give general guidance, not definitive solutions. They are intended to set directions, to provide a theoretical framework for approaching this difficult appraisal issue, and to focus thinking on an interim basis. It is recognized that concrete solutions or fixed rules for many of the issues concerning the appraisal of records containing personal information may only be achieved at some future date. In the first instance, appraisal of any sort of record series, let alone those containing personal information, is a subjective activity, there being no absolute, "scientific" answers. Appraisal is often choosing the best of several bad alternatives. And secondly, individual national situations will colour the application of these guidelines, in light of the indigenous nature of particular kinds of records and their creators, as well as national traditions for the development and use of archives, although, ideally, sound archival theoretical frameworks and principles should transcend such factors. Nevertheless, despite these qualifications, it is intended that these guidelines will not only aid archivists in the interim to deal with a difficult problem, but also set a path towards more permanent solutions as these guidelines are tested and then revised in the light of practical experience.
15. There are various limitations to this study, in terms of what it covers and what it leaves out:
a. It covers only records created by governments or the state. While its observations and conclusions may be useful to archivists appraising records containing personal information created by businesses, universities, labour unions, churches, or private individuals, archivists responsible for the records of such institutions will not find those situations explicitly addressed here. The author's bias in argument and examples is furthermore towards national archives.
b. It deals almost exclusively with records containing personal information in paper (or textual) format. Micrographic or electronic versions of these records are mentioned only as they interrelate with paper records. Similarly, personal information recorded on maps, photographs, and film is also only mentioned briefly in passing.
c. This study focuses tightly on appraising records containing personal information. Five closely related and germane subjects -- appraisal in general, records management and the records disposition processes, machine-readable records, privacy, and sampling - are not dealt with directly. These are all the subject of other RAMP studies, and there is no intention here to repeat that work. Some conclusions from these studies, especially regarding sampling, are summarized for the reader's convenience, but the full argument and analysis in those reports is not restated.
d. Additional limitations concerning the types of records containing personal information covered by this study will become apparent in the next chapter, where definitions will be advanced which will narrow the topic to manageable scale.
1. Arthur G. Doughty, who was Canada's National (then Dominion) Archivist, writing in The Canadian Archives and its Activities (Ottawa, 1924), cited in Archives: Mirror of Canada Past/Miroir du passé du Canada ((Ottawa, 1972), forepiece. Doughty's aphorism is carved in stone on a statue in Ottawa as well as emblazoned on Canadian archivists' posters and coffee mugs.
2. For a general discussion, see Judith Roberts-Moore, "The Office of the Custodian of Enemy Property: An Overview of the Office and Its Records, 1920-1952," Archivaria 22 (Summer 1986), pp. 95-106. The role of the National Archives of Canada in making its records available for the Japanese-Canadian redress programme is explicitly dealt with in Nancy McMahon, "Coming Full Circle: Contemporary uses of the Records of the Office of the Custodian of Enemy Property," address delivered at the Annual Conference of the Association of Canadian Archivists, Victoria, B.C., 1 June 1990.
3. The best statement is Tom Nesmith, "Archives from the Bottom Up: Social History and Archival Scholarship," Archivaria 14 (Summer 1882), pp. 5-26. This thematic issue of Archivaria, of which Nesmith was guest editor, was entitled "Archives and Social History" and contains numerous articles demonstrating the imaginative use of records containing personal information to gain fresh insights into society. Two other important studies aimed at archivists are G.J. Parr, "Case Records as Sources for Social History," Archivaria 4 (Summer 1977), pp. 122-36; and Peter Gillis, "The Case File: Problems of Acquisition and Access from the Federal Perspective," Archivaria 6 (Summer 1978), pp. 32-39. Beyond these more generic studies, there is a growing number of articles on the value and use of particular types of personal case records in an archival context; see, for example, R. Joseph Anderson, "Public Welfare Case Records: A Study of Archival Practices," American Archivist 43 (Spring 1980), pp. 169-79; David J. Klaassen, "Achieving Balanced Documentation: Social Services from a Consumer Perspective," The Midwestern Archivist 11 (1986), pp. 112-24, and especially pp. 118-19; and John C. Rumm, "Working Through the Records: Using Business Records to Study Workers and the Management of Labour," Archivaria 27 (Winter 1988-89), pp. 6796. For another perspective on uses of archival records, see Michel Duchein, Obstacles to the Access. Use and Transfer of Information from Archives: A RAMP Study (Paris, 1983), pp. 8-9.
4. Danielle Laberge, "Information, Knowledge, and Rights: The Preservation of Archives as a Political and Social Issue," Archivaria 25 (Winter 1987-88), pp. 44-50. The article deals with these broad issues by using records relating to juvenile offenders as the example.
5. It is important to emphasize that most public policy research uses personal information extracted from records, and therefore is interested more in runs of data rather than in series of records. Similarly, the research methods used by sociologists, public policy makers, and Annales researchers for the evaluation of personal information are not equivalent to the research and appraisal methods of archivists, although there may be a useful cross-fertilization.
6. Parr, "Case Records," Archivaria, p. 136.
7. For France, United States, and Canada, see Carol Couture and Jean-Yves Rousseau, The Life of a Document: A Global Approach to Archives and Records Management (Montreal, 1987), p. 184 (English translation of Les archives au XXe siècle, 1982); for Britain, see "Sampling Particular Instance Papers," RAD Occasional Paper No. 8 (London: Public Record Office, 1984), p. 2; and for Illinois, see F. Gerald Ham, "Archival Choices: Managing the Historical Record in an Age of Abundance," in Nancy E. Peace, ea., Archival Choices: Managing the Historical Record in an Age of Abundance (Lexington, Mass., 1984), p. 133. For Canadian immigration records and total metres/pages, see Terry Cook, "Billions of Records: What to Keep -What to Destroy?" The Archivist 13 (March-April 1986), pp. 1-2. Other examples of these explosive trends are given in Felix Hull, The Use of Sampling Techniques in the Retention of Records: A RAMP Study With Guidelines (Paris, 1981), p. 2.
8. This was of course the central contention in the Federal Bureau of Investigation case file incident; the FBI assertion that information on individuals held in field office case records was either duplicated at headquarters or incorporated in reports filed there was shown, after long study, to be untrue. The best, short summary of this important case is found in James Gregory Bradsher, "The FBI Records Appraisal," The Midwestern Archivist 13 (1988), pp. 51-66
9. F. Gerald Ham, "The Archival Edge," in Maygene Daniels and Timothy Walch, eds., A Modern Archives Reader (Washington, 1984), pp. 330-31 (article originally published in 1975).
10. This important distinction and example was made in a letter to me by Trudy Peterson, 19 March 1990.
11. See Duchein, Obstacles to the Access. Use and Transfer of Information From Archives, for an excellent summary of this problem. A good introduction to the issues involved is D.H. Flaherty, Protecting Privacy in Surveillance Societies: The Federal Republic of Germany. Sweden, France. Canada, and the United States (Chapel Hill, NC, and London, 1989).
12. On this issue relating to the prosecution of alleged war criminals, see Terry Cook, "Nazi Cases Not A Factor. For the Record: Archivists Honorable," The Globe and Mail, Toronto, 11 August 1986, p. A7; and Robert Hayward, "'Working in Thin Air': Of Archives and the Deschênes Commission, " Archivaria 26 (Summer 1988), pp. 122-36.
13. The quoted phrase and many of the ideas in this paragraph are Trudy Peterson's, in her letter to me of 19 March 1990.
14. Ham, "Archival Choices," Archival Choices, p. 133.
15. Maynard J. Brichford, Archives and Manuscripts: Appraisal & Accessioning, Society of American Archivists Basic Manual Series (Chicago, 197?), p. 1.
16. The charge, made effectively and in my view with justification, is Gerald Ham's, in "The Archival Edge," Modern Archives Reader, p. 326. He adds (p. 328) that archivists have failed to deal with acquisition policy on any "coherent and comprehensive basis" because of "nuts and bolts" or craft tradition dominating the profession from its older custodial or curatorial emphasis.
17. Richard C. Berner, Archival Theory and Practice in the United States: A Historical Analysis (Seattle and London, 1983), pp. 67. This relatively undeveloped state of archival appraisal theory, at least in North America, is reinforced, although sometimes only implicitly, by two surveys of writing in the field: Harold T. Pinkett, "American Archival Theory: The State of the Art," American Archivist 44 (Summer 1981), pp. 217-19; and Nancy E. Peace, "Deciding What To Save: Fifty Years of Theory and Practice," in Nancy E. Peace, ea., Archival Choices: Managing the Historical Record in an Age of Abundance (Lexington, 1984), pp. 118. As will be seen in Chapter Three of this study, the European development of appraisal theory is more advanced.
18. Michael Cook, Archives Administration: A Manual for Intermediate and Smaller Organizations and for Local Government (London, 1977), p. 64.
19. The best introduction to this relationship and to records management generally is found in Ira A. Penn, Anne Morddel, Gail Pennix, and Kelvin Smith, Records Management Handbook (Brookfield, VT. and Aldershot, 1989). See as well Eric Ketelaar, Archival Appraisal and Records Management Legislation and Regulations: A RAMP Study With Guidelines (Paris, 1985); and Derek Charman, Records Surveys and Schedules: A RAMP Study With Guidelines (Paris, 1984).