The Next Cycle of FOIA Policy?  A Model for the Digital Future.

A DRAFT article published in:
 Access Reports 20(13):3-10 (22 June 1994.)
Jeremy R. T. Lewis
PhD, MA, Johns Hopkins University
MA, BA, Oxford University

Now: Associate Professor of Political Science
Huntingdon College, 1500 East Fairview Ave,
Montgomery AL 36106

Subject: Freedom of Information Act (FOIA).


During the Bush administration, proposals in Congress for an electronic FOIA languished for fear of a reaction in the executive. However, there are new signs of interest -- in Congress and in the Clinton administration -- in a policy cycle leading to an electronic version of the FOIA. Rep. Bob Wise (D-WVA) proposed such a change during his chairmanship of the House Government Information Subcommittee in the late 1980s, and Senator Patrick Leahy (D-VT) of the Subcommittee on Technology and the Law recently sponsored an "Electronic FOI Improvement Act" (S.1782, a successor to S. 1940) that deals with the issue of curbing delays in processing. In responding to the Office of Management and Budget concerning this proposal, agencies have been consistent with the long-term pattern of opposition to greater administrative burdens, although there was modest support for the proposed partial retention of the fees collected from requesters. Requiring them to supply records in a reasonably convenient format according to requesters' wishes would, they argued, impose delays and reduce the economies of scale found in delivering frequently requested material in a standard format (responses are summarized in Access Reports, 20:10, 11 May 1994.)

The first sign of a new FOIA policy cycle in the Clinton Administration has been a changed direction at OMB and the Justice Department. In 1993, Sally Katzen of OMB's Office of Information and Regulatory Affairs made policy changes to Circular A-130 that encourage release of electronic information by public agencies rather than by having private sector contractors deal with official data as the Reagan administration had preferred. The Justice Department, traditionally not a leader in the direction of open government, has also revised its guidance towards a more open policy. (On the other hand, the Clinton administration also announced that henceforward it would not consider the National Security Council staff -- highly visible in FOIA lawsuits based on the Iran-contra affair e-mail records -- as an agency subject to the FOIA.)

This article briefly examines the previous cycles of Freedom of Information policy and then considers what will likely emerge from the current cycle. We must recognize that there are often severe lags in the public sector in the adoption of technology, partly because of limited resources but often largely because of the onerous procedures for planning, bidding and approval. A case in point is the FDA's acquisition of a minicomputer system for its burdened FOIA staff only after thirteen contracts and seven years. With generations of computer equipment succeeding each other at three year intervals (20% compound growth of technology), obsolescence is inbuilt.


A policy cycle is conventionally expected to begin with the stage of Initiation (or Agenda-setting), followed by Investigation (congressional hearings, think tank reports), Legitimation (in public debate), and Legislation (enactment). This in turn is followed by Regulation, Implementation (putting the regulations into effect), Adjudication (by tribunals or by the courts), and -- perhaps the weakest link in the chain -- Evaluation (by think tanks, universities, consultants, and congressional staff). In the US case, initiation is not controlled tightly by political parties, but usually is performed by a multitude of interest groups who have access to Congress and the executive agencies. Investigation usually exceeds that done by other national assemblies. Legislation is unusually messy because Congress is decentralized. Adjudication normally occurs in American courts rather than in administrative tribunals, which means it is expensive and slow -- but thorough.

The US led other nations in the case of Freedom of Information (FOI). Initiation came via a 1955 alliance of the press and historians. Rep. John Moss and Sens. Edward V. Long and Thomas Hennings were the early congressional leaders. Opposition came from the Eisenhower administration and the executive agencies.

During the first cycle of FOI policy (late 1950s, following the McCarthyite and Cold War excesses of official secrecy), the House FOI committee pressured agencies through intensive hearings to open more documents to the public. In the second cycle (mid 1960s, following the credibility gap over the Vietnam War), it joined with the Senate to enact Freedom of Information (5 USC 552) as an amendment to the 1946 Administrative Procedures Act. However, the principle that "any person" could have access to public records in section (a) was tempered in a compromise, with section (b) permitting secrecy under nine broad exemptions.

In the third cycle, extensive hearings in 1972 found that implementation of the FOI Act had been anemic. In 1974 (amid the Watergate confrontation) both chambers amended the 1966 FOIA to render it effective. Procedures for responding to requests, charging of fees, administrative appeals and judicial review were tightened. The exemption for classified material was symbolically restricted, moreover, and the law enforcement records exemption was spelled out more carefully.

In the fourth cycle of policy (mid 1980s), when Congress was torn between liberals (e.g. Sen. Leahy, D-VT) and conservatives (e.g. Sen. Hatch, R-UT), FOIA amendments were hastily attached as a rider to the Anti-Drug Abuse amendments of 1986. While attempting to draw distinctions between different classes of requester purely for the purpose of charging fees, these amendments loosened the reins on the law enforcement agencies, permitting greater secrecy. An Executive Order on Pre-Disclosure Notifications of 1987 also required that under some circumstances, an enterprise would after submitting documents to an agency have opportunities to argue for their protection from release under the FOIA. At the same time, OMB's Circular A-130 urged agencies to use private contractors rather than in-house resources for the creation, maintenance and dissemination of some electronic databases. If it had been widely implemented, this would have reduced the scope for FOIA requests, and through higher fees would have reduced the information available to less wealthy firms or individuals.

Commonalities in the FOI Policy Cycles.

During the first three cycles, a few liberal Democrats reacted to excessive administration claims to secrecy and developed a broad, bicameral and bipartisan consensus. This fit the classical model of congressional policymaking, with consensus through information and compromise. (It is often known as pluralistic bargaining, since a number of interests are active.) Contrary to the usual expectations of congressional oversight, Congress has shown a long attention span with repeated rounds of investigation of the executive agencies' praxis.

In the fourth cycle, midnight clause-trading suddenly produced a set of amendments as a rider to the omnibus crime and drug abuse bill of October 1986. Given little time to develop a consensus, members of both chambers resorted to reading material into the Congressional Record to guide judicial interpretation of the amendments. In principle, slow consensus through education and compromise favors stable policy, whereas rapid compromises are more likely to be amended in future.

A public-interest group strategy of educating judges to the FOIA disclosure principle was highly successful from the mid 1970s to the mid 1980s, particularly in the District of Columbia where judges were more experienced with the Act. In part this was helped by the Carter administration's sympathetic appointments to the federal bench. Given the many Reagan administration appointees to the bench, however, the balance turned and public interest groups became wary of pursuing disclosure in D.C.. Perhaps this "information-conservative" appointment pattern can be moderated by the Clinton administration. Fortunately, the US Supreme Court (whose conservatives are more libertarian on public information) normally rules in line with the spirit of the FOIA.


One thing on the minds of all three branches of government is the extension of FOI through computerized record processing and digital publishing. Just as agencies are experimenting with electronic publishing (e.g. Census Bureau statistics on CD-ROMs; and online bulletin boards) to supplement their print publications, so the future of freedom of information lies with electronic submission, storage, requesting, and releasing of public documents. This may or may not save money, but it will lead to significant benefits. The model can be described in straightforward terms.

Tracking and Indexing.

First, draw a distinction between the longtime uses of computers and the newer possibilities created by friendlier, less expensive systems. Computer tracking of requests, as well as computer processing of stored database indexes, already exists at most agencies. Placing those indexes on a file server for requesters to browse by telephone would be a new service. Many libraries have allowed an analogous kind of access for several years via the internet. Indeed, one can search libraries across the country using menu-driven software such as "Gopher," and then request copies of articles by traditional mail. Why not arrange the same for agencies' indexes?

Facility of Database Reporting: new Wine or new Bottles?

Secondly, accepted doctrine has it that although agencies can generally choose the format of a released record, they may sometimes be required to generate a new report subject to the proviso "a reasonable amount of effort." What is possible with a reasonable amount of effort has recently expanded substantially. Conversion of record formats and generation of "new" documents, although not required under the FOIA itself, have been greatly facilitated. Now, in the age of microcomputers, it can take little more than menu and button choices for a secretary (or other professional) without much additional training. Indeed, some databases with graphical interfaces now allow query by drag-and-drop with a mouse pointer or a pen gesture, and they will soon accept voice-activated queries. Tabular data used to be charted via multiple commands, but now this may need only four or five mouse clicks. Crosstables from a large spreadsheet required a delicate sequence of commands two years ago, but now fresh crosstables can be created by dragging variables across the screen

Note that the new feature that transforms the situation is not the arrival of the computer or the database which can be manipulated by computer technicians. It is the user-friendly and inexpensive database which is manipulable by secretaries and analysts. Since efficient FOIA request processing must rely, sooner or later, on thousands of front-line staff rather than a few technicians, this may soon amount to a substantial evolution in FOIA processing.

Developing Databases with a View to Improving FOIA Processing.

Thirdly, the vast quantities of data submitted to government by contractors and by regulated companies could in future be submitted in standard database form to facilitate review. Once submitted in any electronic form, data could be rearranged into the agency's standard database "fields" of information. Ideally, each field would be coded according to its status under the FOIA for release or a claim of withholding. Then purging records would be accomplished simply by reporting only releasable fields. Alas, withholdable portions of text will normally be scattered across fields of a database, requiring a global search for proprietary material, just as in hardcopy documents. Even so, digital searching for a "confidential" flag amid boilerplate text is faster than with hardcopy.

Public Online Access to FOIA Indexes.

Fourthly, frequently requested data will be reviewed and categorized as static or as changing. For static records, it is now inexpensive to master small read-only disks (CD-ROMS), such that a large volume of records can be served repeatedly from a single drive array without risk of damage or alteration. (A disk also happens to be convenient to mail to requesters wanting the whole set rather than a single file.) Many public and university libraries already use such systems, typically with a rack of twenty drives each holding a disk with up to 600 Megabytes of data. Although the search software is not yet standardized or elegant, a single sheet of instructions is sufficient for most users to find what they need. A menu interface will enable the requester to browse through indexes and abstracts and then to download selected material without even submitting a FOIA request. Fees of course could be recouped by the use of a 900 number, with charges by the minute for downloading. Commercial requesters could be given a different password than news media requesters, so long as the law permits the distinction.

Where the data change daily, FOIA officers already report frequent requests for a standing order system whereby new information is continually released. In the short run, there are hybrid solutions for the transmittal of large documents and indeed whole databases. Pocket-sized devices weighing just ounces (CD-ROMs, flash memory PCMCIA cards, or matchbox-sized hard drives) and holding scores of megabytes of data can be mailed conventionally just as microfiche are currently. Although this may be functional for frequent, commercial requesters, printouts will still be needed for the many casual requesters.

Repetitive copying and mailing of changing records is, however, burdensome. The preferred solution would be to upload the records weekly to a hard drive array in a file server, with public access via bulletin board system and telephone line and (for mass access) fiber-optic or wireless channels. For short and popular documents -- especially for instruction sheets on using the system -- there are also fax-back boards which fit into microcomputers at minimal cost. Should demand for the material increase, the machine rather than staff would be overburdened. This method turns on its head the current agency dilemma whereby more convenient requesting (e.g. by fax) leads to a greater burden of requests. The new goal becomes convenient dissemination sans staff increases.

To allow outsiders access to the actual agency computer bank would be risky, because no password system is unbreakable. To ensure data security, the servers would best be dedicated to public access, with their files updated on a daily or weekly basis. The speed and low cost of today's microprocessors make this dedication feasible.

Commonly requested classes of unpurged documents -- e.g. the FDA's so-called "510(k)"s, which are requested by the tens of thousands -- could be purged and fed directly into an open database. Thus, material purged for one requester would become generally available, thereby rendering subsequent FOIA requests unnecessary. The merit of this system is readily apparent from the success of service companies (e.g. FOI Services Inc of Rockville, MD) which store released documents, index them on computer and "fill-from" their own collections faster than the agency can deliver -- for a substantial fee.

Input to Electronic Format.

There are of course major obstacles -- other than the perennial problems of budgetary famine and lengthy procurement processes -- to implementing electronic redaction and disclosure of records. Most obviously, these occur on the input side: there is often a large archive of hardcopy documents, or there may be an incoming stream of handwritten notes, for example from factory inspectors.

It is increasingly feasible, however, to convert hardcopy files to electronic form, depending on the clarity of the original documents. (Creating the Biblical world in six days did not require this backward compatibility.) The clearest can be scanned, then converted by Optical Character Recognition (OCR) software and stored and transmitted economically as text. But many are unsuitable for recognition, and must be scanned, stored and transmitted as graphical images. Scanning is becoming as fast as xerography: while a hand-fed scanner is too slow for long documents, commercial scanners are achieving 2 pages per second throughput and approximately 97% OCR accuracy, rates expected to increase. Despite advancements in mathematical compression techniques, however, these images are heavy users of storage in electronic form.

Another bottleneck for electronic records processing concerns remote input from the field. Inspectors' field notes are usually prepared on paper, and often in handwritten form. While officers' opinions can sometimes be protected under the internal memoranda exemption ((b)(5)) of the FOIA, the court ruled in Sterling Drug v FDA that scientists' opinions are factual and hence must be considered for release. If these can be input at source to electronic systems, FOIA processing would be greatly reduced. Two competing technologies gaining acceptance are the notebook computer with keyboard and the palmtop pen computer. It is becoming a routine matter to upload notes from these portable computers in the field via modem and telephone wire, something journalists have done for a decade.

Pen-based computing complicates data formats, but this can be resolved in two ways, graphical and textual. First, writing can be stored as mathematical "ink," whereby formulae represent the direction, pressure and speed of the stylus on the glass panel. Or, it can be read as text using OCR software, and then utilized as any other electronic text. In future, computerized transcription of voice records from field telephone calls may obviate the need for this intermediate technology.

Onscreen redaction (excision or purging.)

Redaction, the most characteristic feature of FOIA processing, perhaps has the most to gain in productivity. Where exempt material is interlaced with releasable material, redaction (or purging) would be done with a mouse cursor painting out portions of onscreen text. A click on a button (or function key) could interpolate a notice of the rationale for the excision. Another click or button press might add boilerplate notices of appellate rights, etc. A new version of the document would then be saved for printout or online release to the requester. (Indeed, such a system from Severn Inc is being used with some success at the State Department.) Given material in electronic form, painting-out should prove substantially faster than the hardcopy method of excising passages with scalpel or chinagraph pencil. Swapping photocopier for laser printer should permit an officer to remain at the desk and thereby increase processing speed. As at present, the removal of exempt material would be indicated in the released document, with marginal notes as to the legal justification for the redaction.

Too much should not be expected of the productivity increase gained via onscreen redaction. Wherever possible, "wholesale" rather than "retail" processing is to be preferred. Where classes of documents can be identified for release or withholding without editing, the time saved will probably far exceed savings made from onscreen redaction. In the case of submission of material to an agency by an outside party, it may prove possible for the source to annotate documents with limited proprietary claims before submission, to the guidelines set by the agency. Training regular submitters of documents to be more selective in claims of confidentiality (and verifying this by sampling) might be more cost-effective than conducting in-house line-by-line review of their documents.

Digital Dissemination: Beyond Text.

Once documents are transmitted in digital form, it will be easy to set up a text-only bulletin board system into which requesters can connect, search for and retrieve documents. But multimedia imaging is still an emergent technology, reliant upon technical advances in compression and decompression. Sending compressed images along wires is not only a slower process than sending text, but the recipient needs compatible software with which to decompress and play back the images. Viewer software for richly formatted text and for video already appears capable of playing back these signals on other machines as they appeared to their creator, and digital signal processors currently being incorporated in microcomputers appear to be the medium term solution for the compression problem.


Where would electronic FOIA processing find practical application? First, in agencies that rely primarily on electronic storage (e.g. the Department of Defense). Secondly, where an agency faces political pressures to improve its FOIA responses (e.g. the State Department.) Thirdly, where most requests come from corporations and lawyers with the resources to take advantage of electronic access (e.g. at the Food and Drug Administration.) Fourthly, where previously purged and released material ("fill-froms" in FOIA parlance) forms a substantial open database. The higher the proportion of "fill-froms," the more likely the fulfillment of the model.

A few systems already point to this electronic future. Moving beyond the nearly ubiquitous FOIA request tracking and accounting systems, the State Department has onscreen redaction, the FDA has experimented with scanning into graphical files, and the Environmental Protection Agency (Toxic Resource Inventory) and Federal Maritime Commission have open online databases. The FMC even has its database open to uploading of commercial import data from external sources.

Program innovation commonly occurs during budgetary feast and fiscal famine, or political crises. There being little prospect of substantial budgetary increases in the 1990s, one must look to famines or political crises for innovation. Freedom of Information programs have never been accorded budgetary priorities, and frequently are not accorded separate budget lines.

Yet FOIA computerization, however expensive the initial costs of hardware, software and training, will eventually be seen as an investment since it can reduce the costs of document handling and warehousing. Text entry will not necessarily be faster, but document transmission, manipulation, storage and retrieval will often show substantial savings. Data will also become amenable to statistical and graphical analyses that previously could not be explored because of the effort involved. Further, scheduled document destruction will become less expensive. Productivity gains may well be smallest where the submitters and requesters are individuals unused to computers. In many cases, however, subsets of records are requested by commercial organizations well versed in digital technology. Where commercially valuable material requires line-by-line review of segregable portions, digital technology is likely to show productivity gains. A computer bulletin board system that holds regularly updated records will permit records to be downloaded repeatedly by requesters without necessitating further staff time. The outlay of a printed document published by the US GPO could be avoided.

Given continuing reductions in the cost of microcomputer hardware, a wave of activity towards converting agency records to machine-readable form is likely by the end of the century. Application to Freedom of Information Act processing can reasonably be expected to follow. Dissemination of redacted records via bulletin boards should eventually dwarf the present FOIA processing. We may be entering the era of the "Virtual FOIA request."

Dr. Lewis is a frequent author, lecturer and a training consultant on the FOIA.