Draft as submitted to Access Reports,
published in two parts as:
"New Technologies and FOIA Processing: Part I." Access Reports, 19(15):2-7 (July 21, 1993)
"New Technologies and FOIA Processing: Part II." Access Reports, 19(16):4-9 (August 4, 1993).
Jeremy R. T. Lewis
PhD, MA, Johns Hopkins University
MA, BA, Oxford University
Now: Associate Professor of Political Science
Huntingdon College, 1500 East Fairview Ave,
Montgomery AL 36106
This piece was intended to introduce FOIA officers to simple, individual workstation software setups
as models of more powerful industrial-strength solutions. It was intentionally kept light.
Let us first consider the types of FOIA office function which could be assisted by computer software. In order of arrival they are wordprocessing; database tracking; spreadsheet accounting; communications of various types; document scanning, redaction, editing and compressed storage; and interactive personal training tutorials.
But first, a caveat: all of the technologies discussed here are available inexpensively on the market already, but technology in this field doubles in capability every three years for the same price, and just lately has been accelerating rapidly through price cuts in personal computers. Consequently anything written here will sound archaic by 1996; but there again, as many FOIA officers have told me ruefully, government offices' technology is always archaic!
By now, simple wordprocessing has become almost universal, and almost all offices have shifted from dedicated machines (e.g. the Lanier systems which the EPA FOIA staff had in 1979) to networked personal computers running WordPerfect 5 or later (e.g. the INS FOIA staff of 1989) or a similar solution. Each typist switching to wordprocessing can generally be four times as productive as before, but only if it fits the exact mission or the office. That is, wordprocessing doesn't help so much if half the time is spent answering the phone or retrieving paper files. In general, surveys have shown that about two-thirds of microcomputer use is wordprocessing and so it has probably brought the most benefits to the office since the arrival of the Xerox photocopier in the 1940s.
Wordprocessing still has some enhancements to make in conjunction with other technologies: communications, graphs, diagrams, video and sound can now be integrated in wordprocessing documents, although each of these requires some equipment and does cost you some time, processing speed and hard drive storage room. See below.
Tracking Requests by Database.
The second most obvious use, as Roberts pointed out, is the tracking system to ensure a central office knows the status of requests being processed in multiple component or field offices and multiple filing systems or databases. Given the ten working day limit for determinations, this is a natural choice for the two thirds of the more than ninety agencies which are subject to significant numbers of FOIA requests. For example, the State Department which for years had a terrible reputation among FOIA requesters, but has reportedly improved its performance dramatically in 1992-3, has the INFORM tracking system. At the Food and Drug Administration, a custom database using off the shelf ORACLE software on a minicomputer not only keeps track of the more than forty thousand requests per year, but enables one officer to grind out six hundred invoices per month to their mostly commercial requesters. Printouts of the tracking are valuable enough to be themselves requested by a service company on a regular basis, probably for business submitters to check whether their material has been requested by competitors.
If your present tracking system is obsolescent, or if you have no computerized one available, what should you try out? If you need to tie together mainframes, minicomputers and micros, you may well want to consider ORACLE because it runs on almost anything. However, it is an instance of the conventional, programmable database, and requires professional help to program it to deliver specific reports such as the monthly billing.
Throughout this article, we shall be looking at IBM compatible equipment, which is far friendlier than mainframe computers, cheaper than scientific workstations, and far more common than Macintoshes in government offices.
There are now much friendlier databases running under Windows 3.1 on personal computers which enable an intelligent layperson to set up custom reporting without writing programs ar all. One of the best is Borland's Paradox for Windows, which has recently been followed by Microsoft's Access, Fox Pro and others. As a relational database it stores data in multiple files (and yours will likely be large files) and supports multiple users, locking each record already in use but allowing many officers to work in the same database file simultaneously. The idea is that you design and fill in forms onscreen, choosing from pull-down menus and dragging buttons representing database fields and conditions into place to make a statement of what fields and values you want reported out. For example, you want all the requests from each requester combined into one monthly charge, but to eliminate charges under $25 which is your cost of billing. What would have taken a professional a page of code to write in dBase III plus two years ago, now takes an hour for an amateur (after a day learning how to do it once, of course.) Some of the new databases use the mouse for you to draw arrows between the fields when you want to indicate links. For those of you tied to mainframe and minicomputers by the sheer size of your databases, graphical software is on the way, particularly on systems running the UNIX operating system.
This of course puts pressure on the notion of an agency record under the FOIA: if a database report does not currently exist, with the traditional database it could take a professional some hours to write the code to generate a report requested; but with a new style graphical database, a FOIA officer with a little training could generate a new document without serious effort. My prediction would be that courts will expect FOIA staffs to generate new documents increasingly in future where material is stored in electronic form.
Costing by Spreadsheet.
If you lack a database (which does calculations as part of its functioning) you can calculate costs easily in a spreadsheet, with fewer errors than with hand calculators. The central office types in the formula so that all the other staff have to do is fill in their time and other costs to the spreadsheet template on the screen, then beam it back on the network to the accounting staff. If you were put off spreadsheets by the older generation such as Lotus 1-2-3, fear not: Windows spreadsheets such as Microsoft's Excel Windows, Borland's Quattro Pro Windows and the new version of 1-2-3 Windows are vastly friendlier and much more attractive to use. Individual sheets are combined automatically into a huge spreadsheet for the whole team's costs this month, and other sheets for the total for each requester this month, according to the design of FOIA central staff.
Communications: Within the Office.
Whether you'll use the new communications technologies depends largely on whether you work individually, or as part of a "workgroup" where someone edits and supervises your work, and someone else comments on your draft or asks you for some expertise. This is the hottest field in network computer products, but you may or may not want to take advantage of it. Let's consider the current possibilities.
The leading wordprocessors are now becoming mail-enabled, which means you can beam a document directly down a telephone or a network wire to your supervisor, and have it beamed back with comments. This is known as Fax-Like-You-Print or Mail-Like-You-Print. WordPerfect 5.2 for Windows and the new version 6.0 for DOS are good examples, in conjunction with WP Office 4.0 which works with just about any network and mailing protocol.
What if there's no network, just a telephone line? Well, a $60 fax-modem board in my computer makes it straightforward to type a document (like this one) and with a simple menu choice beam it down a wire in Iowa so that it prints on the Access Reports printer in Virginia ("Yes, Virginia, there is a fax-a-clause.") This is not only cheaper than fax machines, the results are crisper and (given the board and software at both ends) the image can be rotated, read by OCR (optical character recognition) software as text, stored and edited in a wordprocessor. If there is a non-fax modem at each end, the text can be transmitted and edited directly, but obviously any sketches, logos, photos are other graphical material will be lost. A twenty page document can be transmitted in minutes, but a book-length document would need automatic overnight transmittal which is usually scheduled from a menu.
Would your office use a network or telephone transmittal this way? Perhaps not, if you work with individual responsibility rather than as a workgroup and are not afraid of variations in processing quality. Where technology makes it easier for requesters to send in their requests, it may stimulate workload, so perhaps you won't adopt the technology externally, but only internally. Technology always serves specific purposes, and a certain cost, and the latent functions of an organization are rarely the same as the manifest mission.
Communications: Long Distance. The Internet model.
Up till now you may only have thought of communications within the office, plus the occasional impatient request via fax. But this is the least spectacular use of communications technology. There is already an international network of government offices, universities and research institutions called "internet", which uses leased fiberoptic lines (commonly) to let faculty beam research data and mail messages to each other, search remote libraries etc. Although there is no convenient menuing system and you are usually left at the old command-line prompt, this inelegant system has extraordinary resources. From my home study or my office, for example, I have sent messages to Massachusetts and Brazil, and I can search the catalogs of the major research libraries of the midwest. Obviously there are levels of password protection for defense research establishments, while university libraries tend to be opened to access. (For a fascinating paperback account of trapping an electronic spy using the internet, see Clifford Stahl, The Cuckoo's Egg.)
The system is powerful and growing rapidly, and could be used extensively for FOIA purposes; but friendly it is not. To illustrate this, let me give you a real example from my own work, which uses an invisible gateway from our local e-mail system to other universities' gateways clean across the country to Washington.
Addressing is hardly elegant or facile: if I want my students to send mail to president Clinton's staff, they have first to type "IN%" to send mail from our system to the internet, then "firstname.lastname@example.org", and if the president ever wants to reach me, I'm email@example.com. ("jeremy.lewis" is my username; "uni" is my university; and "edu" is the educational category of institution. Try it if you have access to internet.) You'll probably be in the "gov" category; the organization is by categories which cut across international boundaries, which is a nice touch.
The president could just reply to my message without knowing my address. But to initiate the messaging, he would have to know my username and address. To find the address of my institution, he would have to issue the internet command whois followed by northern iowa or some similar phrase, and then to find my username he'd have to issue the unpleasant command finger lewis which would give him a list including "lewis" (a colleague) and "lewisj" (myself).
The main difficulties at present are twofold: first, lacking menus, to get much out of the system you need a book on the internet system; and secondly, usage is increasing at 20% per month so that the system is in desperate need of expansion. That's where Senator Al Gore came in, sponsoring proposals (NREN, WINDO) to build data superhighways across the nation, analogous to the Interstate highway system for vehicles built in the 1950s (and now itself sorely in need of modernizing.) The president echoed his theme during the 1992 campaign, although it remains to be seen whether a substantial program will result.
Communications: Wholesale Data Submission and Broadcasting.
What does all this mean for the FOIA office of the future? Many government data are supplied by third parties: local jurisdictions provide the FBI crime figures, local ports provide the SEC's maritime database, oil companies the Energy Department's figures, and chemical companies the EPA Toxics Resource Inventory database. These will in future be shipped wholesale down a wire from one computer to another, as some are already; the technical problems of file formats and transfer protocols are usually overcome once the technicians at both ends have faced the challenge. Indeed, we can expect far more communication between the computers of different government agencies and levels, just as during the Reagan administration staff overcame the technical problems of "computer matching" various databases to check welfare fraud. Where there is "wholesale" uploading of data from the private sector or another government agency, it will in future be done electronically.
The EPA TRI database is actually published or broadcast wholesale, also. A researcher for a public interest group (let's say) can dial in and download to her personal computer the raw figures on three hundred different chemicals held in her state, although she might need considerable expertise to interpret the figures. This is surely an idea whose time has come for other agencies, wherever data are demanded which can be broadcast without line by line redaction being necessary. Once the computer has been set up with communications equipment, FOIA officers can get on with other work. Once installed, the method is cheaper than GPO and updated daily if necessary; remember, if you "publish" from a computer bulletin board this way, the FOIA implies you don't have to respond to individual requests in retail fashion. If you don't want to set up your own bulletin board (which needs one computer, one phone line and a $300 board) you may find the data are suitable for the National Technical Information Service (NTIS.)
Communications: Individual Submission of Forms.
But what about "retail" submission of data to government, where the data come in one form at a time? Let's try a scenario which presents a possible solution to a common real problem today. A doctor's office sees an adverse drug reaction in a patient and wants to alert the FDA to the problem. At present they fill in an ADR form in hardcopy (paper) and mail it. But it would be a straightforward matter for the nurse administrator to dial in to the FDA's computer, choose "ADR entry" from an extensive menu, and fill in the form on the screen to put it directly into the database. When entered as text, the fields for patient identifiers would of course be locked automatically by the database, with a code to FOIA exemption (b)(6) for personal privacy, and the FDA's particular regulations on minor deletions. An official might need also to scan the reports for other exempt material, though as far as possible this should be designed into the database fields up front.
People will always be afraid that learning a new procedure will take more time than it saves for occasional use, so we are still going to be seeing paper forms for the doctor's office that has one ADR per year: voice messaging boards in computers may be the answer there, so that one could fill in database fields orally and have the computer type in the text afterwards.
Communications: Individual Requests for Data.
Now suppose a drug company researcher from Prostate and Grumble dials in to the FDA computer to find all the ADR data on the drug in question. She follows simple menus to find the particular drug, the time periods she is interested in, etc, and then chooses either the data summary or the full text of the forms minus any exempt material. The computer sends her any references to rights of appeal against exemptions, where to find other information, and so on. In future, she ought to receive optionally also a compressed computer program for her PC which uses a friendly menu driven interface to help her analyze the statistics. Upon reaching her computer it expands, stores in a directory, and asks (by voice and by dialog box on the screen) whether it is needed to run right away.
Backward Compatibility: Let's Start from Reality.
Even though all of this uses existing technology, it would require implementation which inevitably is messier than the scenario. For example, the FDA FOIA staff itself has been plagued with difficulties for years with successive contractors setting up its minicomputer system; any agency has to enter a long bidding process which often results in current purchases of the last generation of technology. Their Parklawn Building does have plenty of PCs generally, but they come from twenty seven manufacturers and are difficult to network. Remember the Tower of Babel?
Let's make it worse: take away the assumption that text is entered from a supplier's keyboard and beamed down a wire, and consider the case of the agency with millions of pages of manila folder files stored in warehouses using the technology of the 1950s. Retyping into a computer is wildly impractical. For selected files, but almost certainly not for the vast mass, there is an alternative to the photocopier and the red marker, chinagraph pencil or scalpel which most FOIA officers still use for redacting.
Document Imaging and Onscreen Redaction.
Take a personal computer with the largest hard drive you can obtain, a scanner, and image editing and OCR software. The leading scanner is the Hewlett-Packard Scanjet, a flat bed whole-page machine which looks like a miniature photocopier, and costs a few hundred dollars from your local discounter. You load the software, preferably a serious package such as Caere's OmniPage Professional, Aldus's Photoshop or equivalent, and click the mouse on the Scan button. An image appears on the screen, which you save to the hard drive in a selected standard format, transferable to another machine and software later.
You can edit the image by adjusting sliders on the screen for brightness and contrast; if cleaning up is necessary, you can click the mouse on a tool such as an eraser, spray can or pencil and swab it over an area to clarify foreground from background. It would be a simple adaptation to use the mouse as a chinagraph pencil, purging exempt paragraphs from the text. Then you would save the redacted version under a different filename, knowing there is no chance of the redacted text "showing through", because it has been zapped electronically. It should be possible to do this also to blow-backs from microfiche, not just original paper documents.
A caveat: you could buy a hand-held scanner (e.g. the Logitech Scanman, or the Niscan) from your local CompUSA store or Walmart for under $200, but it'll take two passes to cover an eight inch wide sheet of paper, which have to be auto-stitched together with imperfect software. Then the image editing software may be extremely limited, with only two or three tools and contrast and brightness controls. Stick to the flat-bed scanner except for small technical illustrations.
Second caveat: deriving text from an image using even the best OCR software such as Caere's, or Calera's Wordscan, is s-l-o-w and difficult; save it for very clear images with simple typefaces such as Courier, and let it run overnight if you have a long document with more complext typefaces. You will still need to correct it afterwards (with spelling and grammar checking software, perhaps, the leaders being Grammatik and RightWriter) because at least 3 letters per hundred will be incorrect with current technology. A 486 or Pentium processor will be valuable here.
Third caveat: you'll need to experiment first with file formats before committing your whole office to one. The most common ones are TIFF (tagged-image file format), CGM (computer graphics metafile), DRW (Windows Draw format), BMP (bitmap: cannot be stretched or expanded), PCX (PC Paintbrush, like a bitmap and used by Windows paint), and WMF (Windows Meta File.) All of these can work well for selected purposes, but the bitmap is the lowest common denominator since it resembles a Seurat painting of dots. TIFF, DRW and CGM files are stored as arithmetic, allowing you to stretch or expand the image; but these formats from one machine and program may not work on another, because they each come in different flavors; I have frequently been frustrated by these variations. EPS files (Encapsulated Post Script) are the Rolls-Royce of formats but they take about three times the storage space of any other. If you need to convert and capture graphic images on a PC, the best software is Hijaak Pro, an inexpensive utility which can do it all. For shareware with no budget commitment, try Image Alchemy.
Fourth caveat: you may want 300 x 300 (or more) dot per inch resolution like a laser printer, but at more than a megabyte of storage per page, the limitations of your hard drive may force you to either use special compression software, or lower the resolution. For photographic images, we have commonly used 75 x 75, and for text images 150 x 150. For an image of 2" by 3" with 75 x 75 resolution (about the same as a computer screen,) we expect about 25K of storage, depending on the file format.
Once you've settled these problems, your office can plug away at digitizing images of documents, at a rate similar to hand-feeding a photocopier and touching up, but with more useful results. The pages of one document tend to use similar settings, which speeds up your work compared to processing individual images, and text is often clearer than grey-shaded photographs. (My student operators can scan and touch up ten photographic images per hour, not counting discarded ones.)
In the near future we can expect dramatic improvements in file compression and faster, sheet-feeding scanner hardware. Compression will also enable sending graphic files across a network, which at present is feasible but too slow to be popular. We are already seeing dramatic improvements in hard drives, the smallest of which are now 1.3" across, and the affordable range for an individual now extends to 600 MB, the same storage as a full CD-ROM. An office system would need gigabytes of storage of course, using rack-mounted drives. Optical drives which pack in more data using lasers have arrived on the market, and are becoming faster at seeking data, as well as more affordable.
The golden rule when linking computers together is actually two, which work in contradictory fashion. First, keep in mind what the net is used for, and buy technology accordingly; secondly, buy for the next generation because demand expands to fill supply. For occasional file exchange, just give each computer a modem and a phone line. For regular use, the first principle leads you to buy inexpensive phone-wire networks with Appletalk or Artisoft's Lantastic software/hardware ($200 per station), enabling simple mail and file transfers with minimal expense and amateur maintenance: the ten-minute networks which are more elegant than "sneakernet" (passing disks around.) The next step up is coaxial cable (as in cable tv connections) and faster network software such as Novell's Netware, the leading operating system, but this is much more expensive and requires technical support. Beyond the current use, and looking at the next generation, you're into a substantial budget request, a feasibility study and loads of fiber-optic cable with bridges and routers as you connect local networks together.
If you're simply sending files and mail between PCs, with no expectaton of expansion, use Lantastic (under DOS and Windows) for a net of twenty machines, and Netware (or possibly Windows for Workgroups from Microsoft) if you're linking forty or more. It is a much better value to give every worker a computer than to give half of them PCs and spend the rest of the money on expensive networking.
If you want to install two way video and audio in future, you'll need all the bandwidth you can get, and that means fiber optic cable. The video at each end is compressed a thousand to one going into the computer, and either stored on hard disks or transmitted down a wire to a remote monitor at which point it is decompressed and played back in real time. At present the effects are slightly jerky when there is movement in three dimensions, but this field is progressing rapidly.
You don't of course need personal computers for two-way video and sound: there are very expensive two-way classroom systems with remote cameras, compression boxes and projection monitors, though obviously you need the same equipment pre-installed at both ends. I have experienced some equipment in the $100,000 bracket which works fairly well.
Question is, how could you use the technology in FOIA processing? This is more of a long-shot. First of all it would obviously be valuable in training with remote classrooms, (just as in education, where Iowa is gearing up to have teachers reach classes in remote parts of the state simultaneously with their local class, having their questions answered by two-way video. The systems currently available are useable, assuming we teachers actually want to teach twice as many people while desperately manipulating cameras and keeping up the flow of a lecture.)
A FOIA office in San Francisco could zoom in to show a document and zoom out to discuss its redaction with its head office in Washington. Secondly, a FOIA officer could hold up a document on a live video link (with or without computers) and ask whether this it what the requester desires. But this is unlikely to occur on a daily basis, and as usual it requires the same technology at both ends.
Uncompressed video can currently be squirted into a window in a computer screen using inexpensive $500 boards such as the Video Blaster or Video Spigot or compressed video with the more sophisticated $1,000 Intel/IBM ActionMedia II boards. Software already allows the video clips to be trimmed, stored and attached to most kinds of document together with audio (e.g. Microsoft's Video for Windows, Apple's Quicktime for Windows, and Asymetrix's Multimedia Toolbook and Mediablitz.) Currently 10 seconds of audio sound-byte stores in about 25K of space and one minute of video in 1MB of space. If you send software on disk, the recipient still needs equivalent equipment to play it back.
Sound is catching on faster than video for this reason. The sound quality can be absolutely gorgeous at CD level quality (try a $300 Turtle Board with $400 Altec Lansing speakers with a subwoofer, and you'll fill a lecture hall) or it can be merely functional and cheap (a $90 Sound Blaster compatible board with $20 speakers.)
FOIA officers could send redacted documents in electronic form through mail or wire to suitably equipped requesters, with video and audio clips inserted to explain the redactions and advise of rights. Of course this is more complex than mailing the paper document with a cover letter, so the video and audio clips would probably be standardized, like form letters. If the technology is to develop it would have to be shown that the personal touch on video eased the reactions of today's aggressive requesters. A more likely format would be that the document has an interactive video form letter supplied with it on disk, which is reproduced more cheaply than video cassettes, and which explains processing and advises of rights, etc. While in-house systems can ensure the same equipment exists at both ends of a wire, this is always going to be a problem with outside parties.
If I receive your WordPerfect document enhanced with video and sound explanations, what do I do? I'll click the mouse on the video badges placed in the document to see you show me something in a video window about 3' x 4", or click on the icon of a mouth or an ear, and your voice comment plays back. The comment might explain your redaction of a paragraph and cite the law; of course the image might actually be of a charming tv spokesmodel who quickly disarms me of my indignation, and you might re-use the same clip a thousand times to unsuspecting requesters.
Interactive Training Software.
Many FOIA officers work part-time, and many rotate after a couple of years. Consequently, at my training sessions it is common to hear that half of the class has been attending to FOIA requests for months or years without really knowing how to comply with the law. Manuals are usually heavy and thick, so staff often resort to rules of thumb passed around the office. I still remember the trainee who said that her boss suggested just purging anything with numbers in it, and releasing the rest.
In the spirit of providing a cheap and lively solution to those of you who are comfortable with personal computers running Windows, I began experimenting this year on developing a "trainingware" tutorial working under Windows 3.1. A brief demo will be shown at the ASAP symposium this fall. It uses the pull-down menus, click buttons and pop-up boxes familiar to Windows or Macintosh users, and uses "hotwords" (themselves acting as buttons) to jump between different categories of information in hypertext fashion. The program will be available on disk for a nominal charge (wait for further news in these pages,) and a much larger version can be customized for your agency's regulations and procedures, with multimedia if required.
The aim is to serve up something for everyone, regardless of their FOIA experience level. A database within it allows you to reference different FOIA concepts, jargon or law clauses and obtain an explanation, examples of use and citations to law decisions where appropriate. Some simple animation helps you to understand redaction if you're a beginner, and the legislative history of the FOIA and its amendments is stepped through from a time line for FOIA buffs. There is a segment with news of FOIA developments abroad for those of cosmopolitan tastes.
Being considered for addition are a rogues' gallery of photos of leading FOIA officers and requesters with their hot tips, and a simulation game where you wrestle with the computer to get information either released or protected.
In order to play back on existing equipment, there is no audio or video ... yet. We can certainly add these if we find that plenty of FOIA staff are wired for sound and video.
In the past FOIA officers who wanted to use electronic technology had to deal with expensive mainframe computers and suffer reliance on consultants and contracting, only to find the software extremely unfriendly. But now, rapid advances in the commercial marketplace are letting enthusiastic amateurs put together systems which could perform some of the FOIA processing tasks using off the shelf hardware and software, plus some ingenuity and a local technician -- if they want to. To cut through the confusion, there is undoubtedly the sort of niche that Russ Roberts suggested in these pages, where a service company comes in with a complete hardware/software and training package.
However, to say the technology is affordable is not to say it will be adopted. That depends on its match to the real needs (or latent functions, in organization theory) of the office. If a FOIA office works by individual decisions, having a local network with e-mail won't make much difference to the redaction method. Having a bulletin board for publishing data probably will be most useful for "fill-froms", that is repeat requests for material already redacted, which can simply be scanned into a huge set of hard disks. Since this could save you from extensive human labor, it is the most likely productivity advance from technology. On the other hand, if a third party service company serves your agency's requesters, it may relieve the pressure of repeat requests (as FOI Services Inc does for the FDA.)
For relatively small workloads, personal computers in a network are generally cheaper than mainframe computers, and they run software which is now vastly friendlier than that running on mainframes. Graphical software running under Windows 3.1 (even in its first generation, mimicking the Macintosh) is enormously usable compared to the equivalent text-based software, and promises to improve substantially in the next two years, owing to the intense competition in the market. Setting up custom databases from a standard commercial package is almost certainly a better deal than buying a dedicated package and being tied to the company which wrote it.
There is a premium on having data typed in at its source, from which point on processing by computer database and communication by network is relatively straightforward. In future, it may well be possible to design government databases for direct public access, together with interpreting software to assist the requester. This potentially would reduce the use of the FOIA by extending government publication wholesale. But where data already exist in typed form on paper, scanning can be used only selectively. Where the data consist of doctors' handwriting on dog-eared paper, no technology except file tracking is likely to be worth the effort.