The Valley of the Shadow: Scholarship in the Digital World

Edward Ayers

The University of Virginia’s Institute for Advanced Technology in the Humanities (IATH), created in the early 1990s, is based on a straightforward principle: to match motivated humanists with broad-minded computer scientists to create new tools for understanding the record of human experience. The Valley of the Shadow: Two Communities in the American Civil War is one of IATH’s two founding projects. Edward Ayers, dean of the College and Graduate School of Arts and Sciences and the Hugh P. Kelly Professor of History at the University of Virginia, conceived of the Valley of the Shadow Project and has been its driving force for over a decade. Ayers describes the evolution of the project—which began before the World Wide Web existed—from his original vision of a brief book on the difference the Civil War made in the lives of people on both sides of the Mason-Dixon Line to what has been called in the Journal of American History the “most sophisticated historical site on the Web.”

The Valley of the Shadow

The Civil War offers a wonderful opportunity to do social history, the kind of history to which I am devoted. One of my primary aims in starting the Valley Project was to understand how two similar places in close proximity to each other and to the border between the North and the South could go to war. I hoped to try my hand at a kind of social history that involves following characters throughout a comparative story.

I began by sitting down with maps and guides to military units and indexes of newspapers to find two places centrally involved in the Civil War from start to finish. It did not take long to discover that two places stood out: Augusta County, Virginia, and Franklin County, Pennsylvania. Lying about 200 miles apart in the Great Valley that cuts across the Mason-Dixon Line, these two communities were similar in many ways. The ethnic backgrounds of the white population, the climate, soil, and crops, the religious denominations and political parties of the two places shared a great deal.

Only one real difference divided the two places: slavery. Five thousand enslaved people, about a fifth of the population, lived in Augusta. Slavery had become woven into the economy, the political beliefs, and the hearts of white residents of the Valley. Although many of the Virginians in the Valley harbored doubts, public and private, about the Confederate cause and its effects on Virginia, most threw themselves into the fight once the line had been drawn. The situation in Pennsylvania bore ambiguities and tensions. White Pennsylvanians, all, did not hesitate to circumscribe black rights and opportunities; in the districts so close to the Mason-Dixon Line, sympathizers with the South were more visible than abolitionists. Yet white Pennsylvanians fought and died in a war that became a war to end slavery, finding ideals and aspirations they had not held before. It was the sudden and powerful redefinition of people on both sides that drove my imagination.

As I considered the book, with its close attention to millions of details found in newspapers, diaries, census entries, and so on, and the effort to find patterns hidden in masses of scattered records, I slowly realized that computers—something that few of us in the English or history departments at the University of Virginia (UVA) had on our desks at the time—could be of enormous help, as the project entailed a large database with many variables from diverse sources.

With the support and encouragement of William Wulf, now president of the National Academy of Engineering and then a professor of engineering in the University of Virginia’s computer sciences department, the book was transformed into a lead project of UVA’s nascent Institute for Advanced Technology in the Humanities. The idea for the project was straightforward: to put every piece of information about every person in a Northern community and a Southern community in the era of the Civil War in a digital context so that students and scholars would have an unprecedented command over those millions of pieces of evidence. With these tools, people would be able to understand the immense complexity of historical change.

The largest single problem we faced—similar to others in the humanities—was getting material into the computer. We paid to have a company scan and digitize about 10,000 pages of newspapers off microfilm, but there was no software that could conveniently handle such images, so we had to rely on awkward file formats and image-manipulation tools. Those pages, set in type seven generations before, had then been subjected to wars, water, fungus, neglect, tape, and sometimes careless microfilming. Even more challenging, much of the other evidence in the Valley Project was hand-written. Not only letters and diaries had been produced in pen and ink, but so had tens of thousands of names in the population, agricultural, manufacturing, and slaveholder censuses. My original dreams of automatic conversion of text by optical character recognition died a quick if painful death. There would be no choice but to transcribe virtually everything that went into the digital archive. And so we began that seemingly endless task, with a few hours from work-study graduate students.

Slowly, the digital archive began to take shape. Within a year, we amassed enough transcribed newspaper articles, enough names from the census, enough diaries and letters, and enough military records to create prototypes of the databases of those records. We scanned enough maps and images to demonstrate what those would look like in the archives.

The Valley Project follows an idiosyncratic way of approaching a familiar historical event. Rather than gathering the information that people might think they need to understand the Civil War—biographies of generals and presidents, say, or editorials from leading newspapers—the Valley Project gathers mundane information about anonymous people in obscure places. The material it presents is common and, in isolation from other material, not particularly meaningful. Put in context, however, newspaper articles take on drama and power; personal letters and diaries evoke tears and laughter; even census entries and military records can bring chills.

Because the Valley Project was conceived by a historian and built by several talented younger historians who have learned to think digitally, it is firmly rooted in the discipline in which it is used. Only a historian interested in social, political, economic, and cultural history would have devised such a strange tool, so far removed from the commercial reference tools, document collections, and games about the Civil War. The success of the Valley Project has been surprising.

American Historical Review Electronic Article

As fortunate as the history of the Valley Project has been, however, it has by no means established digital media as an entirely legitimate form of scholarship. The project, after all, makes no scholarly argument of its own; it puts forward no thesis to be tested. It does not provide a narrative of events against which students can test their own interpretations, and it does not engage the immense scholarly literature on the Civil War.

As fortunate as the history of the Valley Project has been, however, it has by no means established digital media as an entirely legitimate form of scholarship.

As a result, the team behind the Valley Project decided to try to close the circle. Commissioned by the American Historical Review, the leading journal in the discipline in the United States, William Thomas and I attempted to make an argument—along the lines of traditional historical scholarship—based on the Valley Project. We decided to return to the long-glimmering hope that hypertext might permit new kinds of exposition and analysis, using the possibilities afforded by XML for more flexible linking as an opportunity for a new pass at the problem. Thomas took the role of lead author, for he combines professional scholarly knowledge and computer expertise in a way few historians can.

The “article,” “The Differences Slavery Made: A Close Analysis of Two American Communities,” ended up being more than 300 pages long in printed form. The capacious digital medium imposed none of the familiar constraints of word count and page size; as a result, the number of references to the scholarly literature and the amount of evidence deployed rose to levels far beyond what is normally possible or permitted. In this way, the new medium allowed the argument to be articulated with a precision impossible on paper. It also revealed the compromises that scholars routinely make merely to fit their work into a fixed medium.

Our principal goal was to open up the process of scholarly inquiry by fusing the electronic article’s form with its argument and by using the medium as effectively as possible to make the presentation and navigation of our work express and fulfill our argument. As a result, this piece of electronic scholarship operates on several levels to connect form and analysis. First, it allows one to reconstruct the process by which our argument was developed by following the logic of our thinking. It also uses spatial analysis and presentation to locate its subjects and its readers within the context of the historical evidence and interpretation. The methodology of the emerging field of historical GIS (geographic information systems) informed our analysis and led to the creation of a comparative spatial database of our communities. Third, the electronic article presents itself in a form to allow for unforeseen connections with future scholarship. We consider this last goal critically important for scholars working in the digital medium because the rate of technological change will certainly offer new opportunities even as it displaces current practices. Publishers, scholars, technologists, and librarians have worked out international standards that govern the basic structures and forms of digital work to take advantage of technological development. Our article seeks to work within these standards in expectation of change.

…the new digital medium allowed the argument to be articulated with a precision impossible on paper. It also revealed the compromises that scholars routinely make merely to fit their work into a fixed medium.

The openness that technology affords and the alternative readings possible within the article raise questions about the role of narrative and argument in electronic scholarship. Literary critics have often speculated on the future of narrative in cyberspace, yet examples of nonlinear analysis remain few and far between. Despite all of the new technologies and the excitement of the medium, the Web, it turns out, is full of traditional linear narrative in large measure because so much of the material on the Web has been migrated from print. Yet the digital medium offers—in some respects demands—hyperlinking in that it excels in the presentation of linked information and modules of analysis and explanation. In the end, the nonlinearity that is necessarily a part of digital scholarship should not obscure its argument, but at the same time the argument in digital scholarship cannot ignore the nonlinearity of the medium.


The range of digital scholarship is likely to be quite broad, and the key advantage of the digital medium for publication is the openness it provides to scholarly authors to design and create a presentation uniquely suited to their topic, field, period, or problem. In our case, we hoped to create a flexible framework for digital scholarship for historians. The idea behind our approach was to develop a set of common categories and provide some definition to their relationship. We avoided creating idiosyncratic elements peculiar to our work and we strove to produce a template other scholars might use for a piece of digital scholarship.

We plan to make something similar to this article’s technologies broadly available, using its structures for different objectives, historical questions, periods, and concerns. As an extension of our work on the electronic article, we have begun to create an application called CHART, for Comprehensive History Analysis and Research Tool, which could work in a high school classroom as well as in a professional journal. CHART permits the use of XML and its advantages without requiring the large scale, daunting complexity, and considerable cost our prototype article has demanded. It seems that digital scholarship is particularly well suited for some forms of historical analysis; we put our attempt forward as an early experiment to see if that might be so.

Ultimately, we hope that new technologies can offer tools for wide audiences at a minimal cost, helping to democratize higher education and offering the possibility of rich learning environments to everyone who has access to a computer and the Internet.

Edward Ayers is the Dean of the College and Graduate School of Arts and Sciences and the Hugh P. Kelly Professor of History at the University of Virginia. His 1992 book, Life After Reconstruction, was a finalist for both the National Book Award and the Pulitzer Prize. Ayers has won several teaching awards, including most recently the 2003 National Professor of the Year award from Carnegie Foundation for the Advancement of Teaching and the Council for Advancement and Support of Education.