Information Overload

By 8500 BCE Sumerians settled between the Tigris and Euphrates Rivers in what is now southern Iraq. Annual flooding brought fresh soil to replenish the delta so grain could be grown year after year without depleting the land.  Farmers produced more than they needed to sustain themselves and their families, so they could sell the excess.  Trade required a record—information that could be entered, saved, and retrieved, and long before other records were written economic data was being collected.  The Sumerians made quarter-sized lumps of clay and inscribed marks that stood for the name of a commodity and how much of it was represented—five goats, two sheep, six jugs of wine.  The tokens were collected and saved in tennis ball-sized clay “purses” that recorded ownership and transactions in the oldest extant form of data storage and retrieval.

                Five thousand years went by before the Sumerians created a set of symbols that represented words rather than quantities, and it was another fifteen hundred years before Hammurabi had his laws inscribed on a seven-and-a-half-foot piece of black stone.  At first the accumulation of information was slow, but by the seventh century it had picked up.  In 1849, at the site of the ancient Assyrian capital Nineveh, British archaeologist Austin Henry Layard and his assistant Hormuzd Rassan made an astounding discovery.  They found the two main chambers of a library largely collected by Ashurbanipal (668-627 BCE), the last major Assyrian king.  Tens of thousands of clay tablets inscribed in cuneiform were stored on shelves in cabinets arranged by topics that included not only administrative records and laws but also medicine, astrology, history, and a variety of chants, prayers, and stories.  Each cabinet was given an identifying letter, and the shelves were numbered from the bottom up so new tablets could be piled on as needed.

Cuneiform is not easy to read, and the library arrangement made it likely that only the librarian  knew where things actually were.  As far as we know, Ashurbanipal’s library marked the first time the volume of information threatened the ability to find it.  Cumbersome, poorly organized libraries were fine for preserving information but were poor places to do research.  That was not as significant a problem as one might think since scribes were essentially the only ones who could read, and they spent their time copying what had already been written rather than trying to create anything new.

                Farther west, clay tablets inscribed with styli were superseded by marks made with ink on either papyrus or vellum sheets that were glued together and rolled up.  The scrolls could hold a huge amount of information—50-foot-long rolls were common.  One scroll that documented the rule of Rameses II stretched to 133 feet. There were no chapters and no page numbers, and the title of the book and its author were almost always at the end of the roll rather than the beginning.  The scrolls were stacked on shelves or, if a single volume needed more than one roll, in buckets on the shelves.  Although the scrolls lacked titles, some did have tags with a title hanging out—σίλλλυβόι, hence syllabi.

                The problem with retrieving information from a scroll was that, besides their being bulky,  one had to unroll the whole thing to find a passage unless the volume had been committed to memory, so the text was mostly used to verify what the reader remembered.  As in Assyria, Egyptian libraries were places to preserve what was already known with little intent to add anything new.

                The greatest of the ancient libraries was founded at Alexandria during the reign of Ptolemy I Soter about 288 BCE.  Tradition has it that Demeterius of Phaleron wanted to succeed Aristotle as head of the Athenian Academy but did not get the job.  Instead, he went to Egypt when Ptolemy offered him the chance to start a great library.  Demetrius came with no small amount of ambition;  his initial goal was to collect everything that had been written which he estimated would be about 500,000 books.  From the beginning, he bought some books and copied others.  For decades any ship that docked at Alexandria was forced to turn over every book on board.  What happened next varied.  Sometimes the original was returned; sometimes just the copy went back and the original stayed in Alexandria.  Ptolemy II put down a 15-talent deposit (equivalent to one skilled workers wages for seven years) to borrow a definitive copy of the works of Aeschylus, Sophocles, and Euripides from Athens.  When it came time to return the book, he forfeited the deposit and kept the original. 

                The Alexandrian collection may not have had the full 500,000 volumes, but it was huge and encompassed vast parts of Greek literature that no longer exists.  The library was organized by subject and topics within the subjects, but scrolls were undoubtedly hard to find and certainly hard to research if they were found.

                Over centuries the library  at Alexandria suffered from neglect and from repeated assaults on the city.  Some of the collection may have been lost in the 3rd century CE When Aurelian’s forces burned part of the city.  One contested theory is that the library was finally destroyed when Amr ibn al-As took the city in the 7^th century.  It is said that he decreed that all books in the library that did not deal directly with Islam should be burned.  Islamic literature that was not the Koran was not necessary and should be burned as well.  It is said that he distributed the scrolls to local bath houses to be used as fuel. At any rate, the library vanished along with an irretrievable treasury of ancient knowledge.

Romans copied the Alexandrian Greeks.  Augustus founded two libraries in Rome—one in Greek at Campus Martius and another similar in size and design for Latin on Palatine Hill.  Tiberius, Vespasian, and Trajan all built libraries in Rome, and Hadrian built one in Athens.  For the Roman emperors, libraries were status symbols.

An important change came in the early centuries of the Christian Era.  Codices—individual pages stitched together and put between covers—had been suggested by the Roman poet Martial in the 1^st century.  He praised a friend for folding parchment pages into leaves—a codex—instead of rolling them into a scroll.  Martial pointed out that codices were smaller and more appropriate for the traveler than bulky rolls, but his suggestion never caught on.

It was left to early Christians to popularize the form.  A codex held six times the content of a scroll; all four gospels fit in a book that would have taken five rolls.  Although they were not paginated, it was easier to flip pages to find a passage than to unroll an entire scroll, and the Christians placed a high value on having accessible liturgical writing. Besides being cheaper and more portable,  codices were easier to copy.   As long as manuscripts were hand copied however,  pagination, even in codices,  was impossible.  No two copies were exactly alike.  

During the Middle Ages, universities began collecting collect books that were not just stored but used for research.  By the 13^th century Oxford, Cambridge, and the Sorbonne all had reference libraries, although important books like dictionaries, concordances, Biblical commentaries, and the works of Thomas Aquinas were chained to desks.

Printing brought a fundamental change to information storage and retrieval.  As we saw earlier, there was an explosion of books in the first century and a half after Gutenberg’s Bible.  Ten to  fifteen thousand new texts were printed in the first fifty years, and 150-200 million books came out of the presses in the 16^th century.  Throughout the Middle Ages libraries had almost all been in religious or educational institutions.  Now private individuals could own books.  By the end of the century, “haberdashers, weavers, drapers, tanners, grocers, cheese mongers, hawkers, locksmiths, pastry cooks, skinners, dyers, shoemakers, and watchmakers had libraries.”  Gottfried Leibniz, David Hume, Johann Goethe, and Immanuel Kant all spent time as librarians. Books were affordable and abundant.  The increase in available information was staggering, but retrieval had become a huge problem.  The amount of information gushing out of printing presses far outstripped the ability of researchers and librarians to find it. 

One solution was to make it easier to search individual volumes.  As we saw in an earlier post, ancient Ugarit, a city in what is now Syria, developed an alphabet of consonants between 1400 and 1300 BCE.  The revolutionary thing about the Ugaritic system is that the letters were in strict order, something not true of either hieroglyphics or cuneiform.  That “abecedary” passed on to proto-Canaanite and to the Greeks, and eventually the orderly alphabet was used to create indexes.  In fairness, it was impractical to index hand copied scrolls or even codices since pagination varied from copy to copy.[1]  Nonetheless, rough alphabetization had been used in Alexandria.  The librarian Callimachus sorted the library into areas of learning and used an alphabetical list of the illustrious men hung above each shelf as an indication of what was there.  He also compiled a catalogue of what was in the library that took up 120 scrolls.

Sorting knowledge remained a serious problem through the Middle Ages.  Robert Grosseteste, a British poet, polymath, Chancellor of Oxford, and Bishop of Lincoln made a valiant attempt to categorize everything he had ever encountered.  Around 1230, Grosseteste divided what he had learned into groups of similar ideas.  He then divided each topic into one of three areas—Biblical, Church fathers, and pagan literature.  He created a system of 400 symbols that he drew in the margins of books to indicate what was in the text.  None of his books were paginated so it was necessary to scan each page looking for the right symbol to find a passage, but Grosseteste had created a search engine, albeit a clumsy one.

At about the same time, Cardinal Hugo de Saint-Cher enlisted 500 monks to sort through the Bible which had recently been divided into books and chapters (although not verses).[2]  Each monk was assigned a letter or part of a letter and sent to find every word that started with that letter and record where that word was found.  The alphabetized list of 10,000 terms would prove invaluable to local clerics writing sermons.  The concordance was a term based search engine and a major advance in information retrieval and presaged Google.

Printing eventually changed the world of data retrieval in another way; every page of a printed edition is precisely the same from volume to volume so the pages can be numbered. The first paginated book was printed in Cologne just fourteen years after the Gutenberg Bible, and tables of contents and indexes promptly followed.  Given the avalanche of printed material, it was just in time.  Still, indexes were only good for the individual volume, and the number of editions was staggering.  At the Second Conference of Librarians held in London October 2, 1877, J. Ashton Cross, the librarian of the Oxford Union Society, suggested a Universal Index of Subjects.  The goal was to compile an index of every branch of knowledge—essentially an index of indexes with each library to provide an index of its holdings and contribute a guinea a year to fund the project.  Cross’s great index proved too ambitious and it fizzled in less than a decade.

Another alternative is to condense information into a single volume or set of volumes.  The oldest known attempt at an encyclopedia (from the Greek enkyklios paideia or learning within a circle—literally well-rounded education) was the aggressively titled Assyrian All Things Known in the Universe from the 2^nd millennium BCE.  Plato’s nephew Speusippus published an encyclopedia in 370 BCE, and Marcus Terentius Varro published a nine volume summary of grammar, rhetoric, geometry, astrology, and music  between 16 and 27 BCE.  Unfortunately none of those survived.

Pliny the Elder’s Naturalis Historia was interrupted by the eruption of Mount Vesuvius that killed him in 79 AD.  He had already published ten books of a project that ultimately came to thirty-seven volumes completed by his nephew also named Pliny.  In the aggregate, it is the largest single work from the Roman Empire to survive.  It is divided into topics including Astronomy, Geography, Physiology, Zoology, Botany, Pharmacology, and Minerology.  Together, the Plinys claimed to have consulted 2,000 books by more than 1,200 authors to accumulate in excess of 20,000 facts and observations.

Pliny’s encyclopedia was unsurpassed until 1255 when Vincent of Beauvais published Speculum Maius (The Greater Mirror) , 80 books with 988 chapters and 4.5 million words divided into four sections.  Speculum Naturale dealt with astronomy, anatomy, agriculture, and light in the order in which they were created according to Genesis.  Speculum Doctrinale covered medicine, the mechanical arts,  and theology. History from the creation to the Last Judgment was in Speculum Historale.  Speculum Morale was added after Vincent’s death and is mostly taken from the writings of Thomas Aquinas.  In all, it was the largest western book until the mid-1500’s.

Europeans were not the only ones dealing with an excess of information.  Zhu Di, the 3^rd Emperor of the Ming Dynasty who ruled between 1402 and 1424 and who built the Forbidden City, ordered all Chinese knowledge to be gathered.  The resulting Yongle Dadian comprised 22,937 sections in 11,095 volumes bound in yellow damask.  Since it was written in Chinese ideograms, organization was by topic and there was no index.

The Enlightenment made a catalogue of rapidly growing western knowledge imperative.  Denis Diderot undertook the job and, between 1745 and 1772 compiled and edited (along with Jean Le Rond d’Alembert) an Encyclopedie that eventually came to 17 volumes containing 71,8128 articles accompanied by 3,129 plates in 11 more volumes.  The Parlement of Paris suppressed the work after the first seven volumes in 1758, but with the assistance of government Director of Book Trade La Moignon de Malesherbes who warned Diderot and hid his papers, the project was surreptitiously finished along with supplementary updates and a two volume index.

Not to be outdone, the British published their own three-volume Encyclopedia Britannica between 1768 and 1771 in Edinburgh.  Like the Encyclopedie, the Britannica was sorted by topics arranged alphabetically.  The work is a cautionary example of information based on the interests and biases of the editors.  Farriery (treatment of the diseases of horses) got 40 pages.  Morals merited another 40 and the Law 76, but Murder got only 4 lines.   Surgery warranted 38 pages, Midwifery 39, and Medicine 112 more.  Brasil (sic) got 12 lines while Norway got only 7, the first 5 of which were latitude, longitude, and surrounding countries.  The rest of the entry was brief:  “It is a cold and barren country.”[3] Panama got 5 lines also comprising the location to which was added:  “Where the treasures of gold and silver and other rich merchandize of Peru are lodged in magazines till they are sent to Europe.” 

Americans got into the act as well.  Thomas Dobson began advertising subscriptions for an American version of the Britannica in 1789, but he added a number of local contributors and dropped Britannica from his title before the encyclopedia was published.  One of his  authors, who wrote entries on individual state histories, American Indians, and Revolutionary War battles, was a man we have encountered before.  Jedidiah Morse was Samuel F.B. Morse’s father.

Britannica’s high point was the 11th edition published in 1910.  My own copy is 28 volumes of double column small print on tissue thin pages that occupy an entire library shelf. The 11^th edition has been justifiably called the last time it was possible to put all western knowledge in a single set of books.

Libraries concordances, indexes, and encyclopedias were all attempts to manage an exponentially increasing amount of information.  By the last half of the 20^th century they had all proven woefully inadequate.  An entirely new approach was necessary.  Fortunately it appeared, and that will be where we will pick up the story in the next post.

[1] The Torah was an exception.  Exact copying was a religious obligation and the Torah scrolls were identical from one to another.  Regardless, they were not indexed.

[2] The illustration at the beginning is the first known image of a person wearing glasses, although it was painted more than a century after Hugo’s death.

[3] The 11^th edition of the Britannica had twenty-two double column pages that would have run to something more than fifty pages in the same type and page set as the first edition.