1339 Mission, 2000 by Erica Fischer , CC BY 2.0 . A while ago, colleagues from our imaging studio approached me about an old Inmagic DB/TextWorks database. The database was originally created around the year 2000, and contained information about photographed items from our collection. From an old machine of a retired former colleague, they had recovered both the database, and installer files of…
Tools by Wilfredor , CC0, via Wikimedia Commons. My recent work on the Microsoft Multiplan and Quattro Pro for DOS formats made me think how far LibreOffice has come over the past years in its support of legacy file formats. These formats are supported through software libraries that are developed within the Document Liberation Project (DLP). This project was set up in 2014 by The Document…
Image sourced from Internet Archive , license unknown. in 2014 I wrote a post on the Quattro Pro for DOS spreadsheet formats. This documents my attempts at reading a few old Quattro Pro for DOS spreadsheets from my personal archives with modern (at the time) software. Back then, neither Microsoft Excel nor LibreOffice Calc supported these formats. Only the then-current Quattro Pro X7 was able to…
Image sourced from Internet Archive , license unknown. Last week I was contacted by Tristan Zondag from the Netherlands Institute for Sound and Vision for advice on files he had recovered from some old floppy disks as part of an ongoing digital archaeology project . Based on the files’ byte structure, Tristan suspected that these were Multiplan files. Multiplan is a spreadsheet application that…
Donkey and cart icons licensed from the Noun Project . Github Octicon icon from Wikimedia Commons , released under MIT license. Codeberg logo from Wikimedia Commons , released under CC0 1.0 . Ever since its start in late 2013, this blog has been hosted on Github Pages, using the Jekyll static site generator. On a technical level this always worked flawlessly, but in the current geopolitical…
Digital Dark Age Crew logo. One of the most elusive items in the Digital Dark Age Crew back catalogue is “Y2K”, which deals with the Year 2000 problem . Originally planned as a December 1999 release, the track was never finished due to a succession of technical problems. Some early demos of “Y2K” have surfaced as bootlegs, and many fans of the group rate these amongst the most sought-after Digital…
Control room in Fossil fuel power plant in Point Tupper, Nova Scotia . Achim Hering, Public domain, via Wikimedia Commons. This post introduces Pdfquad , a software tool that for automated quality assessment for large digitisation batches. The software was developed specifically for the Digital Library for Dutch Literature (DBNL), but it might be adaptable to other users and organisations as well.…
A man in armour is confronted by a ghost and a skeleton. Aquatint. Wellcome Collection, Public Domain. In a recent blog post , colleagues at the National Digital Preservation Services in Finland addressed an issue with PDF files that contain strings with octal escape sequences. These are not parsed correctly by JHOVE , and the resulting parse errors ultimately lead to (seemingly unrelated)…
Adapted from Quality Coal by Greenville Daily Photo . Used under CC0 1.0. license . In my previous post I addressed several problems I ran into when I tried to estimate the “last saved” quality level of JPEG images. It described some experiments based on ImageMagick ’s quality heuristic, which led to a Python implementation of a modified version of the heuristic that improves the behaviour for…
Bailey AKA the "I have no idea what I'm doing" dog . License unknown. In this post I explore some of the challenges I ran into while trying to estimate the quality level of JPEG images. By quality level I mean the percentage (1-100) that expresses the lossiness that was applied by the encoder at the last “save” operation. Here, a value of 1 results in very aggressive compression with a lot of…
"Confused, muddled, illogical" . Used under Pixabay License . The KB has been using JP2 (JPEG 2000 Part 1) as the primary file format for its mass-digitisation activities for over 15 years now. Nevertheless, we still use uncompressed TIFF for a few collections. At the moment there’s an ongoing discussion about whether we should migrate those to JP2 as well at some point to save storage costs. Last…
"The Barnum & Bailey greatest show on earth" . Used under CC BY-BY 2.0 , via Boston Public Library. Last month I wrote this post , which addresses the use of JHOVE and VeraPDF for identifying preservation risks in PDF files. In the concluding section I suggested that VeraPDF’s parse status might be used as a rough “validity proxy” to identify malformed PDFs. But does VeraPDF’s parse status…
"Rock 'em Sock 'em Robots Game" by Lorie Shaull, used under CC BY-SA 4.0 , via Wikimedia Commons. The PDF format has a number of features that don’t sit well with the aims of long-term preservation and accessibility. This includes encryption and password protection, external dependencies (e.g. fonts that are not embedded in a document), and reliance on external software. In this post I’ll review…
Clockwork picture of an itinerant dentist performing an extraction in French rural scene, wood frame, metal workings, first half 19th century. Science Museum, London . Attribution 4.0 International (CC BY 4.0) (cropped from original). This blog post provides a brief introduction to extracting unformatted text from EPUB files. The occasion for this work was a request by my Digital Humanities…
Donkey, cart and sun icons licensed from the Noun Project . I recently moved the two Internet domains I own away from the UK-based domain registrar I’d been using since 2004 to a EU-based registrar. While the actual domain transfer was fairly simple, finding a registrar that suited my specific situation turned out more difficult than expected. Leaving my old registrar also resulted in a surprise.…
In 2017 I wrote a blog post on Iromlab (an acronym for “Image and Rip Optical Media Like A Boss”), a custom-built software tool that streamlines imaging and ripping of optical media using an Acronova Nimbie disc robot. The KB has been using Iromlab since 2019 as part of an ongoing effort to preserve the information contained in its vast collection of legacy optical media. This project is expected…
"Fail whale" by Kuni (ka-92) (license unknown), based on "Lifting a Dreamer" by Yiying Lu . As a total collapse of Twitter is becoming more likely every day, many Twitter users have started to archive their personal data from the platform while it still exists. Twitter allows you to request and download your personal archive. Even though this works well, and the quality of the archive is…
Dutch electro outfit the Digital Dark Age Crew are one of the forgotten legends that used to be a mainstay of Rotterdam’s late 90s to mid-2000s underground electro scene. Their music was characterised by relentless electro beats, sparse synth lines, and lyrics that typically commented on the fragility and transience of digital media and digital information in general. In a twisted turn of events,…
This blog post covers some techniques that can be used to identify storage media and storage devices using Python and the Windows API. This can be useful for distinguishing between different types of portable storage media, such as floppy disks and USB thumb drives. It also presents a demo script that integrates these techniques. Preservation of portable storage media In 2019 the KB started a…
It’s been a while since the last release of the Isolyzer tool, but after four years of near-inactivity I just published Isolyzer 1.4 . In this post I provide some background information on how this release came about, and I briefly explain the main changes. History of Isolyzer For those unfamiliar with the Isolyzer tool, here’s a brief recap. Isolyzer started its life in 2015. At that time I had…
Intensive Breeding by Jean Marc Cote, Public domain, via Wikimedia Commons. At the KB we’ve been using JP2 ( JPEG 2000 Part 1) as our primary image format for digitised newspapers, books and periodicals since 2007. The digitisation work is contracted out to external vendors, who supply the digitised pages as losslessly compressed preservation masters, as well as lossily compressed access images…
Earlier this month saw the publication of The Significant Properties of Spreadsheets . This is the final report of a six-year research effort by the Open Preservation Foundation’s Archives Interest Group (AIG), which is composed of participants from the National Archives of the Netherlands (NANETH), the National Archives of Estonia (NAE), the Danish National Archives (DNA), and Preservica. The…
Plumbers Tool Box by pszz on Flickr. Used under CC BY-NC-SA 2.0 . Over the years, I’ve been using a variety of open-source software tools for solving all sorts of issues with PDF documents. This post is an attempt to (finally) bring together my go-to PDF analysis and processing tools and commands for a variety of common tasks in one single place. It is largely based on a multitude of scattered…
Production photo from "2001: A Space Odyssey". ©Stanley Kubrick Archives/TASCHEN. My previous post addressed the emulation of mobile Android apps. In this follow-up, I’ll explore some other aspects of mobile app preservation, with a focus on acquisition and ingest processes. The 2019 iPres paper on the Acquisition and Preservation of Mobile eBook Apps by Maureen Pennock, Peter May and Michael Day…
"Android Robot" by Google Inc., used under CC BY 3.0 , via Wikimedia Commons. So far the KB hasn’t actively pursued the preservation of mobile apps. However, born-digital publications in app-only form have become increasingly common, as well as “hybrid” publications, with apps that are supplemental to traditional (paper) books. At the request of our Digital Preservation department, I’ve started…
“World Wide Web - Digital Preservation” by Jørgen Stamp , used under CC BY 2.5 DK ; “Wadden Sea” by Envisat satellite, used under CC BY-SA 3.0-IGO . Earlier this year I wrote a blog post about geo-locating web domains, and the subsequent analysis of the resulting data in QGIS . At the time, this work was meant as a proof of concept, and I had only tried it out on a small set of test data. We have…
Original artwork copyright ©Liesbet Zikkenheimer. In 2019, Dutch telecommunications company KPN announced its plans to phase out its subsidiary XS4ALL , which is one of the oldest internet service providers in the Netherlands. With this decision, thousands of homepages and personal web sites that are hosted under the XS4ALL domain are at risk of disappearing forever. The web archiving team of…
“The Scream” , undated drawing by Edvard Munch, Bergen Kunstmuseum, Public domain. Earlier this week Library of Congress added a new entry on the standard “Digital publishing — EPUB3 preservation” (ISO/IEC TS 22424) to its excellent Digital Formats web site. This standard was developed by the ISO Technical Committee on Document description and processing languages , and was published in January…
“Broken zip £12.50” by dichoecho , used under CC BY / Cropped from original. We recently started using Microsoft OneDrive at work. The other day a colleague used OneDrive to share a folder with a large number of ISO images with me. Since I wanted to work with these files on my Linux machine at home, and no official OneDrive client for Linux exists a this point, I used OneDrive’s web client to…
Following earlier work on the preservation of optical media and data tapes, I recently got a request to make an inventory of offline digital data carriers in the KB’s deposit collection. The goal was to obtain approximate figures on the various carrier types in the collection. This was partially prompted by a project on at-risk digital heritage on physical carriers by the Dutch Digital Heritage…
A few weeks ago one of my web archiving colleagues approached me with an interesting question. From a list of Dutch web domains, he wanted to identify the (Dutch) province in which each domain is hosted. He was particularly interested in domains hosted in the province of Friesland . After some experimentation I was able to answer this question using a two-step procedure: Geo-locate the web domains…
Earlier this year I published this blog post on the recovery of data from ’90s data tapes. I will give a presentation on this during the upcoming iPres 2019 conference , and wrote a paper that discusses this work in more detail than my earlier blog post. The original paper (in PDF format) can be found here . The paper references a wealth of useful resources, but some of these are not easily…
SATA hard disk, USB Flash drive and 3.5" floppy disks As I explained in the introduction of this earlier blog post , as part of our ongoing web archaeology project we are currently developing workflows for reading data from a variety of physical carrier formats. After the earlier work on data tapes and optical media , the next job was to image a small box with 3.5” floppy disks. Easy enough, and…
In 2015 I wrote a blog post on preserving optical media from the command-line . Among other things, it suggested a rudimentary workflow for imaging CD-ROMs and DVDs using the readom and ddrescue tools. Even though we now have a highly automated workflow in place for bulk processing optical media from our deposit collection, readom and ddrescue still prove to be useful for various special cases…
When the KB web archive was launched in 2007, many sites from the “early” Dutch web had already gone offline. As a result, the time period between (roughly) 1992 and 2000 is seriously under-represented in our web archive. To improve the coverage of web sites from this historically important era, we are now looking into Web Archaeology tools and methods. Over the last year our web archiving team…
In a previous blog post I showed how we resurrected NL-menu , the first Dutch web index. It explains how we recovered the site’s data from an old CD-ROM, and how we subsequently created a local copy of the site by serving the CD-ROM’s contents on the Apache web server . This follow-up post covers the final step: crawling the resurrected site to a WARC file that can be ingested into our web…
NL-menu was the first Dutch web index. The site was originally founded by a consortium of SURFnet , Dutch universities and the KB. From the mid-nineties onwards it was maintained solely by the KB. NL-menu was discontinued in 2004 , after which the site was taken offline. In 2006 the domain name was sold to a private company that used it for hosting a web index that was partially based on the…
Earlier this year I blogged about Isolyzer , a tool designed to help the detection of broken ISO images. Today I released a shiny new beta version that adds a significant amount of new functionality. Below is an overview of the main changes, followed by some warnings and caveats. Support of more file systems Where previous versions only supported disc images with an ISO 9660 file system (with…
Over the last months we’ve been working on the development of a provisional workflow for preserving the content of optical media in our collection. The main result thus far is Iromlab , a custom workflow application that streamlines the imaging and ripping process. This blogpost gives an overview of Iromlab , as well as the reasons why we created it in the first place. The challenges The first…
Some four years ago I wrote a blog post that demonstrated how Apache Preflight (the PDF/A validator tool that is part of Apache PDFBox ) can be used to detect features in a PDF that are potential preservation risks. A follow-up blog applied Schematron rules to the Preflight output in an attempt at doing policy-based assessments. The results of that work were quite promising, but dealing with…
The development work on an imaging/ripping workflow for optical media is shaping up steadily, and you can expect a write-up with more information about our software and hardware setup here in the near future (you can get a sneak peek here ). However, this blog is about a very specific problem that we ran into while testing the workflow with a selection of discs from our collection. This selection…
In my previous blog post I addressed the detection of broken audio files in an automated workflow for ripping audio CDs. For (data) CD-ROMs and DVDs that are imaged to an ISO image, a similar problem exists: how can we be reasonably sure that the created image is complete? In this blog post I will discuss some possible ways of doing this using existing tools, along with their limitations. I then…
At the KB we have a large collection of offline optical media. Most of these are CD-ROMs, but we also have a sizeable proportion of audio CDs. We’re currently in the process of designing a workflow for stabilising the contents of these materials using disk imaging. For audio CDs this involves ‘ripping’ the tracks to audio files. Since the workflow will be automated to a high degree, basic quality…
Earlier this week the National Archives of the Netherlands (NANeth) published a report on preferred file formats . It gives an overview of NANeth’s ‘preferred’ and ‘acceptable’ formats for 9 content categories, and also explains the reasoning behind the selected formats. Even though in Dutch language only, the report is well worth a look. However, I found a few of the choices a little surprising,…
EpubCheck is an invaluable tool for assessing the quality of EPUB files. Still, it is possible that EPUB s that are valid according to the format specification (and thus EpubCheck ) are nevertheless inaccessible to some users. Some weeks ago a colleague sent me an EPUB 2 file that produced some really strange behaviour across a number of viewer applications. For a start, the text wouldn’t reflow…
About a month ago the International Digital Publishing Forum , the standards body behind the EPUB format, published an Editor’s Draft of EPUB 3.1 . This is meant to be the successor of the current 3.0.1 version . IDPC has set up a community review , which allows interested parties to comment on the draft. The proposed changes relative to EPUB 3.0.1 are summarised in this document . A note at the…
Yesterday (7 December) we released version 1.16.0 of the jpylyzer tool, which is this year’s third release of the software (excluding bugfix releases). This blog post gives a brief overview of the main jpylyzer improvements that have been implemented over this year. Changes in XML output The 1.14 release introduced two output improvements. Most importantly, an XML Schema Definition (XSD) was…
The KB has quite a large collection of offline optical media, such as CD-ROMs, DVDs and audio CDs. We’re currently investigating how to stabilise the contents of these materials using disk imaging. During the initial phase of this work I did a number of tests with various open-source tools. It’s doubtful whether we’ll end up using these same tools in our actual workflows. The main reason for this…
Today my attention was caught by this report of an “Expert round table” on JPEG2000 and Digitisation , which was published on the TownsWeb Archiving blog. Although the report as a whole is quite balanced, it’s unfortunate that it provides fuel to some long-running myths about JPEG 2000 not supporting fully lossless compression. Since I wasn’t able to leave a comment on the Townweb blog itself, I…
This is the second and final instalment of a 2-part blog on the use of PDF/A validators for identifying preservation risks in PDF. You can read the first part here . In Part 1 I showed how PDF/A validators can be used to identify preservation risks in a PDF. I illustrated this with an example that uses the PDF/A validator component of Adobe Acrobat’s Preflight tool. Needless to say, Acrobat is not…