Fixing almost-valid EPUBs

I opened an EPUB with Zathura, my document reader of choice, and it instead displayed every image in the document on its own page, with no sign of any text. EPUBs are essentially a collection of HTML, CSS, and image files in a zip file, so I guessed that the file was being interpreted as a mere zip file, rather than as a document in its own right. file agreed, describing it as Zip data. So to fix it, we’ll need to figure out how software determines if a zip file is also an EPUB.

We find that the EPUB specification has a section on this. EPUBs must:

• contain a file called mimetype with the content application/epub+zip
• list the mimetype file first in the archive
• store the mimetype file uncompressed

I could see the EPUB had a mimetype file, but it looked to be compressed. We can use unzip to take apart the archive, then zip -0 to put it back together again without compression. We should also add the mimetype first, followed by the rest of the document.

The result still wasn’t detected as an EPUB. Out of ideas, I read some of the source code of libmagic: the most-used library for figuring out the type of unknown files. libmagic uses a comprehensive database of file signatures to help it determine what a file is. The only mention we see for EPUB is the following:

#  EPUB (OEBPS) books using OCF (OEBPS Container Format)
#    https://www.idpf.org/ocf/ocf1.0/download/ocf10.htm, section 4.
#    From: Ralf Brown <ralf.brown@gmail.com>
>>50	string	epub+zip	EPUB document
!:mime application/epub+zip

We can take an educated guess that fourth line is checking for the string epub+zip within the first 0x50 (80) bytes of the file. Does our EPUB file match this signature? We turn to hexdump -C to figure it out:

00000  50 4b 03 04 0a 00 00 00  00 00 29 a3 b4 5a 3b d1  |PK........)..Z;.|
00010  f6 ef 15 00 00 00 15 00  00 00 08 00 1c 00 6d 69  |..............mi|
00020  6d 65 74 79 70 65 55 54  09 00 03 1d d7 2c 68 1d  |metypeUT.....,h.|
00030  d7 2c 68 75 78 0b 00 01  04 e8 03 00 00 04 e8 03  |.,hux...........|
00040  00 00 61 70 70 6c 69 63  61 74 69 6f 6e 2f 65 70  |..application/ep|
00050  75 62 2b 7a 69 70 0a 50  4b 01 02 1e 03 0a 00 00  |ub+zip.PK.......|
00060  00 00 00 29 a3 b4 5a 3b  d1 f6 ef 15 00 00 00 15  |...)..Z;........|

We can see the contents of the mimetype file—and it looks to be uncompressed. However, we see that the epub+zip portion extends past the first 0x50 bytes, so this is probably why the file isn’t being identified as an EPUB. Looking back a bit we see a gap of unreadable bytes, but after that we can see the title of the file. This gap presumably stores other information about the file, but it’s taking up too much room! Sure enough, the EPUB specification mentioned this, but I just missed it.

Scrolling the manpage for zip, we find the argument -X or --no-extra, which instructs it to not save extra file attributes. This sounds like exactly what we want. If we use this argument (I couldn’t get the long form to work, but the short one did) and run hexdump -C again, we get this:

00000  50 4b 03 04 0a 00 00 00  00 00 29 a3 b4 5a 3b d1  |PK........)..Z;.|
00010  f6 ef 15 00 00 00 15 00  00 00 08 00 00 00 6d 69  |..............mi|
00020  6d 65 74 79 70 65 61 70  70 6c 69 63 61 74 69 6f  |metypeapplicatio|
00030  6e 2f 65 70 75 62 2b 7a  69 70 0a 50 4b 01 02 1e  |n/epub+zip.PK...|
00040  03 0a 00 00 00 00 00 29  a3 b4 5a 3b d1 f6 ef 15  |.......)..Z;....|

The gap is gone, and the signature should be in the first 0x50 bytes. Sure enough, file agrees:

test.epub: EPUB document

Furthermore, Zathura renders the document appropriately. The file is fixed!

Now we can write a short shell script to automate this procedure for a file:

#!/bin/sh

DIR=$(mktemp -d)
TMPOUT=$(mktemp -d)
trap 'rm -rf "$DIR" "$TMPOUT"' EXIT

unzip -d "$DIR" "$1"

if [ -f "$DIR"/mimetype ]
then
    (cd "$DIR" && zip -X -0 "$TMPOUT"/out.zip mimetype)
    rm -f "$DIR"/mimetype
fi

(cd "$DIR" && zip -X -9 -r "$TMPOUT"/out.zip .)
mv "$TMPOUT"/out.zip "$1"

This script operates on the file in place, so you may want to back it up beforehand. We make some temporary directories and extract the EPUB to one of them. Next, we add the mimetype file to an output zip without compression or headers. Then we add the rest of the files with maximum compression and no headers. Finally, we move this output file back where we started, and delete the temporary directories.

To conclude the story, I used this script to fix up every malformed EPUB I had, and now I can use them again!