Reading · Marco Campos' Blog

Yet another time I stand in front of a problem that needs to read files from the user device. Furthermore, read and keep a reference to files from the user device.

There are two parts to this story, one with broad browser support, and another that is Chrome Only™.

The problem

I was working on an application that two things: a directory with images, and an SQLite database with metadata about those images.
The application viewing logic itself was simple, it queried the database for some properties and then displayed images accordingly.

Then there was the application editing logic, that one, was not so simple. In short, a user could crop the images and update the metadata on the database.
On the surface, it is all nice and doable from a browser, thanks in part to the amazing sql.js (External link) library that allows to query an SQLite database and avoid the wonders of IndexedDB (External link) ¹.

But then… How do I export the database and the files? Specifically, how do I export the files without asking for the user to download them every single time? Or rather, how to export a whole folder without forcing the user to click to download hundreds of individual files?

File System API and OPFS

The good part is all modern browsers support the File System API (External link) and Origin Private File Systems (OPFS) (External link) .

Those APIs provide two things:

The File System API: An abstraction over files, directories, and handles for those.
OPFS: A directory that the application has access to and can use however it sees fit.²

The OPFS is easy, you call one method and it gives you what you need:

const opfs = await navigator.storage.getDirectory();

Then you use the File System API to read and write files and directories.
Simple… In theory…

No paths, only handles

The first “oddity” for a web developer is that you don’t deal with paths to things, you deal with handles. Similar to using file descriptors for everything when using node.js or doing low level Linux things.

It is indeed a good idea to use handles instead of paths for security reasons. In short, if the file is moved or something else happens, the path may not match the file anymore³. With a path you can’t guarantee the file is what you think it is, with a file descriptor you still hold a reference to the original file itself.

That is the crux of path traversal attacks (External link) and path normalization issues (External link) .

But then, if there are no paths to follow between two things, how do you handle nested files and directories when working with the directory tree?

Recreating basic file system utilities

Now that is the painful part… You have to recreate some utilities like mkdir or rm yourself. As well as some path to handle resolutions…

One interesting method is FilesystemDirectoryHandle.resolve() (External link) . If you pass another handle to it, the method resolves a list of the intermediary steps to that handle. It covers the case where the handle is a descendant to the directory.
But… if we start resolving from the root of the file system, we can basically move anywhere as everything is a descendant of the file system root⁴.

That gives the tools needed to recreate some tools like creating directories and files recursively.

But that leaves some other utility functions that don’t directly affect the file system: Path utilities.

Node `path` module as a start

Although re-creating those functions can be tedious, the node.js path module (External link) is a good inspiration. And as it is inspired by POSIX (External link) , so there are lots of implementations out there in LLMs training data.
Yeah, I basically prompted a recreation of functions like basename and dirname.

Then just added a little bit of glue code to use those functions to convert between strings and FileSystemHandles.

Unfortunately, nothing is ever easy…

Naming things is hell in a handbasket

That is the moment when I was hit in the face by one of the best problems in computer science: naming things.

Specifically, what is and is not allowed in the names of files (External link) . Unfortunately for us, the specification leaves it open for interpretation… How I love those “undefined behaviours (External link) ”… 🫠

Anyways, the very basics are covered by the spec:

File name can’t be an empty string
File name can’t be “.” or “..”
File name can be a path separator (“/” or “\”)

Anything else you have to try and see if it throws a TypeError to find out.

Luckily, the code I had was already splitting paths by forward slashes (/) because I’m lazy and it is used everywhere but Windows⁵. So that solves one part of the problem.

Checking for empty strings, and the “.” and “..” patterns is easy as those are exact strings to check.

The problem then becomes what is covered by spec but not specified:

(…) Additionally underlying file systems might have further restrictions on what names are or aren’t allowed (…)

From my testing, the only weird things are strings starting in null bytes. But I’ve also wrote some helper functions to handle weird file names as not every OS supports all names, so it becomes important later.

It’s all helper functions…

All those things considered, and a couple hundred lines of helper functions written it ends up being very simple to manipulate your own app private and handy directory.

Now getting out of it is more interesting…

File System Access and Chrome

To read or write to the user’s file system, outside of the browser sandbox, we do have the File System Access API. Unfortunately it is Chrome only⁶

It is composed of the following methods on the window object:

showOpenfilePicker (External link) to get a file handle for reading, and potentially writing.
showSaveFilePicker (External link) to get a file handle to write on.
showDirectoryPicker (External link) for get a directory handle for reading, and potentially writing.
DataTransferItem.getAsFileSystemHandle (External link) to integrate with drag and drop.

All those are very easy to use and intuitive in what they do. And as with this entire API surface, if gives you a handle to the file or directory.

With that you can do pretty much the same operations you can do on the OPFS.

But you need permissions for that.

You need permissions

As you are escaping the browser sandbox with this API, you need permissions to do things with it. Similar to notifications or geolocation.
The permissions are checked like so:

const existingPermissions = await handle.queryPermission({ mode });
const newPermissions = await handle.requestPermission({ mode });

The first method, queryPermission, checks the existing permissions you have for the handle, and the second one requestPermission prompts the user for the permission.

You need to call this if the type of permission you need changes, for example, if you open a file for reading and now needs to write to that file.
Or when you start a new browser session, you need to re-request permissions for in case things have changed between sessions like a file deleted or moved.

You need permissions 2: Button Boogaloo

The “problem” with the permissions API is that is requires user activation (External link) . It means the user has to click, touch the screen, or type any key before you can ask for permissions.
Security sometimes gets in the way of ease of use, this is one of the cases and we have to live with that.

Persisting access

If all of that seems too much work when you can simply add a file input and the user opens it, that’s because it is. The main difference is browser support and API power.

With file inputs you can’t:

Write back to the files.
Create files in a directory.
Delete files and directories.
Open a file picker without an input.
Persist access to handles.
Persist permissions for handles.

When you need to keep access to a handle between browsing sessions, you can do so by saving the handle on IndexedDB.

But nothing is easy, so IndexedDB is an old API and doesn’t understand promises. It is fairly straightforward to use though. Oh well…

Error handling

One final detail I glossed over for all these operations is: you need to handle errors.
Here are the most relevant ones:

TypeError: if the file or directory name has invalid characters.
NotAllowedError: when you don’t have permissions for that handle.
NotFoundError: If the entry the handle refers to does not exist. Like a deleted or non existent file.

So… IDB can be a pain to deal with, and working with that meant converting between formats from and to SQLite anyways. Using a library that allows to query and manipulate an SQLite database in the browser made it easier to work with. Back to reference 1
There is a quotas and there are other details like eviction (External link) to take care of, but that is beyond the scope of this post due to extra complexity. Back to reference 2
One type of attack is to replace the underlying file if a program only verifies the paths, so it will happily execute a malicious file. Back to reference 3
A little foreshadowing: when you get a handle to a directory on the user’s computer, outside of the OPFS, it can be considered a file system root. Back to reference 4
Fun fact: Windows canonically uses backslashes (\) for paths, but in modern versions, it will accept forward slashes (/) and convert them in some applications. So, yeah, they are mostly interchangeable and Windows won’t complain much if you use one or the other. Back to reference 5
All the other browser vendors oppose to implementing this API, so yeah, this is the the really bad news. Back to reference 6

The File System and You