LumoSQL ↗ protects data on mobile phones using a new data storage technology which is highly compatible with most existing devices. With LumoSQL, the device owner decides who can read or change their data down to the level of individual rows if they choose. This decision continues to be enforced even after it has been copied off the phone to (for example) a bank or dating or insurance company for processing with their in-house database software. Today, device owners are rarely in control of the privacy of their own data, despite the many privacy laws.

If someone takes a phone away from its owner, LumoSQL data rows cannot be read without the consent of either the phone owner or someone to whom the phone owner has granted access even if the phone has been unlocked. These controls are fine-grained, meaning different levels of permission can be granted.

The mobile data storage problem

A mobile phone stores all its non-streaming data (contacts, messages, health records, browsing history, app settings) in hundreds of SQLite ↗ databases. SQLite is also found in web browsers, operating systems and vehicles, and it is a standard data format ↗ relied on by companies such as Airbus whose products last decades. Therefore SQLite is likely the most-deployed software in the world, and probably the only trillion-scale software in existence.

And yet SQLite stores all of this data in plain text. Whole-device encryption does not address the problem, because once the phone is unlocked, any process that can open a database file can read every row in it. It is unacceptable to store personal data this way, but SQLite works so well and is so pervasively ubiquitous that it is hard to imagine how an alternative could be implemented. There are forks of SQLite that can do older page-based encryption and this is what banking apps use, but none at a comparable scale.

The word unacceptable here is a legal judgement as well as a professional opinion. Article 7 ↗ of the EU Charter of Fundamental Rights protects an individual’s private life and communications, and Article 8 ↗ grants the right to protection of personal data. The GDPR and the national laws beneath it are built on those two articles. LumoSQL’s claim is that it can meet the mandatory privacy and security requirements of that legislation while SQLite cannot, because software holding personal data in plain text has no way to protect it.

LumoSQL does not try to be an alternative, because it modifies SQLite (rather than replacing it) so that existing apps continue to work without change, but the data they store gains protection. At-rest encryption of the whole database file works today, so that even an unlocked phone cannot open a database unless the user gives a key. The larger idea is a Lumion, a row of data that carries its own rules with it, like a letter in a locked envelope where only the people named on the envelope can open it, even after it’s been forwarded. A Lumion still enforces its owner’s permissions no matter where it goes by email or in the cloud, or copied to a corporate database or some other device. The phone owner decides who can read or change each piece of data, or at least set default policies.

The detailed strategy document LumoSQLMotivation-1.0.pdf explains the social, business and technical pressures behind LumoSQL. As the strategy document says, “LumoSQL assumes software development should never be relied on and is getting worse”.

Surprising facts about SQLite

  • SQLite is a full-featured database, supporting the standard SQL language despite being tiny compared to all the other mainstream SQL databases.
  • SQLite is open source, exceptionally well-maintained, mostly by just 3 people. Many more people contribute occasionally to SQLite, and the community of deeply technical users is very large.
  • SQLite is exceptionally reliable, given the policy decisions not to change certain fundamentals, and despite frequently adding new features and support for the latest SQL standard.

The corollaries to these unusual facts are significant:

  • This is uncharted territory for Computer Science: is SQLite’s ultra-conservative compatibility commitment to its (at least) hundreds of billions of installations the right choice? Is SQLite’s fast-moving support of formal database standards the best way forward? It does certainly work very well.
  • Of the many forks of SQLite, none seem to have more than a relatively trivial deployed base when compared with SQLite. This includes forks with useful features (such as SQLCipher), so it appears the nature of the SQLite project ensures its success and discourages replacements. This does constrains its future in important ways too, especially that it cannot introduce encryption.
  • These obvious strategic problems with SQLite around privacy do not seem widely discussed. But why? Perhaps LumoSQL will help make these discussions more common.

Lumions

If combined even to some extent with the immense scale of SQLite, a new concept for encryption is potentially transformational for all device users. Lumions are just such a new concept, being a row of data that carries its own rules within it.

Wherever a Lumion goes, any user can prove what the data is, who wrote it, and who is allowed to read or change each part of it, without contacting a server or an authority. That is the opposite of the usual arrangement, where the rules live in a program: one developer’s code decides that you may not read someone’s salary, and if the code is wrong, or doesn’t know the salary is your salary, there is nobody to argue with.

A Lumion takes a plain row of information (think of a line in a CSV file) and adds encryption, a global name, selective disclosure of individual fields, and a way for one row to point at another. It is closed under copy: everything it needs to work travels inside it, so it keeps working after being copied to the cloud, to another device, or into someone else’s database. Lumions wouldn’t be needed if JSON already had these ideas, and we have considered the prior art carefully.

Two draft RFCs ↗ are complete: the Lumion object itself, and its byte encoding. Between them they contain everything needed to write a conformant implementation. A Python demo shows how to build and use Lumions in a plain text file from the commandline.

Funding and contributors

LumoSQL exists due to the volunteer effort contributed by many skilled people. The Vrije Universiteit Brussel ↗ funded valuable contributions in cryptography and mathematical analysis. The NLnet Foundation ↗ supported LumoSQL from 2020 to 2022. The pandemic years took their toll and the project was dormant from the end of 2022 until 2026. The timing of the reboot turned out well, because Howard Chu was readying LMDBv1.0 for release and LumoSQL inherits a great deal from it. The project history ↗ names everyone.

Information for developers

For much more detail see the code development page ↗.

In technical summary: LumoSQL is a modification (not a fork) of the SQLite embedded data storage library. v0.85 is a preview release for curious technical developers wondering why anyone would want to touch trillion-scale software. Production use is not yet recommended, although we are fairly confident LumoSQL will not eat your data. If you want encryption, simply say make OPTION_LMDBV1_ENCRYPT=on.

You can compile LumoSQL against applications that already use SQLite, because from the application’s point of view it is libsqlite3. The tree maintains working examples for Fossil and libfossil ↗, rpm, and the better-sqlite3 Node.js binding ↗. Fossil is the demanding case: it is developed alongside SQLite and often needs SQLite versions ahead of the latest release, so LumoSQL follows the tip of SQLite’s development branch to build against it. If LumoSQL can run Fossil, there is a good chance it can run millions of other applications.

LumoSQL offers multiple key-value backend storage systems selectable by the user, and features not found in any other mainstream database:

  • ability to checksum every row on write and verify on read, stored in an ordinary sqlite3 file that any standard SQLite binary can still read and write
  • ability to trigger arbitrary functions on per-row read and write
  • a general test suite for benchmarking precisely how LumoSQL (or SQLite) is performing and the full context of that benchmark run. Results land in an SQLite database and can be merged between users, so runs on different hardware and different architectures can be compared. Database benchmarking is very poorly done, including by the TPC, founded in 1988 for that purpose.
  • a general build system able to mix and match multiple versions of the database with multiple versions of multiple backends. Of the order of a thousand binaries can be built from a decade of SQLite and LMDB releases, and the test suite includes cross-binary interoperability checks. Never before has it been possible to compare the different strategies of various Key-Value stores with the same database frontend.

Notable outcomes from LumoSQL already include:

  • The only mainstream database with swappable Key-Value stores, where all stores are peers rather than one store having special knowledge that gives it technical advantages. Three ship today: the existing SQLite btree with optional binary-compatible modifications such as rowsums; LMDB 0.9.x ↗, long stable and now in maintenance; and LMDBv1.0, which brings page-based encryption, page checksums and incremental backup, all of which LumoSQL inherits without further work. We look forward to integrating other stores, and are interested in 21st-century designs such as Adaptive Radix Trees.
  • At-rest encryption for an ordinary SQLite application, open source.
  • The only mainstream database optionally without a Write-ahead Log ↗, in the case where the user chooses an LMDB backend.
  • Benchmarking results ↗ comparing the SQLite btree, LMDB 0.9.x and LMDBv1.0 backends: 900 runs at SQLite 3.53.1 across ten data sizes, with and without encryption, checksums and rowsums.
  • The Not Forking tool, which avoids forks in both simple and complicated source code, and which is what makes all of the above possible without carrying the burden of forking SQLite.

Exciting things in progress:

  • Internet draft-style RFC for row-level encryption.
  • Documented API for arbitrary key-value stores
  • Documented API for accessing the key-value stores via the SQLite library, instantly making the SQLite key-value store the most widely-distributed key-value store. Nothing calls the SQLite key-value store today except SQLite