Tekne is an outline editor I’ve been working on on and off for a while now.
Although it’s not pure Markdown like Obsidian, it supports Markdown with extensions for formatting and convenience. A Tekne document is fundamentally stored as a JSON array of text lines, like this:
{"type": "line", indent: 0, mdContent: "#project"}
{"type": "line", indent: 1, mdContent:
"Worked on [[Subproject]] today"}In this example, we’re working on #project and have linked out to a Subproject
document. What this means is that there are little bits of information we care about
interacting with programmatically buried in snippets of text.
For example, we might like to know which documents link to a document we’re viewing. Or we might want to rename that document and update all links to the document to reflect the rename. But this is hard to do when the data is fundamentally a pile of strings that (it is hoped) follow some formatting convention.
A couple of weeks ago, I was trying to add support for this and it turned out to be quite hairy. I hacked it together with regular expressions, but it didn’t work very well.
What we really need here is a way to interact with Markdown programmatically. I was
already using the @lezer/markdown package to
support rendering Markdown along with my extensions. With Lezer, we can turn a string into
a tree, for example, Hello [[MyPage]] becomes:
{
"from": 0,
"text": "hello [[MyPage]]",
"to": 16,
"type": "Document",
"children": [
{
"from": 0,
"text": "hello [[MyPage]]",
"to": 16,
"type": "Paragraph",
"children": [
{
"from": 6,
"text": "[[MyPage]]",
"to": 16,
"type": "InternalLink",
"children": [
{
"children": [],
"from": 8,
"text": "MyPage",
"to": 14,
"type": "InternalLinkBody",
},
],
},
],
},
]
}So now we have a JSON object we can robustly query to find information about links. We can run this parser every time a document is updated and store it in the database.
Except querying it is still potentially pretty awkward: a link to a document could come from anywhere. Do we want to load every document into memory and then dive into the JSON document to check for internal links?
There’s some efficient ways to solve the problem, but given that the amount of data in any Tekne install is still relatively small (~hundreds of documents), what I wanted was something that would have pretty low maintenance/LoC burden and flex to solving similarly shaped issues (for example, renaming tags).
Postgres has some built in JSON querying functionality, and it turns out you can make pretty concise queries for this:
SELECT
title
FROM
notes
WHERE
jsonb_path_exists(parsed_body, '$.** ?
(@.type == "InternalLinkBody" &&
@.text == $v)',
jsonb_build_object('v',
to_jsonb(cast("Document name" as text)))) Where .** recursively descends the entire tree, ? ... filters on a condition, and then
we check for type=InternalLinkBody and text="Document name". The json_build_object
builds an intermediary object simply for passing the document name into the JSON path
expression.
Of course we’re not done quite yet; the source of truth is still a pile of arbitrary text, which we need to update reliably.
const parsedContent =
TEKNE_MD_PARSER.parse(ln.mdContent);
const newMdContent = new MagicString(ln.mdContent);
visitMdTree(
parsedContent.topNode,
'',
0,
(node: SyntaxNode) => {
const txt =
ln.mdContent.slice(node.from, node.to);
if (
node.type.name === 'InternalLinkBody' &&
txt === oldName
) {
newMdContent.update(
node.from,
node.to,
newName
);
}
})The visitor pattern works well here, and combined with magic-string we can update the original text without needing to track anything about how our changes are altering it.
Although it only nets out to being maybe a hundred lines of code, it was quite a pain to figure out how to do this well. But it does feel like the right set of abstractions for dealing with Markdown. It’s farther along on the backlog, but I’d like to do the same sort of thing with my Obsidian vault to be able to query some stats about it.