Tsonnet #37 - Call me maybe, but make it typed, part 3

Welcome to the Tsonnet series!

If you’re not following along, check out how it all started in the first post of the series.

In the previous post, we added default function arguments:

Now let’s implement closures.

We should accept a closure that is going to be assigned to a local, and immediately called closures:

// samples/functions/closure.jsonnet
// binding closure
local id = function(x) x;
// immediately called closure
(function(x) id(x) * id(x))(5)

We need two new AST variants:

Then we update the lexer to recognise the new keyword:

And the parser:

I stumbled upon a shift-reduce issue trying to make closure part of the assignable_expr parsing rule — it would make more semantic sense there, but the grammar conflicts beat me. I’ll come back to fix it later.

Here’s the short version of why it’s not feasible as-is: if closure lives in assignable_expr, then (function(x) x) becomes reachable via two paths simultaneously — as a parenthesised closure via scoped_expr, and as the first half of a closure_call. When the parser sees ( function(x) body . ) and hits ), it can’t decide whether to reduce into scoped_expr or shift ) as part of closure_call — that would require two tokens of lookahead, which LR(1) doesn’t have. The only clean fix would be generalising funcall to accept any expression as the callee, so (function(x) x)(5) parses naturally as “call this expression with these args” — but that’s a bigger refactor touching the AST, type checker, and interpreter.

Here’s how the new closure and closure_call rules are implemented for now:

The new type variants mirror the ones in ast.ml:

The collect_free_idents function needs updating for the new variants — I also noticed I’d forgotten FunctionCall here earlier (oops!):

Then we pattern-match on the new AST nodes in translate:

I also added a catch-all for Tany in binary operations — if either side is Tany, we let it through rather than erroring:

translate_function_call also needs to handle Tclosure — if a bound name resolves to a closure, we re-dispatch to translate_closure_call:

translate_closure is pretty similar to translate_function_def:

And translate_closure_call is pretty similar to translate_function_call:

There’s room for simplification here, but let’s move on for now.

It’s worth pausing on a behaviour difference between Tsonnet and standard Jsonnet. Given this sample:

Jsonnet fails at runtime after evaluating the first call, because && expects a boolean and gets a number first:

$ jsonnet samples/semantics/invalid_function_call_type.jsonnet
RUNTIME ERROR: Unexpected type number, expected boolean
	samples/semantics/invalid_function_call_type.jsonnet:2:1-38	$
	During evaluation

Tsonnet instead type-checks the function calls before even getting to the &&, and catches the type mismatch earlier:

$ dune exec -- tsonnet samples/semantics/invalid_function_call_type.jsonnet
ERROR: samples/semantics/invalid_function_call_type.jsonnet:2:18 Expected type Number, got String
2: my_function(3) && my_function("oops")
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I think that’s a more sensible error. Worth noting: I want to support polymorphic parameters and return types eventually, but for this prototype, this is enough.

During interpretation, a closure definition is returned as-is — it just gets stored in the environment. The actual work happens at call time:

The function application logic was straightforward to generalise into apply_function, which both interpret_function_call and interpret_closure_call now share:

interpret_function_call and interpret_closure_call now just delegate to it:

I also used this opportunity to fix a leftover hard-coded error string I’d missed during an earlier refactoring:

The rename also moves the message out of the type_-prefixed block, since both the type checker and the interpreter now share it.

$ dune exec -- tsonnet samples/functions/closure.jsonnet
25

And the cram test:

Closures are in. Binding them to locals, calling them immediately, passing them around — it all works. The type checker was the trickier part, requiring extra plumbing to re-dispatch when a bound name turns out to be a closure, but the interpreter side practically wrote itself once apply_function was factored out. The shift-reduce issue in the parser is still there, nagging at me from the grammar file — but that’s a problem for future me, probably involving a refactor to let funcall accept any expression as callee.

Here is the entire diff.

Next up, I think this can be simplified, and I will do so.