Tsonnet #47 - The devil in the details #3

Welcome to the Tsonnet series!

If you’re not following along, check out how it all started in the first post of the series.

In the previous post, we replaced the proactive cycle-checking AST walk with on-demand detection during translation:

On-demand caught simple cycles during translation, but lazy types in arrays, function defaults, and object fields could still hide cycles until interpretation. I needed to manifest every type fully during type checking.

The on-demand pattern from post #46 caught variable cycles via TranslatingVar and field cycles via TranslatingObjField. But recursive function calls were a blind spot — when a function body references another function that hasn’t finished translating, the cycle goes undetected.

I added a TranslatingFunction key:

And wrapped the function body translation in with_translating:

This catches local f() = f() — the body translation fires TranslatingFunction f before starting, and if f() in the body triggers the same key, with_translating raises the cycle error.

The same logic applies to closure calls — local f = function() f() now triggers TranslatingFunction f around the closure body.

While I was at it, I fixed collect_free_idents to exclude bound names from function and closure bodies. A function’s own name and its parameter names shouldn’t count as free variables in the body:

The new samples cover the full matrix of recursive call scenarios:

• invalid_recursive_function_call.jsonnet — local f() = f(); f()

• invalid_recursive_closure_call.jsonnet — local f = function() f(); f()

• invalid_mutual_recursive_function_call.jsonnet — local f() = g(); local g() = f(); f()

• invalid_mutual_recursive_closure_call.jsonnet — local f = function() g(); local g = function() f(); f()

• valid_closure_param_shadowing_unused_outer.jsonnet — local shadowing doesn’t trigger false positives

• valid_function_body_uses_outer_local.jsonnet — function body referencing an outer local is fine

The on-demand approach — wrapping each lazy translation in with_translating — works well for bindings hit during translation. But array elements, object fields, and function default parameters are stored as Lazy expr nodes in the type. Translation never visits them until they’re actually accessed. If two lazy nodes reference each other through an intermediary, the cycle passes the type checker silently and blows up at interpretation time.

Consider:

{ a: self }

With the on-demand pattern alone, the object field a is stored as Lazy (ObjectFieldAccess ...). Translation of the object doesn’t resolve self.a — that only happens when the field is accessed. So { a: self } used to type-check successfully and fail at manifestation.

I needed a post-processing step that recursively resolves every Lazy, LazyIn, and LazyDefault wrapper in the type tree, catching cycles along the way.

The three lazy wrappers each need different handling:

• Lazy — translate in the current environment (the default for most lazy bindings)

• LazyIn — translate in the environment captured at array construction time (array elements should resolve against the scope where the array was defined, not where it’s accessed)

• LazyDefault — translate in an environment where sibling parameters are also lazy and the current parameter is shadowed out (so f(x = y, y = 1) resolves y in the default for x)

For Tarray, I went from a single element type to an element-level list. This lets deep_translate_type resolve each element independently:

And translate_array now wraps each element in LazyIn with the current environment:

translate_object needed two changes. First, it now builds a field list alongside the object environment, so deep_translate_type knows which fields to visit:

Second, TobjectPtr now carries an optional captured environment for resolving self and $ during deep translation. When deep_translate_type encounters a TobjectPtr, it raises a cycle error — manifesting self or $ in the root type means the top-level expression references the object itself, which can’t be manifested.

Since every lazy type is now resolved during type checking, the interpreter no longer needs its own cycle detection. The entire evaluating_bindings infrastructure — about 150 lines across interpret_ident, interpret_object_field_access, interpret_runtime_object_fields, and interpret_seq — came out:

I removed the ObjectFields.mem checks from every interpreter path. The type checker now guarantees that by the time the interpreter sees a type, all cycles have been detected. The interpreter can focus on evaluation.

The top-level check function now runs deep_translate_type after translation:

This single extra line is the architectural change. Translation produces the type; deep_translate_type walks the result and blows up on any leftover lazy reference that would cycle.

The old translate_ident had a manual TranslationKeys.mem check before Env.find_var. Since with_translating inside the Lazy branch already handles that check, the early guard was redundant:

The type checker now stores function defaults as LazyDefault in the AST — a new variant that carries the outer environment, sibling parameters, and the default expression:

The interpreter evaluates a LazyDefault by building an environment where sibling params are also lazy:

add_default_params_to_env skips the current parameter (to prevent trivial x = x loops) and wraps sibling defaults in LazyDefault:

apply_function stopped resolving defaults eagerly at call time. It used to fold each default straight into the environment and wrap the body interpretation in with_fresh_evaluating_bindings. Now it keeps the raw bindings, wraps each default in LazyDefault, and lets the interpreter resolve them on demand in the body:

The with_fresh_evaluating_bindings call and the whole evaluating_bindings infrastructure went away with it. The interpreter stays lazy, but it no longer has to watch for cyclic references while it evaluates — the type checker already caught them during deep translation.

The scope checker and type checker handle LazyDefault as a pass-through — it’s a runtime construct that doesn’t need source-level validation.

The cram test diffs show three categories of change.

Error positions moved. The old runtime-based cycle detection pointed at object field IDs; the new manifest-based detection points at the actual source location:

New error cases. Self-referencing object manifestation now produces proper errors:

  $ tsonnet ../../samples/semantics/invalid_manifest_self.jsonnet
  ERROR: .../invalid_manifest_self.jsonnet:1:5 Cyclic reference found for 1->self
  1: { a: self }
      ^^^^^^^^^
  [1]

And $, the top-level reference:

  $ tsonnet ../../samples/semantics/invalid_manifest_toplevel.jsonnet
  ERROR: .../invalid_manifest_toplevel.jsonnet:1:5 Cyclic reference found for 1->self
  1: { a: $ }
     ^^^^^^^
  [1]

Both self and $ still report 1->self as the key. That’s because TobjectPtr doesn’t distinguish the two scopes when raising the cycle error. The key encodes the object ID, not the scope. Something to clean up later.

Function default cycles now produce type errors instead of runtime errors. The old error for local a = [a]; local f(x = a) = x; f() was “Invalid binary operation” — the cycle leaked past the type checker and manifested as a runtime crash in the + operator that happened to be nearby. Now:

  $ tsonnet ../../samples/semantics/invalid_function_default_cycle.jsonnet
  ERROR: .../invalid_function_default_cycle.jsonnet:1:11 Cyclic reference found for a
  1: local a = [a];
      ^^^^^^^^^^^^^
  [1]

Passing a non-cyclic argument bypasses the cyclic default entirely:

$ tsonnet ../../samples/semantics/valid_function_provided_arg_ignores_cyclic_default.jsonnet
2

local f(x = x) = x; f(2) — the provided argument shadows the self-referencing default. Lazy evaluation means the default is never evaluated, so no cycle error. Correct.

$ dune exec -- tsonnet samples/semantics/invalid_manifest_self.jsonnet
ERROR: samples/semantics/invalid_manifest_self.jsonnet:1:5 Cyclic reference found for 1->self
  1: { a: self }
     ^^^^^^^^^
  [1]
$ dune exec -- tsonnet samples/semantics/valid_function_provided_arg_ignores_cyclic_default.jsonnet
2
$ dune exec -- tsonnet samples/semantics/invalid_function_default_mutual_cycle.jsonnet
ERROR: samples/semantics/invalid_function_default_mutual_cycle.jsonnet:1:19 Cyclic reference found for x
  1: local f(x = y, y = x) = x;
     ^^^^^^^^^^^^^^^^^^^^^
  [1]
$ dune exec -- tsonnet samples/semantics/valid_function_default_later_param.jsonnet
1
$ dune exec -- tsonnet samples/semantics/valid_function_default_outer_shadow.jsonnet
1
$ dune exec -- tsonnet samples/semantics/valid_function_body_uses_outer_local.jsonnet
1

Every lazy type is now resolved during type checking. No cycle slips through to the interpreter.

with_translating catches a re-entered binding during translation; deep_translate_type forces the lazy types translation never reached — arrays, object fields, function defaults — so those cycles surface too. Both live in the type checker; the interpreter does no cycle detection anymore.

The entire diff can be seen here.