Prompt Brewery: Escaping the Jinja Hell

Not so long ago, I wrote about what's wrong with chat-templates format for LLMs. Among other things, I mentioned what I called "jinja hell" — debugging issues with jinja-based prompt templates. Jinja notation has emerged as the de facto standard for templating LLM prompts. However, its flexibility comes at a cost: it's notoriously difficult to debug, test, and maintain. Things get even worse, combined with multiple rendering engines without cross-compatibility and unpredictable user inputs — you never know on which input your template will crash.

Prompt Brewery

We at Inworld (powering millions of daily sessions) spent way too much time hunting down and debugging errors in our prompt templates. So eventually, we had to come up with Prompt Brewery — a tool that helps with both automatic testing and interactive debugging.

So today we're publishing it as an open-source repository with code for static analysis of jinja prompt templates, automatic generation of minimal payload, and detection of common problems. The minimal usage pattern is straightforward — one function call for the template you want to check returns errors, warnings, and recommendations, so these checks are easy to integrate into tests.

We also provide an interactive tool online that allows you to edit and check your template on the fly.

So, does it work?

Static analysis cannot guarantee all cases, so we tested this tool on:

• 400+ real templates we collected (different use-cases, LLM evaluation templates, open-sourced templates);

• Another 400+ generated synthetic tests to cover edge cases;

However, if your template doesn't work, please report it as an issue so that we can add support for such cases.

Looking back at all the hours we've spent debugging template failures in production, I wish we had built this tool earlier. So, now, at least you can safe some time.

Have fun debugging, and please provide your feedback!

The repository is open-source, and we're always looking for more real-world templates to test against.