GSOC 2026 Work Product

Introduction

This summer I again worked on the loo package as part of the Google Summer of Code 2026. The loo package is widely used to cross-validate Bayesian models and has over three million downloads and several thousand citations to date. This project is under Stan⁠, which is sponsored by the NumFOCUS⁠ organization. This year I worked on parallelization in loo, focusing on unifying our parallelization by using mirai. Of course, I also did a number of smaller side quests over the summer. This year I wrote a proper report as part of the parallelization work, and I will leave the details to that report.

Of course, I would like to thank my mentors, Dr. Aki Vehtari, Dr. Florence Bockting, and Jonah Gabry for their patience and time. Just like last year, I am very grateful that they take the time to discuss and explain things to me. Just like last year, none of my parallelization work would have been possible without their support and guidance. Getting to meet and work with my mentors has once again been one of the highlights of my GSOC experience.

What Did I Do?

Parallelization

As mentioned, I worked on a report on parallelization in loo which is far more detailed and can be found here⁠. The report is somewhat self contained, in that you don’t need to know much about parallelization or even loo to understand what I have worked on. Instead of copying it here in full, or summarizing sections, please find just the introduction of the report below. I encourage you to read the report in full.

Currently loo relies on the parallel package from base R to parallelize certain computations, such as moment matching, Pareto smoothed importance sampling, and more. This works well on Unix systems, where parallel can use forking, but Windows requires a separate, socket-based path. This project will replace loo’s current parallelization structure, swapping to a platform-agnostic approach using mirai. This will have the added benefit of allowing users to parallelize execution over heterogeneous compute resources (e.g., over local cores, a distributed system, GPUs, or any mixture). This freedom decouples execution profiles from loo’s software, allowing users with varying compute needs to exploit their available resources without any extra work. Further, a major part of the project here would be writing strong documentation which teaches users how to make use of some common resources such as SLURM or SSH access, building off of the documentation in packages like mirai, but with a focus on applications to loo.

This report is a survey of where loo stands before any of that work lands. Section 2 lays out the four maps loo could dispatch work through and what each one costs. The next section covers why we picked mirai, what we pay for using it in place of lapply() when a user asks for serial execution (Section 3.1), as well as what we would pay for going through future rather than using mirai directly. Section 4 covers why memory is a concern when parallelizing, and whether mori can help (Section 4.3). The remaining sections are an audit of the package itself: which functions parallelize today (Section 5), where we could speed up code using parallelization (or vectorization) (Section 6), and where loo already has to guard against nesting one parallel call inside another (Section 7.1).

Assorted PRs

I made a few chore PRs like bumping Roxygen versions (which would be nice to have as an automated workflow, like Dependabot, but I haven’t thought too hard about that), as well as adding .gitattributes into more repositories to solve line ending issues. Codecov also had an issue⁠ which caused our coverage actions to fail. I happened to notice it, so I took care of that in a few repos as well.

I’ve also been working on this PR⁠ from last year to rely more on posterior; I had an initial solution which wasn’t performant, so as Aki suggested I wrote a new function just for loo. That led to a few changes in posterior as well, including a specialized path for already-sorted log ratios and some work attempting to speedup the Pareto smoothing code (#452⁠, #453⁠, #454⁠).

I also made a somewhat big mistake in loo! Like I just mentioned, we have been trying to deduplicate code across loo and posterior—ensuring the former uses the API provided by the latter instead of maintaining two copies of the same code. I made a seemingly innocuous PR earlier this year to do just that, but it turned out that posterior and loo had slightly different semantics which I didn’t check or catch. As such, loo 2.10 had an inadvertent breaking change which was fixed in 2.10.1 by #381⁠. We’ve implemented stricter merge criteria after my little fiasco.

I also spent some time on #382⁠ trying to move our reverse dependency checks onto GitHub Actions so they wouldn’t tie up someone’s machine and could be run more easily. My approach was to neatly wrap revdepcheck⁠, but my runs were taking extremely long so I had to do some recursive jobs which ended up taking double digit hours. My approach relied on revdepcheck’s own resume functionality to continue work from the previous runs. Andrew Johnson was working on a cleaner approach for rstan at the same time, which is now being generalized in ci-scripts (#42⁠), which is a bit more manual but seems to work better, so we will probably end up using that.

A little bit ago I built the Stan pkgdown theme⁠ that all of the R packages have been adopting, and I recently⁠ (finally) got brms on it too. However, someone noted that the function reference page disappeared⁠, which I finally tracked down to an upstream pkgdown bug which I opened a PR⁠ for, and patched in the brms builds.

foil

We currently measure PR performance using touchstone which compares a branch against main to catch performance regressions. touchstone is pretty cool, but it can’t do everything we need from it, and isn’t as actively developed anymore. I’ve made some small changes to our touchstone setup before, across stan-dev—primarily updating the same comment instead of posting new ones every run—but the more I was using touchstone and its results the more I wanted to change things¹, so I decided to build out my own tool, foil⁠².

Like touchstone, foil runs both revisions on the same machine and randomizes their order, but foil keeps those measurements paired in the analysis and accounts for drift and run order when estimating the revision effect. foil uses a Bayesian bootstrap to get uncertainty quantification without specifying an error structure which may not be appropriate for benchmark times. It also works with arbitrary commands rather than only R expressions. The big piece still missing is memory measurement, which I am planning to pursue through an ISC grant so foil can catch regressions in compiled code and child processes as well. foil is designed to be very general, though I will be building out first-class support for R, which will be helpful for measuring the actual impact of these parallelization changes.

Unsurprisingly, once foil is a bit more stable, I will be advocating for adopting it in a few Stan repos.

What Did I Learn?

I learned a lot about parallelization and implementations in R. I learned about mirai last year at posit::conf and used it to parallelize my tidymodels workflow but I never looked deeply into how it worked or what alternatives there were. I learned more about the importance of a thorough test suite and code reviews, the hard way. I also got a much better appreciation for careful benchmarking, which directly aided my foil work.

What is Left?

We need to discuss and finalize the new loo parallelization API, and once it is settled, the implementation itself should be very straightforward. We will then have to do some testing, maybe with a release candidate, to see how the transition goes. Further, I need to finish foil—I’m planning on submitting it for an ISC grant⁠ to fund development, primarily for memory measurement.