GSOC Work Product

Introduction

This summer I worked on the loo package as part of the Google Summer of Code 2025. The loo package is widely used to cross-validate Bayesian models and has over three million downloads and several thousand citations to date. This project is under Stan⁠, which is sponsored by the NumFocus⁠ organization. My goals for the project were to create extend the package API by creating a new data structure to allow for more predictive measures, and doing some summer cleaning by closing some smaller issues and adapting the package to current tooling and best practices.

Quick note of some acronyms I may use and forget to define: GSOC == Google Summer of Code; PR == Pull Request; GHA == GitHub Action; API == Application Programming Interface; CV == Cross Validation.

At the onset, I would like to say a quick thank you to my mentors: Dr. Aki Vehtari, Dr. Jonah Gabry, and Noa Kallioinen—all extremely knowledgeable in this work and invaluable to lean on and learn from. They have been quite patient and kind, and are always ready to explain (usually rather simple) things to me. They’re all very busy, and I’m very lucky to have them take the time to help me. Needless to say, none of the following work would’ve been possible without their support and guidance, and getting to meet and work with my mentors has been one of the highlights of my GSOC experience.

What Did I Do?

Non-Measures

To get acquainted with the package I spent the start of GSOC cleaning up things. loo is a storied package, and development time spent on developer niceties is time spent not working on more important, user-facing, critical features. Since, at the onset of the project, I was hesitant to work on those kinds of difficult issues, my mentors and I thought it best to clean up some old issues and bring certain parts of the package up to modern standards.

The following sections are unordered.

Deduplicating Code in loo

The Stan ecosystem is wide⁠, and is centered around tools to better do Bayesian modelling. Whilst the Stan ecosystem encompasses far more than just R, there are quite a number of R packages. Some, like CmdStanR⁠ act as bridges to other software (CmdStanR is “a clean interface to Stan services”), whilst others are novel software, exposing a particular API to R users. loo sits in this latter category—and so does posterior⁠, a loo dependency. We noticed some overlap⁠ in functionality between the two packages, and to reconcile it I rewrote parts⁠ of loo to rely on posterior more⁠. This PR will remain a draft PR until posterior gets updated and a new version is pushed to CRAN.

Air

I tend to keep up with Posit⁠’s (née RStudio) work and am usually an early adopter. I’m typing this up in Positron⁠, Posit’s fork of Code OSS⁠ which I’ve been using since its public beta a year ago. Before that, I was writing R in RStudio; I use pak (mostly); I write up documents in Quarto when I’m not using Markdown + Typst; I write tidyverse and tidymodels code; I follow the Tidyverse style guide⁠; etc. I would stop short of calling myself a fanatic, but I am most certainly a fan. So when they announced air, their R formatter, written in everyone’s beloved Rust that too (just like stuff from Astral⁠! They’re very cool too, uv is a godsend for Python work), I just had to try it. Unsurprisingly, I like air a lot. It is opinionated, it is fast, it is mostly stable and accurate. It does what it says on the tin—so I exclusively use air in my personal work, and wherever I can dictate tooling. When Posit further announced some beta GitHub Actions to automatically apply air to incoming PRs⁠, I had to bring that into loo.

It’s a super simple PR⁠, but is currently still a draft. As perhaps is expected from me, the young, maybe capricious contributor, I didn’t really think about the knock-on effects. One of my mentors, Jonah, brought up some good points, viz. that air is still in beta, so the PR is shelved for now. But air 1.0 will come and I can finally not worry about accidentally automatically formatting the file⁠ I’m working on. I must admit that is a minor annoyance, and is entirely solvable by setting a Workspace setting—but now I need to remember to not check in the freshly minted settings.json which wasn’t in the gitignore. Writing all that down motivated me to (and took more time than) open a PR⁠.

Testing

Perhaps the most substantial change I made was updating the version of testthat which loo uses. In case you aren’t familiar, testthat⁠ is a testing library for packages. It does a bit more than unit tests, viz. “golden tests”⁠ aka snapshots—but I’ll get back to that. We were using testthat 2e, which was fine and would work, but wouldn’t get any of the newest changes. 3e signifies a breaking change⁠, but I thought it would be a good idea to update our testing tooling to the (stable) latest and greatest, in line with bringing this package up to some modern standards.

Unfortunately, due to 3e indicating a breaking change, I couldn’t just go to DESCRIPTION pop in Config/testthat/edition: 3 and call it a day—a number of functions we had used before were deprecated and had to be updated to match the new syntax. While this was a pain to do, it was a one-time pain. Additionally, with access to snapshots, I saw the potential for reducing some tests’ complexity by checking against a snapshot instead of picking apart output and checking each section individually (see test_psislw.R⁠). These were small changes but less code means fewer chances for mistakes, and I thought it prudent to maximally rely on testthat instead of attempting to replicate the functionality.

Another nicety of 3e was parallel testing⁠. That just took a single line in the DESCRIPTION, but coupled with start-first targeting some of loo’s longest tests, checking time should go down. I haven’t measured this, and I don’t know how you would get reliable results for a number of reasons, but at least theoretically things should be faster.

I also made sure that tests don’t library the package⁠—this was inspired by a funny little bug I had where a test was failing on my machine, but was passing on my mentors’ and the GHA runner. I invite you to wonder for a moment why this could be the case—a package I’m working on locally has tests which fail on my machine, but pass on every other machine. No prizes if you get it right (I certainly didn’t), but the issue is that of building a development version of a package versus just running devtools::check(). I was running my checks using devtools::check(), which would eventually load up the specific R test file containing the assertion which was failing—but with the released version of loo instead of the latest dev version. As mentioned prior, I had updated some code to rely more on posterior which caused some output to change, negligibly but still noticeably, so my test was failing. That was a pain, so no more library(loo) in tests; testthat takes care of building and loading the latest version of the package.

I’m think there’s something else but I’m a touch tired of writing so much about testing so just check out the PR⁠ yourself.

On a whole this PR took more time and effort than I expected, but I think it was time well spent. For reference, though lines of code changed (delta lines of code := DLOC) isn’t a great metric, this PR was ~9K DLOC, i.e. added or deleted. That should be belied by noting that the snapshots probably are the bulk of those changes, but if we ignore that for a second it certainly seems like I’ve accomplished something quite impressive doesn’t it. Again, more importantly, loo is now up to date with best R package testing practices.

Updating pkgdown Workflow

This PR⁠ was a lot less involved. I just changed a lot of minor things, which I will keep as a bulleted list for your and my convenience:

• Updated icons
• Built site in main branch in the ./docs subdirectory
• Added GHA to build site on release
• Build a dev site⁠ with the latest version of the code and docs

Note that the DLOC for this PR is ~75k. Just saying.¹

So that was all work that was technically not specified in our initial goals, which were focused on adding more measures.

Measures

The main focus of the project was to extend the API for more predictive measures, so this work is perhaps most critical. To that end, it also has required the most thinking, both in design and implementation, and so progress is hard to write about. I’ve been working on this in my own little repo⁠, but we have mostly been talking a lot about what to do and I’ve been asking lots of questions about how to do things. You can see a lot of the discussion about this topic at this issue⁠. I know that such a short paragraph may be somewhat of a disappointing or surprising given the amount of detail that I have written about admittedly less important contributions before, but there isn’t as much hard content I can put down about this. A lot of time has been spent on getting me kinda-sorta up to speed with the project and bolstering my understanding of both loo the package, and leave-one-out CV the process and its place in the Bayesian workflow. That onboarding time wasn’t really needed for the software stuff.

What Did I Learn?

Lots of stuff!

Hitherto, I have almost exclusively worked on greenfield projects⁠, and have been able to make sweeping changes without much or any oversight or reviews really. I usually work on scripts or starting up projects, so I am the person who would be approving changes and making these decisions, so I tend to go for bleeding edge tooling. loo is a mature package, and working on existing software is a very useful skill which I haven’t been able to practice before. Having to read other people’s code is something that I will definitely have to do a lot in the future; it is also very different to read code when you want to know what a function is doing vs reading code to really understand what a function is doing and extend that package. I’ve had a little experience in this regard, but this summer was a concentrated dosage.

I have also learnt more about Bayesian model checking and Bayesian workflows from this. I know Dr. Vehtari and Dr. Gabry (with Andrew Gelman et al., I presume) are working on a book⁠ on this topic which I will check out when it is released. For now, though, I’ve been learning bits and pieces about this process. The aforementioned book is said to be an extended and updated version of a paper⁠, and I have skimmed parts of that article before, and have read a little bit more after starting work on loo. Also, since loo is all about leave one out cross validation, I have revisited a paper⁠ about CV which still goes way over my head but the parts I understand are interesting. That latter sentiment cannot be overstated; I still have the smallest of understandings of all these topics which I’ve been mentioning, but I have a better understanding than I have before. It will take a lot more background, reading, and experience to understand these things I think—and that is one of my motivations for a PhD—but the little I understand is extremely interesting.

What’s Left?

Also lots of stuff! Measures need further work; documentation, examples, and vignettes need to be written; once things are in place we can start profiling and seeing if there are any performance concerns; and of course, there are always more bugs to squash. I’ve been working exclusively on loo, but would like to look at some of the other R projects in the Stan ecosystem to see if there are any other places where I could port or do some of the work on loo.

I will be continuing on this project, so the end of GSOC has been more of a formality, but the structure this has imposed is good as now I’ve been forced to look back and see what I actually have done this summer. I’m looking forward to being able to close issue 281 at some point, hopefully in the near future :).