Introduction What if changing the code under test — not the test itself and not the execution environment — could turn a perfectly stable test case into a flaky one? That is exactly what my colleagues and I discovered in a new phenomenon that we call “test flimsiness”. The name is an acronym since FLIM siness stands for FL akiness I nduced by M utations to the code under test. In (Parry et al.…
Introduction In a recent episode of the Software Engineering Radio podcast, I interviewed Jennings Anderson , a Software Engineer at Meta Platforms, and Amy Rose , the Chief Technology Officer at the Overture Maps Foundation . Our conversation focused on the Overture Maps project, which creates reliable, easy-to-use, and interoperable open map data. During the interview, Amy and Jennings share…
Introduction In a recent episode of the Software Engineering Radio podcast, I interviewed Kacper Łukawski , a Senior Developer Advocate at Qdrant . Our conversation focused on Qdrant, an open-source vector database and similarity search engine implemented in Rust. During the interview, Kacper shares insights into vector embeddings and similarity search, the performance benefits of implementing a…
Introduction In a recent episode of the Software Engineering Radio podcast, I interviewed Samuel Colvin , the CEO and founder of the Pydantic company. Our conversation focused on Pydantic, a popular data validation library for Python, and its ecosystem, including Pydantic AI and Logfire. During our interview, Samuel offers many insights into how these tools can help Python developers build more…
Introduction Are your tests as effective as you think they are? While statement coverage is a common metric for test suite quality, it often doesn’t tell the whole story. High coverage scores can create a false sense of security, leaving critical “gaps” in your testing strategy. In the paper (Maton, Kapfhammer, and McMinn 2025) , my colleagues and I investigate how to identify an “oracle gap” that…
Introduction Have you ever noticed that when one flaky test fails in your continuous integration pipeline, several others seem to fail at the same time? Well, you’re not imagining things! My colleagues and I recently completed a comprehensive study that reveals flaky tests often exist in clusters, failing together due to shared root causes. We call this phenomenon systemic flakiness , and we think…
Introduction In a recent episode of the Software Engineering Radio podcast, I interviewed Will McGugan , the creator of Rich and Textual and the CEO and founder of Textualize . Our conversation focused on building text-based user interfaces (TUIs) and command line interfaces (CLIs) using the aforementioned Python frameworks. Will explained how concepts from web technologies can be adapted for…
Introduction In a recent episode of the Software Engineering Radio podcast, I interviewed Eran Yahav , the CTO of Tabnine and a faculty member in the Computer Science Department at the Israel Institute of Technology . Our conversation focused on Tabnine, an artificial intelligence (AI) coding assistant that uses large language models (LLMs) to help software engineers with tasks like code…
Introduction When software developers talk about problematic test suites, the conversation often begins and ends with flaky tests. And while flakiness is certainly a serious concern, is it really the only symptom of an unhealthy test suite? My colleagues and I argue that it is not. In (McMinn, Roslan, and Kapfhammer 2025) , published at the 2nd International Flaky Tests Workshop, we present a…
Introduction Ever wondered if your test suite is truly good at finding bugs? What if there are hidden weaknesses that traditional testing methods miss? In the paper (Maton, Kapfhammer, and McMinn 2024a) , my colleagues and I explore how eXtreme mutation testing (XMT) can uncover these hidden weaknesses in your test suite. By identifying pseudo-tested methods — or those methods in your program that…
Introduction In a recent episode of the Software Engineering Radio podcast, I interviewed Itamar Friedman , CEO and Co-founder of Qodo (a company previously called CodiumAI). Our conversation focused on automated testing with generative artificial intelligence (AI), exploring how large language models (LLMs) can augment existing test suites and increase code coverage. Itamar and I discussed the…
Introduction In a recent episode of the Software Engineering Radio podcast, I interviewed Goran Petrovic , a senior staff software engineer at Google. Our conversation focused on mutation testing at Google, a practice that assesses the efficacy of a test suite by inserting small faults into programs and measuring the test suite’s ability to detect them. In this interview Goran offers insights into…
Introduction In a recent episode of the Software Engineering Radio podcast, I had the opportunity to interview Jonathan Schneider , a co-founder of Moderne and the creator of OpenRewrite . Our conversation focused on the unique features of OpenRewrite, an automated refactoring tool for software modernization. I’m thankful that Jonathan took the time to participate in this energizing discussion…
Introduction In a recent episode of the Software Engineering Radio podcast, I interviewed Marcelo Trylesinski , a senior software engineer at Pydantic and a maintainer of open-source Python tools including Starlette and Uvicorn . Our conversation focused on the unique features of FastAPI , a modern and high-performance web framework for building APIs with Python 3.6+ based on standard Python type…
Introduction As a co-host of the Software Engineering Radio podcast, I interviewed Wolf Vollprecht , the founder and CEO of Prefix , to discuss the creation of Pixi , a Rust-based workflow manager for Python. During our conversation, Wolf shared some fascinating insights into both his development process and the unique features of Pixi. I’m thankful that Wolf took the time to participate in this…
Introduction Have you ever been stuck between the proverbial “rock and a hard place” when it comes to flaky test detection? On the one hand, you can repeatedly rerun your tests, which is accurate but can take an enormous amount of time and computational resources. On the other hand, you can use a machine learning model to predict flaky tests, which is fast but often not accurate enough to be…
Introduction In a recent episode of the Software Engineering Radio podcast, I had the pleasure of interviewing Hyrum Wright , a senior staff software engineer at Google and one of the editors of the book Software Engineering at Google . During our conversation, we discussed various aspects of software engineering practices at Google, including testing, static analysis, large-scale code changes,…
Introduction In a recent episode of the Software Engineering Radio podcast, I had the pleasure of interviewing Zac Hatfield-Dodds , a leading researcher, software engineer, and maintainer of open-source software. Zac currently leads the assurance team at Anthropic , an artificial intelligence (AI) safety and research company. He is also known for his contributions to open source projects like…
Introduction As a co-host of the Software Engineering Radio podcast, I interviewed Zach Lloyd , the founder and CEO of Warp , to discuss the innovative features and the development process behind the Warp command-line terminal. During our discussion, Zach highlights the unique aspects of Warp, such as its use of artificial intelligence (AI) for command completion and error explanation, its…
Introduction At Allegheny College, a class of students and I are embarking on a developer development adventure as we take a hands-on approach to learning software engineering. You can participate in our adventure by reading the collaboratively published content on the Developer Development web site and joining our Developer Development Discord Server . As part of the software engineering class…