Open source and virtualization blog
This is the fourth post in a series about building a virtio-serial device in Verilog for an FPGA development board. This time we'll look at processing the virtio-serial device's transmit and receive virtqueues. Series table of contents Part 1: Overview Part 2 - MMIO registers, DMA, and interrupts Part 3 - virtio-serial device design Part 4 - Virtqueue processing (you are here) Part 5 - UART…
This is the final post in a series about building a virtio-serial device in Verilog for an FPGA development board. This time we'll look at the firmware running on the PicoRV32 RISC-V soft-core in the FPGA. Series table of contents Part 1: Overview Part 2 - MMIO registers, DMA, and interrupts Part 3 - virtio-serial device design Part 4 - Virtqueue processing Part 5 - UART receiver and transmitter…
This is the fifth post in a series about building a virtio-serial device in Verilog for an FPGA development board. This time we'll look at the UART receiver and transmitter. Series table of contents Part 1: Overview Part 2 - MMIO registers, DMA, and interrupts Part 3 - virtio-serial device design Part 4 - Virtqueue processing Part 5 - UART receiver and transmitter (you are here) Part 6 - Writing…
This is the third post in a series about building a virtio-serial device in Verilog for an FPGA development board. This time we'll look at the design of the virtio-serial device and how to decompose it into modules. Series table of contents Part 1: Overview Part 2 - MMIO registers, DMA, and interrupts Part 3 - virtio-serial device design (you are here) Part 4 - Virtqueue processing Part 5 - UART…
This is the second post in a series about building a virtio-serial device in Verilog for an FPGA development board. This time we'll look at integrating MMIO devices to the PicoSoC (an open source System-on-Chip using the PicoRV32 RISC-V soft-core). Series table of contents Part 1: Overview Part 2 - MMIO registers, DMA, and interrupts (you are here) Part 3 - virtio-serial device design Part 4 -…
This is a the first post in a series about building a virtio-serial device in Verilog for a Field Programmable Gate Array (FPGA) development board. This was a project I did in my spare time to become familiar with logic design. I hope these blog posts will offer a glimpse into designing your own devices and FPGA development. Series table of contents Part 1: Overview (you are here) Part 2 - MMIO…
With the VIRTIO 1.4 specification for I/O devices expected to be published soon, here are the most prominent changes. For more fine-grained changes like the latest offloading capabilities in virtio-net devices, please refer to the draft specification . New device types The most exciting changes are new device types that allow for entirely new I/O devices to be built with VIRTIO. In 1.4 there are…
My KVM Forum 2025 talk "Making io_uring Pervasive in QEMU" is now available on YouTube . The slides are also available here (PDF) . This talk is about integrating Linux io_uring into QEMU's event loop to enable performance optimizations and use new kernel features available through io_uring. This topic is also relevant for other I/O-intensive applications (network services, software-defined…
A fair amount of the development work I do is related to storage performance in QEMU/KVM. Although I have written about disk I/O benchmarking and my performance analysis workflow in the past, I haven't covered the performance tools that I frequently rely on. In this post I'll go over what's in my toolbox and hopefully this will be helpful to others. Performance analysis is hard when the system is…
My KVM Forum 2024 talk "IOThread Virtqueue Mapping: Improving virtio-blk SMP scalability in QEMU" is now available on YouTube . The slides are also available here . IOThread Virtqueue Mapping is a new QEMU feature for configuring multiple IOThreads that will handle a virtio-blk device's virtqueues. This means QEMU can take advantage of the host's Linux multi-queue block layer and assign CPUs to…
Programming languages currently offer few defences against supply chain attacks where a malicious third-party library compromises a program. As I write this, the open source community is trying to figure out the details of the xz-utils backdoor , but there is a long history of supply chain attacks. High profile incidents have made plain the danger of shipping software built from large numbers…
KubeVirt makes it possible to run virtual machines on Kubernetes alongside container workloads. Virtual machines are configured using VirtualMachineInstance YAML . But under the hood of KubeVirt lies the same libvirt tooling that is commonly used to run KVM virtual machines on Linux. Accessing libvirt can be convenient for development and troubleshooting. Note that bypassing KubeVirt must be done…
File systems and relational databases are like cousins. They share more than is apparent at first glance. It's not immediately obvious that relational databases and file systems rely upon the same underlying concept. That underlying concept is the key-value store and this article explores how both file systems and databases can be implemented on top of key-value stores. The key-value store…
This post is about the AioContext lock removal in QEMU 9.0 (planned for release in 2024), how we got here, and what it means for multi-threaded code in QEMU. Early QEMU as a single-threaded program Until 2009 QEMU was largely a single-threaded program. This had the benefit that the code didn't need to consider thread-safety and was thus simpler and less bug-prone. The main loop interleaved running…
I recently looked at papers about free space management and snapshots in storage systems like file systems, volume managers, and key-value stores. I'm publishing my notes in case you find them useful, but the real value might simply be the links to papers in this field. They might be a useful starting point for someone wishing to read into this field. My aim was to get an overview of data…
Update: The video is now available here and the slides are available here (PDF) . I will be at Linux Plumbers Conference 2023 to present "Trust, confidentiality, and hardening: the virtio lessons" at 2:30pm on Wednesday, November 15th. Michael Tsirkin and I prepared this talk about the evolution of the trust model of the Linux VIRTIO drivers. It explores how the drivers have been hardened in…
I hacked up a prototype multi-player game in just a static HTML/JS files. The game runs in players' browsers without the need for a centralized game server. This peer-to-peer model - getting rid of the server - is something I've been interested in for a long time. I finally discovered a way to make it work without hosting my own server or relying on a hosted service that requires API keys,…
The 1.3.0 release of the libblkio high-performance block device I/O library is out. libblkio provides an API that abstracts various storage interfaces that are efficient but costly to integrate into applications including io_uring, NVMe uring_cmd, virtio-blk-pci, vdpa-blk, and more. Switching between them is very easy using libblkio and gives you flexibility to target both kernel and userspace…
Every once in a while a bug comes along where a guest hangs while communicating with a QEMU VIRTIO device. In this blog post I'll share some debugging approaches that can help QEMU developers who are trying to understand why a VIRTIO device is stuck. There are a number of reasons why communication with a VIRTIO device might cease, so it helps to identify the nature of the hang: Did the QEMU device…
I started working on libblkio in 2020 with the goal of creating a high-performance block I/O library. The internals are written in Rust while the library exposes a public C API for easy integration into existing applications. Most languages have a way to call C APIs, often called a Foreign Function Interface (FFI) . It's the most universal way to call into code written in different languages…
At FOSDEM '23 I gave a talk about vhost-user-blk and its use as a userspace block I/O interface. The video and slides are now available here . Enjoy!
vhost-user-blk has connected hypervisors to software-defined storage since around 2017, but it was mainly seen as virtualization technology. Did you know that vhost-user-blk is not specific to virtual machines? I think it's time to use it more generally as a userspace block I/O interface because it's fast, unprivileged, and avoids exposing kernel attack surfaces. My LWN.net article about Accessing…
At KVM Forum 2022 Kevin Wolf and Stefano Garzarella gave a talk on qemu-storage-daemon, a way to get QEMU's storage functionality without running a VM. It's great for accessing disk images, basically taking the older qemu-nbd to the next level. The cool thing is this makes QEMU's software-defined storage functionality - block devices with snapshots, incremental backup, image file formats, etc -…
vhost-user-blk is a high-performance storage protocol that connects virtual machines to software-defined storage like SPDK or qemu-storage-daemon . Until now, tool support for vhost-user-blk has been lacking. Accessing vhost-user-blk devices involved running a virtual machine, which requires more setup than one would like. QEMU 7.2 adds vhost-user-blk support to the qemu-img tool. This is possible…
Queues and their implementation using shared memory ring buffers are a standard tool for communicating with I/O devices and between CPUs. Although ring buffers are widely used, there is no standard memory layout and it's interesting to compare the differences between designs. When defining libblkio 's APIs, I surveyed the ring buffer designs in VIRTIO , NVMe , and io_uring . This article examines…