torokernel · GitHub

Writing drivers for x86 in Freepascal

Introduction

This article briefly presents notes for writing drivers for x86 in Freepascal. The article also presents some limitations and how to workaround. The article uses the wording device to refer to the hardware and the driver to refer to the software that handles that hardware. This article gathers experience from the past years writing drivers for Toro unikernel.

What is a driver?

A driver is a program that interacts with the hardware and exposes an API for the OS. The OS proposes different APIs depending on the kind of device, e.g., disk, network-card, mouse, keyboard. Then, a driver implements this API depending on the device. A driver interacts with the hardware with mainly two mechanisms:

  • by writing/reading from a memory address or IO port
  • by capturing interruptions

The driver executes in the context of the kernel thus making the driver different than a user application. For example, a driver won't sleep during the handling of an interruption because it is going to add unnecessary latencies on the thread that hosts that interruption.

Variables, structures, and unions

In Pascal, the primitive types that often drivers use are the following:

  • Byte
  • Word
  • DWORD
  • QWORD

For some structures, we require to prevent the compiler to pad the structure. This is important when we have structures that are used to talk to memory-mapped devices. In this case, we have to use the keyword packed. The following example shows how to define a packed record:

type
  packed record = MyType
    var1: Dword;
    var2: Byte;
    var3: Dword;
  end;

To align var3 to the next DWORD, the compiler will pad between var2 and var3. In other words, without the record keyword, the sizeof(MyType) would return 12 whereas, with record, sizeof(MyType) would return 9. If you want to understand more about padding check the following article http://www.catb.org/esr/structure-packing/.

In the case of unions, you can find an interesting discussion here. It is also possible to work around unions by defining typed-pointers and then casting on the corresponding field.

Interruption Handler

Interruptions are the way that the device tells the driver that something has happened. For example, if a new packet arrives, the device triggers an interruption to inform the driver. When the interruption is received, the processor stops what is doing, and executes the interruption handler. In the driver, the interruption handler is defined as a procedure that is executed when the interruption is triggered. For example, this is the definition of the interruption handler for virtio-devices:

procedure VirtIOIrqHandler; [nostackframe]; assembler;

An interruption handler shall be defined as a procedure with the keywords assembler and nostackframe. This prevents the compiler to save the current stack frame in the RBP register (see https://stackoverflow.com/questions/41912684/what-is-the-purpose-of-the-rbp-register-in-x86-64-assembler/41912747). The assembler keyword specifies that the procedure only contains assembler code.

asm
  push rbp
  push rax
  push rbx
  push rcx
  push rdx
  push rdi
  push rsi
  push r8
  push r9
  push r10
  push r11
  push r12
  push r13
  push r14
  push r15
  call Handler

The interruption handler shall begin by saving the contents of the registers in the stack. After pushing all the registers in the stack, the procedure invokes the handler.

  pop r15
  pop r14
  pop r13
  pop r12
  pop r11
  pop r10
  pop r9
  pop r8
  pop rsi
  pop rdi
  pop rdx
  pop rcx
  pop rbx
  pop rax
  pop rbp
  iretq
end;

The procedure ends by popping the registers from the stack in the correct order. The interruption handler has to finish by executing the iretq instruction which indicates a return from an interruption.

Memory-mapped devices

Most of the time, drivers talk to devices by using shared-memory regions. This can be tricky due to the reordering of memory operations and non-volatile access. As this article states:

Changes to memory ordering are made both by the compiler (at compile time) and by the processor (at run time), all in the name of making your code run faster.

We can categorize these optimizations into:

  • compilator re-ordering and non-volatile access
  • processor re-ordering

To prevent these optimizations, drivers require to use volatile and memory barriers for some memory accesses.

The Freepascal compiler does not do compilation re-ordering as the gcc compiler does (see https://preshing.com/20120625/memory-ordering-at-compile-time/ for more information). However, some loads and stores may be removed, i.e., non-volatile access. Thus, the driver has to explicitly defines which accesses are volatile (there was a discussion about this here) to enforce the compiler to fetch the value from the memory instead of using an internal register or a previous value. In Freepascal, a volatile intrinsic has been added to indicate to the code generator that a particular load from or store to a memory location must not be removed (see here for more information and also note 2).

Memory barriers prevent the processor to reorder memory accesses. They impose a perceived partial ordering over the memory operations (see here for more information). Memory barriers ensure that all the operations that target a memory address have finished after the barrier. For example, the virtio specification requires that the driver perform a memory barrier just before the update of the idx register to ensure that the device sees the most up-to-date copy. In Toro, memory barriers are implemented as follows:

procedure ReadBarrier;assembler;nostackframe;
asm
  lfence
end;
procedure WriteBarrier;assembler;nostackframe;
asm
  sfence
end;
procedure ReadWriteBarrier;assembler;nostackframe;
asm
  lock add DWORD [rsp] - 4, 0
end;

The memory barriers for loading or storing are implemented by using the lfence and sfence instructions. The load and store have been implemented in issue #413.

Note that, in x86, cores have an updated version of the memory so communication over shared memory does not require the use of strong memory barrier but only compiler memory barriers. The later is not required in Freepascal since the compiler does not do instruction ordering.

Macros and Inline

In Freepascal, some inlined functions and procedures are ignored by the compiler, e.g., pure assembler procedure. In these cases, it is better to use macros. For example, these are the macros to enable and disable interruptions:

{$DEFINE EnableInt := asm sti;end;}
{$DEFINE DisableInt := asm pushfq;cli;end;}
{$DEFINE RestoreInt := asm popfq;end;}

In Freepascal, macros are hidden between units that lead to code duplication. To work around this, we can define the macros in a file and then include it by using {$I}.

Save registers when using assembler statements

Assembler statements require to explicitly tell the compiler what registers are going to be clobbered. In particular, this is important when optimization options are enabled and registers are used. For example, Arch.pas defines the following procedure to read the rdtsc register:

function read_rdtsc: Int64;
var
  l, h: QWORD;
begin
  asm
    xor rax, rax
    xor rdx, rdx
    rdtsc
    mov l, rax
    mov h, rdx
  end ['RAX', 'RDX'];
  Result := QWORD(h shl 32) or l;
end;

The assembler statement modifies the RAX and RDX registers. We need to explicitly tell that these registers will be clobbered so the compiler can save and reload them. To understand what registers can be clobbered and which ones must be preserved, check the x64 ABI here.

Note that this seems not working in x86 for 32 and 64 bits. The compiler just ignores the registers. The register should be stored/restored manually. The only registers that needs to be saved are those that are the compiler allows to be clobbered.

Notes

[1] The first versions of Freepascal supported the interrupt keyword to save and load the content of the registers. This keyword is not supported any longer.

[2] I have observed that in some cases, .e.g, global variables, pointers, access are always volatile, which can explain why I never had these problems and I never required to use volatile explicitly.

Contribute

If you find something that is wrong or could be presented in a simpler way, feel free to contact me at matiasevara@gmail.com and I would be glad to change it.

Read the original on github.com ↗