Stupid RCU Tricks: Is RCU Watching?

It is just as easy to ask why RCU wouldn't be watching all the time. After all, you never know when you might need to synchronize!

Unfortunately, an eternally watchful RCU is impractical in the Linux kernel due to energy-efficiency considerations. The problem is that if RCU watches an idle CPU, RCU needs that CPU to execute instructions. And making an idle CPU unnecessarily execute instructions (for a rather broad definition of the word “unnecessarily”) will terminally annoy a great many people in the battery-powered embedded world. And for good reason: Making RCU avoid watching idle CPUs can provide 30-40% increases in battery lifetime.

In this, CPUs are not all that different from people. Interrupting someone who is deep in thought can cause them to lose 20 minutes of work. Similarly, when a CPU is deeply idle, asking it to execute instructions will consume not only the energy required for those instructions, but also much more energy to work its way out of that deep idle state, and then to return back to that deep idle state.

And this is why CPUs must tell RCU to stop watching them when they go idle. This allows RCU to ignore them completely, in particular, to refrain from asking them to execute instructions.

In some kernel configurations, RCU also ignores portions of the kernel's entry/exit code, that is, the last bits of kernel code before switching to userspace and the first bits of kernel code after switching away from userspace. This happens only in kernels built with

CONFIG_NO_HZ_FULL=y

, and even then only on CPUs mentioned in the CPU list passed to the

nohz_full

kernel parameter. This enables carefully configured HPC applications and CPU-bound real-time applications to get near-bare-metal performance from such CPUs, while still having the entire Linux kernel at their beck and call. Because RCU is not watching such applications, the scheduling-clock interrupt can be turned off entirely, thus avoiding disturbing such performance-critical applications.

But if RCU is not watching a given CPU,

rcu_read_lock()

has no effect on that CPU, which can come as a nasty shock to the corresponding RCU read-side critical section, which naively expected to be able to safely traverse an RCU-protected data structure. This can be a trap for the unwary, which is why kernels built with

CONFIG_PROVE_LOCKING=y

(lockdep) complain bitterly when

rcu_read_lock()

is invoked on CPUs that RCU is not watching.

But suppose that you have code using RCU that is invoked both from deep within the idle loop and from normal tasks.

Back in the day, this was not much of a problem. True to its name, the idle loop was not much more than a loop, and the deep architecture-specific code on the kernel entry/exit paths had no need of RCU. This has changed, especially with the advent of idle drivers and governors, to say nothing of tracing. So what can you do?

First, you can invoke

rcu_is_watching()

, which, as its name suggests, will return

true

if RCU is watching. And, as you might expect, lockdep uses this function to figure out when it should complain bitterly. The following example code lays out the current possibilities:

if (rcu_is_watching())
        printk("Invoked from normal or idle task with RCU watching.\n");
else if (is_idle_task(current))
        printk("Invoked from deep within in the idle task where RCU is not watching.\");
else
        printk("Invoked from nohz_full entry/exit code where RCU is not watching.\");

Except that even invoking

printk()

is an iffy proposition while RCU is not watching.

So suppose that you invoke

rcu_is_watching()

and it helpfully returns

false

, indicating that you cannot invoke

rcu_read_lock()

and friends. What now?

You could do what the v5.18 Linux kernel's

kernel_text_address()

function does, which can be abbreviated as follows:

no_rcu = !rcu_is_watching();
if (no_rcu)
        rcu_nmi_enter(); // Make RCU watch!!!
do_rcu_traversals();
if (no_rcu)
        rcu_nmi_exit(); // Return RCU to its prior watchfulness state.

If your code is not so performance-critical, you can do what the arm64 implementation of the

cpu_suspend()

function does:

RCU_NONIDLE(__cpu_suspend_exit());

This macro forces RCU to watch while it executes its argument as follows:

#define RCU_NONIDLE(a) \
        do { \
                rcu_irq_enter_irqson(); \
                do { a; } while (0); \
                rcu_irq_exit_irqson(); \
        } while (0)

The

rcu_irq_enter_irqson()

and

rcu_irq_exit_irqson()

functions are essentially wrappers around the aforementioned

rcu_nmi_enter()

and

rcu_nmi_exit()

functions.

Although

RCU_NONIDLE()

is more compact than the

kernel_text_address()

approach, it is still annoying to have to pass your code to a macro. And this is why Peter Zijlstra has been reworking the various idle loops to cause RCU to be watching a much greater fraction of their code. This might well be an ongoing process as the idle loops continue gaining functionality, but Peter's good work thus far at least makes RCU watch the idle governors and a much larger fraction of the idle loop's trace events. When combined with the kernel entry/exit work by Peter, Thomas Gleixner, Mark Rutland, and many others, it is hoped that the functions not watched by RCU will all eventually be decorated with something like

noinstr

, for example:

static noinline noinstr unsigned long rcu_dynticks_inc(int incby)
{
        return arch_atomic_add_return(incby, this_cpu_ptr(&rcu_data.dynticks));
}

We don't need to worry about exactly what this function does. For this blog entry, it is enough to know that its

noinstr

tag prevents tracing this function, making it less problematic for RCU to not be watching it.

What exactly are you prohibited from doing while RCU is not watching your code?

As noted before, RCU readers are a no-go. If you try invoking

rcu_read_lock()

,

rcu_read_unlock()

,

rcu_read_lock_bh()

,

rcu_read_unlock_bh()

,

rcu_read_lock_sched()

, or

rcu_read_lock_sched()

from regions of code where

rcu_is_watching()

would return

false

, lockdep will complain.

On the other hand, using SRCU (

srcu_read_lock()

and

srcu_read_unlock()

) is just fine, as is RCU Tasks Trace (

rcu_read_lock_trace()

and

rcu_read_unlock_trace()

). RCU Tasks Rude does not have explicit read-side markers, but anything that disables preemption acts as an RCU Tasks Rude reader no matter what

rcu_is_watching()

would return at the time.

RCU Tasks is an odd special case. Like RCU Tasks Rude, RCU Tasks has implicit read-side markers, which are any region of non-idle-task kernel code that does not do a voluntary context switch (the idle tasks are instead handled by RCU Tasks Rude). Except that in kernels built with

CONFIG_PREEMPTION=n

and without any of RCU's test suite, the RCU Tasks API maps to plain old RCU. This means that code not watched by RCU is ignored by the remapped RCU Tasks in such kernels. Given that RCU Tasks ignores the idle tasks, this affects only user entry/exit code in kernels built with

CONFIG_NO_HZ_FULL=y

, and even then, only on CPUs mentioned in the list given to the

nohz_full

kernel boot parameter. However, this situation can nevertheless be a trap for the unwary.

Therefore, in post-v5.18 mainline, you can build your kernel with

CONFIG_FORCE_TASKS_RCU=y

, in which case RCU Tasks will always be built into your kernel, avoiding this trap.

In summary, energy-efficiency, battery-lifetime, and application-performance/latency concerns force RCU to avert its gaze from idle CPUs, and, in kernels built with

CONFIG_NO_HZ_FULL=y

, also from

nohz_full

CPUs on the low-level kernel entry/exit code paths. Fortunately, recent changes have allowed RCU to watch more code, but this being the kernel, corner cases will always be with us. This corner-case code from which RCU must avert its gaze requires the special handling described in this blog post.