Notes and jottings from an ex-Ubuntu Kernel Team Engineer
Last week I was bitten by a interesting C feature. The following terminate function was expected to exit if okay was zero (false) however it exited when zero was passed to it. The reason is the missing semicolon after the return function. The interesting part this that is compiles fine because the void function terminate is allowed to return the void return value, in this case the void return from…
Left shifting values seems simple, but the following code contains a bug: The literal value 1 is actually a signed int, so the promotion to a uint64_t type will sign extend the value before assigning to x, causing x to be 0xffffffff80000000. The fix is simple, just make the literal value 1 an unsigned int using 1U instead of 1.
Intel Hardware P-State (aka Harware Controlled Performance or "Speed Shift") (HWP) is a feature found in more modern x86 Intel CPUs (Skylake onwards). It attempts to select the best CPU frequency and voltage to match the optimal power efficiency for the desired CPU performance. HWP is more responsive than the older operating system controlled methods and should therefore be more effective. To test…
The release of stress-ng 0.12.12 incorporates some useful features and a handful of new stressors. Media devices such as HDDs and SSDs normally support Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T.) to detect and report various measurements of drive reliability. To complement the various file system and I/O stressors, stress-ng now has a --smart option that checks for any changes…
C has the useful feature of adjacent allowing literal strings to be automatically concatenated. This is described in K&R "The C programming language" 2nd edition, page 194, section A2.6 "String Literals": "Adjacent string literals are concatenated into a single string." Unfortunately over the years I've seen several occasions where this useful feature has led to bugs not being detected, for…
The C ternary operator expr1 ? expr2 : expr3 has a subtle side note described in K&R 2nd edition, page 52, section 2.11: "If expr2 and expr3 are of different types, the type of the result is determined by the conversion rules discussed earlier in the chapter". This refers to page 44, section 2.7 "Type Conversions". It's worth reading a few times along with section A6.5 "Arithmetic Conversions".…
The C infinite for-loop gotcha is one of the less frequent issues I find with static analysis, but I feel it is worth documenting because it is obscure but easy to do. Consider the following C example: Since i is a 8 bit integer, it will wrap around to zero when it reaches the maximum 8 bit value of 255 and so we end up with an infinite loop if the upper limit of the loop n is 256 or more. The fix…
Multiplying integers in C is easy. It is also easy to get it wrong. A common issue found using static analysis on the Linux kernel is the integer overflow before widening gotcha. Consider the following code that takes the 2 unsigned 32 bit integers, multiplies them together and returns the unsigned 64 bit result: The multiplication is performed using unsigned 32 bit arithmetic and the unsigned 32…
Shifting integers in C is easy. Alas it is also easy to get it wrong. A common issue found using static analysis on the Linux kernel is the unintentional sign extension gotcha. Consider the following code that takes the 4 unsigned 8 bit integers in array data and returns an unsigned 64 bit integer: C promotes the uint8_t integers into signed ints on the right shift. If data[3] has the upper bit…
Over the past year there has been focused work on improving the test coverage of the Linux Kernel with stress-ng. Increased test coverage exercises more kernel code and hence improves the breadth of testing, allowing us to be more confident that more corner cases are being handled correctly. The test coverage has been improved in several ways: testing more system calls; most system calls are being…
The Linux 5.9-rc6 kernel source contains over 300,000 literal strings used in kernel messages of various sorts (errors, warnings, etc) and it is no surprise that typos and spelling mistakes slip into these messages from time to time. To catch spelling mistakes I run a daily automated job that fetches the tip from linux-next and runs a fast spelling checker tool that finds all spelling mistakes and…
Today I needed to capture a rather large kernel stack dump, this is rather trivial using virsh. Using virt-manager I created a VM named vm-focal and in the guest ran: sudo systemctl enable serial-getty@ttyS0.service Then on the host running the VM I ran : virsh console vm-focal Then all I needed to do was produce the stack dump and the console output was successfully dumped by virsh. Easy.
Last week I attended the Embedded Linux Conference (Europe 2019) and presented a talk on stress-ng . The slide deck for this presentation is now available.
Stress testing CPU temperatures is not exactly straight forward. CPU designs vary from CPU to CPU and each have their own strengths and weaknesses in cache design, integer maths, floating point math, bit-wise logical operations and branch prediction to name but a few. I've been asked several times about the "best" CPU stressor method in stress-ng to use to make a CPU run hot. As an experiment I…
The early boot requires loading and decompressing the kernel and initramfs from the boot storage device. This speed is dependent on several factors, speed of loading an image from the boot device, the CPU and memory/cache speed for decompression and the compression type. Generally speaking, the smallest (best) compression takes longer to decompress due to the extra complexity in the compression…
Whenever a process accesses a virtual address where there isn't currently a physical page mapped into its process space then a page fault occurs. This causes an interrupt so that the kernel can handle the page fault. A minor page fault occurs when the kernel can successfully map a physically resident page for the faulted user-space virtual address (for example, accessing a memory resident page…
Over the past 9+ months I've been cleaning up stress-ng in preparation for a V0.10.00 release. Stress-ng is a portable Linux/UNIX Swiss army knife of micro-benchmarking kernel stress tests. The Ubuntu kernel team uses stress-ng for kernel regression testing in several ways: Checking that the kernel does not crash when being stressed tested Performance (bogo-op throughput) regression checks Power…
Last year I wrote about kernel commits that are tagged with the "Fixes" tag. Kernel developers use the "Fixes" tag on a bug fix commit to reference an older commit that originally introduced the bug. The adoption of the tag has been steadily increasing since v3.12 of the kernel: The red line shows the number of commits per release of the kernel, and the blue line shows the number of commits that…
The Linux kernel I/O schedulers attempt to balance the need to get the best possible I/O performance while also trying to ensure the I/O requests are "fairly" shared among the I/O consumers. There are several I/O schedulers in Linux, each try to solve the I/O scheduling issues using different mechanisms/heuristics and each has their own set of strengths and weaknesses. For traditional spinning…
Forkstat is a simple utility I wrote a while ago that can trace process activity using the rather useful Linux NETLINK_CONNECTOR API. Recently I have added two extra features that may be of interest: 1. Improved output using some UTF-8 glyphs. These are used to show process parent/child relationships and various process events, such as termination, core dumping and renaming. Use the new -g (glyph)…
Bpftrace is a new high-level tracing language for Linux using the extended Berkeley packet filter (eBPF). It is a very powerful and flexible tracing front-end that enables systems to be analyzed much like DTrace . The bpftrace tool is now installable as a snap . From the command line one can install it and enable it to use system tracing as follows: sudo snap install bpftrace sudo snap connect…
I've been running static analysis using CoverityScan on linux-next for 2 years with the aim to find bugs (and try to fix some) before they are merged into Linux. I have also been gathering the defect count data and tracking the defect trends: As one can see from above, CoverityScan has found a considerable amount of defects and these are being steadily fixed by the Linux developer community. The…
The low-latency kernel offering with Ubuntu provides a kernel tuned for low-latency environments using low-latency kernel configuration options. The x86 kernels by default run with the Intel-Pstate CPU scheduler set to run with the powersave scaling governor biased towards power efficiency. While power efficiency is fine for most use-cases, it can introduce latencies due to the fact that the CPU…
Over the past 5 years there has been a steady increase in the number of kernel bug fix commits that use the "Fixes" tag. Kernel developers use this annotation on a commit to reference an older commit that originally introduced the bug, which is obviously very useful for bug tracking purposes. What is interesting is that there has been a steady take-up of developers using this annotation: With the…
The Linux kernel grows at an amazing pace, each kernel release adds more functionality, more drivers and hence more kernel modules. I recently wondered what the trend was for kernel module growth per release, so I performed module builds on kernels v2.6.24 through to v4.16-rc2 for x86-64 to get a better idea of growth rates: ..as one can see, the rate of growth is relatively linear with about 89…