Let see how the Linux IO stack looks on 5.15 kernel. For this depiction I have use Ubuntu 22.04 desktop with 5.15 kernel ======================= Lets make a filesystem. ======================= root@kundan-virtual-machine:~# mkfs.ext4 /dev/sdb mke2fs 1.46.5 (30-Dec-2021) Creating filesystem with 786432 4k blocks and 196608 inodes Filesystem UUID: d9682545-5aee-44db-ae0d-88514de3bf12 Superblock…
In this blog lets look and try to find the physical blocks of a file. Given a struct inode * of a file. Every file has a inode. There is a on disk inode and one struct inode. The disk inode is stored in the disk, so across reboots it holds the information of a file whereabouts and details in the disk. On disk file inode has different layout for various filesystems: If we see the ext4.h, it looks…
LIBAIO Lets see sample program which uses libaio and delve into the IO path for the same. #include <linux/aio_abi.h> #include <sys/syscall.h> #include <linux/aio_abi.h> #include <errno.h> #include <stdio.h> #include <stdlib.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include <unistd.h> #include <inttypes.h> #include <assert.h> typedef unsigned long uintptr_t; int fd = 0;…
Linux io_uring basics and details can be fetched from here https://kernel.dk/io_uring.pdf . Reader is encouraged to complete this document first. Also various examples of io_uring are mentioned here: https://github.com/shuveb/io_uring-by-example I am using 5.3.18-22 kernel in this blog. Lets see the program to do read of twenty(20) 512bytes blocks from a block device. #include <stdio.h> #include…
Kprobe can be used in latest kernels when the jprobe is discontinued. I am updating the blog with latest 5.19 linux kernel. In the pre-handler the arguments need to be fetched using the registers. The arguments are passed in following sequence for x86 : first 6 arguments in rdi, rsi, rdx, rcx, r8, r9 So if we are trying to probe a function "submit_bio" which has signature : void submit_bio(struct…
For developing the understanding of Linux RCU we shall first go ahead with the the Paul McKenney's explanation on YouTube. https://www.youtube.com/watch?v=obDzjElRj9c This helps a lot in understanding the concept behind RCU. Next we try to execute RCU examples from here : https://www.kernel.org/doc/html/latest/RCU/whatisRCU.html#what-are-some-example-uses-of-core-rcu-api Sample kernel module to…
Creation of SPDK NVMe-OF TCP targets We can easily create NVMe targets using SPDK software. This document shall give us clear steps to configure and use SPDK targets and initiator. These targets can be used and NVMe LUNs can be exposed through them. Documentation https://spdk.io/ https://spdk.io/doc/getting_started.html SPDK download : git clone https://github.com/spdk/spdk cd spdk git submodule…
In this blog we will go through the code architecture and internal working of linux FUSE driver, user space callbacks etc. Many good links are present on the internet for FUSE workings. Few of these : https://libfuse.github.io/doxygen/fast17-vangoor.pdf https://github.com/libfuse/libfuse/wiki/Fsnotify-and-FUSE These guys dont say how to form a fuse and codewise details of FUSE. Lets try to address…
We will write a simple C program and write to file. We will see how the write is given to the disk. Which are the layers through which the IO traverses. Let us make a file system on the entire disk: [root@kundan ~]# mkfs.ext4 /dev/sdh mke2fs 1.42.9 (28-Dec-2013) /dev/sdh is entire device, not just one partition! Proceed anyway? (y,n) y Discarding device blocks: done Filesystem label= OS type:…
entry_64.S is collection of functions written in assembly. This file has many routines which are used in interrupts, system calls etc. Let us see one by one how many such routines are present in this file. Functions present in this file : A) save_rest : Handle the saving of extra registers B) ret_from_fork : This function is executed by the child process right after its creation through a fork…
Userspace call of the systemcall: x86_64 user programs invoke a system call by putting the system call number (0 for read) into the RAX register, and the other parameters into specific registers (RDI, RSI, RDX for the first 3 parameters), then issue the x86_64 syscall instruction. http://man7.org/linux/man-pages/man2/syscall.2.html x86-64 syscall rax rax - [5] This instruction causes the processor…
Linux process has many sections. Vaguely speaking these sections can be data, text, stack heap etc. When a Linux process is created these sections are formed and virtual memory is allocated to these. Presence of these sections in Linux kernel. Each of these sections are represented by vm_area_struct in Linux. The Linux task_struct has memory struct in mm_struct . The mm_struct has vm_area_struct…
In this blog we will go through Linux NVMe kernel driver. The NVMe kernel driver has a table of nvme_id_table. As like the pci devices, this table has Vendor and device ID this driver would support. When this driver is inserted the nvme_init function will register this id_table to the PCI. On the insertion of this driver the probe function of this device is called. nvme_probe is the driver probe…
Linux kernel talks about hugepages in their documentation : https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt Using the mount command we can see the hugetlbfs mounted : hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel) the hugetlbfs filesystem is mounted by linux on boot. The available hugepages can be seen using : cat /proc/meminfo ... ... HugePages_Total: 0…
Linux native multipath provides multipathing capability in Linux. It gives the benefits of increased throughput and path failover. Many details for it has been given in https://www.kernel.org/doc/ols/2005/ols2005v1-pages-155-176.pdf In this document we will go through the various user/kernel components of Linux native multipath. The major user space components are libdevmapper shared library and…
In this blog we will see how to analyse a deadlock using crash tool in Linux kernel. We have written a simple kernel module to generate a deadlock: Program to create a deadlock : #include<linux/module.h> #include<linux/version.h> #include<linux/kernel.h> #include<linux/semaphore.h> #include<linux/kthread.h> #include<linux/sched.h> #include<linux/delay.h> #include<linux/slab.h> struct semaphore…
fork, vfork and clone are the system calls which create a process in Linux. Lets go through how these are intercepted at linux. Implementation of vork : #ifdef __ARCH_WANT_SYS_VFORK SYSCALL_DEFINE0( vfork ) { return do_fork(CLONE_VFORK | CLONE_VM | SIGCHLD, 0, 0, NULL, NULL); } #endif Implementation of various clone : SYSCALL_DEFINE5( clone , unsigned long, clone_flags, unsigned long, newsp, int…
In last blog we saw the construction of e820 map. setup_arch then calls : memblock_x86_fill(); This function goes through the memory map provided by the e820 and adds memory regions reserved by the kernel to the memblock with the memblock_add function. setup_arch then calls : init_mem_mapping init_mem_mapping calls memory_map_bottom_up or memory_map_top_down this calls init_range_memory_mapping…
Linux memory init: start_kernel calls setup_arch . setup_arch is a big function where many of the early boot memory allocations and initializations happen. When not many of the kernel memory allocation apis are not available at boot, kernel uses memblock APIs to do the needed allocations. One more mode of allocations is using the early ioremap functionality : EARLY IOREMAP initialization :…
Looked upon the code how iscsid scan works. __scsi_scan_target is the key function which does the SCSI scan. This function first it tries to probe and add LUN 0 using scsi_probe_and_add_lun function. Dec 24 03:24:28 localhost kernel: [<ffffffff813e8c94>] scsi_probe_and_add_lun+0x3f4/0xc80 Dec 24 03:24:28 localhost kernel: [<ffffffff815e5a51>] ? printk+0x77/0x8e Dec 24 03:24:28 localhost kernel:…
Linux segmentation (protected mode): Segmentation is used while converting the virtual address to physical address. This conversion goes through segmentation and paging. In segmentation the segment base address needs to be fetched and added to virtual address. This completes segmentation. The segment base address for each segment is 0 in Linux. This model is called “flat model” . Flat model is…
In this post I will depict how to use jprobes in Linux. Jprobes are the functions which we need to register. Once registered your jprobe will be called just before the Linux kernel test function. For this to work properly your kernel should have been compiled with CONFIG_KPROBES as ''y'' . Also the kernel module shall be compiled with LICENSE "GPL". Here is a sample module which puts prove on…
Wait queues and wakeup in linux kernel : I am using the kernel version 3.19 for the depiction of waitqueue functions : First there is waitqueue head and the waitqueue . The waitqueue structure is : 20 struct __wait_queue { 21 unsigned int flags; 22 void *private; 23 wait_queue_func_t func; 24 struct list_head task_list; 25 }; 12 typedef struct __wait_queue wait_queue_t ; So basically it has a…
Following documents gives many details of DMA-APIs used in Linux : Linux Documentation : Documentation/DMA-API-HOWTO.txt Documentation/DMA-API.txt Documentation/Intel-IOMMU.txt Online tutorials : http://linuxkernelhacker.blogspot.in/2014/07/arm-dma-mapping-explained.html http://www.linuxjournal.com/article/7104?page=0,0 Intel VT-D spec explaining IOMMU, IOTLB, DMA maps etc.…
Lets start with Linux scheduling : Linux Scheduling Classes : In Linux scheduling is determined by the scheduling class to which the process belong. The sched_class data structure can be found in include/linux/sched.h: All existing scheduling classes in the kernel are in a list. stop_sched_class → rt_sched_class → fair_sched_class → idle_sched_class → NULL Stop and Idle are special scheduling…