Bit Fields in C++
Andreas Hohmann February 25, 2025 #c++ #bitfield #templatesWhether we are dealing with network protocols or device drivers, system-level programming often involves bit fields. As a system programming language, C offers a concise syntax to specify bit fields in C strctures, for example
struct CStylePacket {
u16 a : 3;
u16 b : 9;
u16 c : 2;
};Here and in the following examples we are using the short type names popularized by Rust:
using u32 = std::uint32_t;
using u16 = std::uint16_t;
using u8 = std::uint8_t;How are these bit fields mapped to memory? This unfortunately depends on the compiler and processor architecture, and there is no standard way to define the layout precisely.
Most compilers choose the layout depending on the endianness of the target processor architecture. For little-endian architectures (such as Intel's x86), the bit fields are assigned from least to most significant bit ("right to left"), and for big-endian architectures such as ARM, the fields are arranged in the opposite direction from most to least significant bit ("left to right").
On my Intel desktop, I can demonstrate the little-endian layout with the following unit test (using gtest):
TEST(BitFieldTest, SetsBitFields) {
CStylePacket p{.a = 5, .b = 300, .c = 2};
EXPECT_EQ(sizeof(p), 2);
std::span<u8> data(reinterpret_cast<u8 *>(&p), sizeof(p));
EXPECT_EQ(data.size(), 2);
EXPECT_EQ(p.a, 0b101);
EXPECT_EQ(p.b, 0b100101100);
EXPECT_EQ(p.c, 0b10);
EXPECT_EQ(data[0], 0b01100101);
EXPECT_EQ(data[1], 0b00101001);
EXPECT_THAT(data, ElementsAre(0b01100101, 0b00101001));
}Specifications, in particular in networking, typically define the bit layout in big-endian order (also known as network order). Here is the layout of an IPv4 header (omitting the options):
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| IHL |Type of Service| Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Identification |Flags| Fragment Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Time to Live | Protocol | Header Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Destination Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+If we want to map a C structure to such a network packet, we have to define the
bit fields twice using a C macro that checks the endianness. The Linux IP
header ip.h contains the following iphdr structure:
struct iphdr {
#if defined(__LITTLE_ENDIAN_BITFIELD)
__u8 ihl:4,
version:4;
#elif defined (__BIG_ENDIAN_BITFIELD)
__u8 version:4,
ihl:4;
#else
#error "Please fix <asm/byteorder.h>"
#endif
__u8 tos;
__be16 tot_len;
__be16 id;
__be16 frag_off;
__u8 ttl;
__u8 protocol;
__sum16 check;
__struct_group(/* no tag */, addrs, /* no attrs */,
__be32 saddr;
__be32 daddr;
);
/*The options start here. */
};In addition to the bitfield order, we also have to consider the endianness when
reading multi-byte integers. The Posix standard defines the hton (host to
network) and ntoh (network to host) functions (e.g., htons) for this
purpose.
Another problem is the alignment of the mapped structure. If the byte array containing the data (for example, a packet) is not properly aligned, we have to copy the data to a new structure.
This is all managable but definitely lowers the signal-noise ratio of low-level code and invites subtle errors. Can we do better with C++? We would like to take some raw data (given as a byte array) and view it according to a precise layout with bit fields. It should be possible to get and set these fields (directly in the raw data) using normal field accessor syntax and without having to worry about endianness or alignment.
The first step towards this end is a way to read and write sequences of bits as integers. As we are mainly interested in networking, we stick to network (big-endian) order. The following function reads a sequence of bits from a byte array.
// Returns the value of the `data` bits from `bit_first` to `bit_end` (inclusive).
//
// The bits are interpreted in network order. Bit 0 is the most significant
// bit of the first byte (data[0]), bit 8 is the most significant bit of the
// second byte (data[1]), and so forth.
template <std::unsigned_integral V = u32>
constexpr V get_bitfield(u8 const *const data, u32 bit_first, u32 bit_last) {
// Index of the byte containing the first bit.
u32 i_first = bit_first / 8;
// Bits of first byte containing the bit range.
u8 mask_first = 0xff >> (bit_first % 8);
// Index of the byte containing the last bit.
u32 i_last = bit_last / 8;
// Number of bits following the bit range in the last byte.
u8 s_last = 7 - bit_last % 8;
V x = data[i_first] & mask_first;
for (int i = i_first + 1; i <= i_last; ++i) {
x = (x << 8) | data[i];
}
return x >> s_last;
}I use a closed range from bit_first to bit_last because that's how bit
fields are typically specified (as in the IP header example above). After
determining the indices of the first and last byte containing bits of the range,
the only tricky part is the masking of the bits in the first and last byte. The
latter is done by shifting the result in the last line.
Note that I kept the clearer arithmetic division and modulo operation in the hope that the compiler will replace them with the more efficient shift operations.
Writing to a bit field is only slightly more complicated. The main difference is
that we start with the last byte. The set_masked helper function is used for
setting the bits in the first and last byte (which is the same if the bitfield
is contained in a single byte).
// Copies the `mask` bits of the `value` to the `data`.
template <std::unsigned_integral V = u32>
constexpr void set_masked(u8& data, u8 mask, V value) {
data = (data & ~mask) | (value & mask);
}
// Sets `data` bits from `bit_first` to `bit_last` (inclusive) to `value`.
//
// The bits are interpreted in network order. Bit 0 is the most significant
// bit of the first byte (data[0]), bit 8 is the most significant bit of the
// second byte (data[1]), and so forth.
template <std::unsigned_integral V = u32>
constexpr void set_bitfield(u8* const data, u32 bit_first, u32 bit_last, V value) {
// index of the byte containing the first bit
u32 i_first = bit_first / 8;
// bits of the first byte containing the bit range
u8 mask_first = 0xff >> (bit_first % 8);
// index of the byte containing the last bit
u32 i_last = bit_last / 8;
// number of bits following the bit range in the last byte
u8 s_last = 7 - bit_last % 8;
// bits of the last bytes containing the bit range
u8 mask_last = (0xff << s_last) & 0xff;
// shift value so that it aligns with the data bits
value <<= s_last;
// single byte case
if (i_first == i_last) {
set_masked(data[i_first], mask_first & mask_last, value);
return;
}
// set last byte
set_masked(data[i_last], mask_last, value);
value >>= 8;
// set intermediate bytes
for (int i = i_last - 1; i > i_first; --i) {
data[i] = value & 0xff;
value >>= 8;
}
// set first byte
set_masked(data[i_first], mask_first, value);
}If we want to define a structure using bitfields, the bit range will be known at
compile time. The parameters bit_first and bit_last can become template
parameters, and we can sprinkle the code with some constexpr modifiers to show
which operations are performed at compile time.
template <u32 bit_first, u32 bit_last, std::unsigned_integral V = u32>
requires(bit_first <= bit_last)
V get_bitfield_tmpl(u8 const *const data) {
// index of the byte containing the first bit
constexpr u32 i_first = bit_first / 8;
// bits of first byte containing the bit range
constexpr u8 mask_first = 0xff >> (bit_first % 8);
// index of the byte containing the last bit
constexpr u32 i_last = bit_last / 8;
// number of bits following the bit range in the last byte
constexpr u8 s_last = 7 - bit_last % 8;
V x = data[i_first] & mask_first;
for (int i = i_first + 1; i <= i_last; ++i) {
x = (x << 8) | data[i];
}
return x >> s_last;
}Besides the constexpr expressions, the compiler will also unroll the for-loop
or drop it completely. In the setter function, the only if-statement can also
be evaluated at compile time.
template <u32 bit_first, u32 bit_last, std::unsigned_integral V = u32>
requires(bit_first <= bit_last)
void set_bitfield_tmpl(u8 *const data, V value) {
// index of the byte containing the first bit
constexpr u32 i_first = bit_first / 8;
// bits of the first byte containing the bit range
constexpr u8 mask_first = 0xff >> (bit_first % 8);
// index of the byte containing the last bit
constexpr u32 i_last = bit_last / 8;
// number of bits following the bit range in the last byte
constexpr u8 s_last = 7 - bit_last % 8;
// bits of the last bytes containing the bit range
constexpr u8 mask_last = (0xff << s_last) & 0xff;
// shift value so that it aligns with the data bits
value <<= s_last;
// single byte case
if constexpr (i_first == i_last) {
set_masked(data[i_first], mask_first & mask_last, value);
return;
}
// set last byte
set_masked(data[i_last], mask_last, value);
value >>= 8;
// set intermediate bytes
for (int i = i_last - 1; i > i_first; --i) {
data[i] = value & 0xff;
value >>= 8;
}
// set first byte
set_masked(data[i_first], mask_first, value);
}These two functions take care of accessing the bits in network order, but don't give us the easy field access yet. C++ usually provides ways to define custom objects that look like the built-in objects. For fields, we can take advantage of the ability to override conversion and assignment operators. A first solution for a "field object" could look like this:
template <u32 bit_first, u32 bit_last, std::unsigned_integral V = u32>
requires(bit_first <= bit_last)
class PtrBitField {
public:
explicit PtrBitField(u8 *const data) : data_(data) {}
operator V() const { return get_bitfield_tmpl<bit_first, bit_last, V>(data_); }
PtrBitField &operator=(V value) {
set_bitfield_tmpl<bit_first, bit_last, V>(data_, value);
return *this;
}
private:
u8 *const data_;
};Using this class, we can define a packet view that lets us act on the bit fields as if they were normal fields.
class Packet {
public:
explicit Packet(u8 *const data) : a(data), b(data) {}
PtrBitField<0, 2, u16> a;
PtrBitField<3, 8, u16> b;
};
TEST(PacketTest, GetsBitFields) {
u8 data[]{0xc7, 0x96};
Packet p(data);
EXPECT_EQ(sizeof(p), 2 * sizeof(u8 *));
EXPECT_EQ(data[0], 0xc7);
EXPECT_EQ(p.a, 6);
EXPECT_EQ(p.b, 15);
p.a = 5;
EXPECT_EQ(p.a, 5);
EXPECT_EQ(p.b, 15);
p.b = 17;
EXPECT_EQ(p.a, 5);
EXPECT_EQ(p.b, 17);
EXPECT_EQ(data[0], 0xa8);
EXPECT_EQ(data[1], 0x96);
}There is one problem, however. Each field object has its own copy of the data
pointer so that the size of the Packet class is proportional to the number of
fields. To work around this, we can employ a trick and turn the packet structure
into a union (see, for example, BitFieldMember).
The following ThinBitField assumes that it uses the same memory as a type T
that has data() methods returning a pointer to the (const and non-const) byte
array, that is, a type satisfying the concept
template <typename T>
concept DataHolder = requires(T &a, T const &b) {
{ a.data() } -> std::convertible_to<u8 *>;
{ b.data() } -> std::convertible_to<u8 const *>;
{ T::data_size } -> std::convertible_to<u32>;
};We also include a data_size constant so that we can check the start and end
of the bit fields at compile time. It's up to the type T to make sure
(possibly at runtime) that the data has at least as many bytes.
Unfortunately, we cannot use the concept in the follwoing ThinBitField
template, because the type is not fully defined when we use the fields.
template <typename /* DataHolder */ T, u32 bit_first, u32 bit_last,
std::unsigned_integral V = u32>
requires(bit_first <= bit_last && bit_last < T::data_size * 8)
class ThinBitField {
public:
operator V() const {
return get_bitfield_tmpl<bit_first, bit_last, V>(data());
}
ThinBitField &operator=(V value) {
set_bitfield_tmpl<bit_first, bit_last, V>(data(), value);
return *this;
}
private:
u8 *data() { return reinterpret_cast<T *>(this)->data(); }
u8 const *data() const { return reinterpret_cast<T const *>(this)->data(); }
};The packet view is now defined as a union so that all fields overlap the memory of the packet object:
union ThinPacket {
public:
constexpr static u32 data_size = 2;
explicit ThinPacket(std::span<u8> data) : data_(data) {
assert(data.size() >= data_size);
}
// `data()` getters have to be public to be visible from `ThinBitField`. One
// cannot declare a partially specialized class template as a friend.
u8 const *data() const { return data_.data(); }
u8 *data() { return data_.data(); }
ThinBitField<ThinPacket, 0, 2, u16> a;
ThinBitField<ThinPacket, 3, 8, u16> b;
private:
std::span<u8> data_;
};The usage looks the same:
TEST(ThinPacket, AccessesBitFields) {
u8 data[]{0xc7, 0x96};
ThinPacket p(data);
EXPECT_EQ(sizeof(p), sizeof(u8 *));
EXPECT_EQ(p.a, 6);
EXPECT_EQ(p.b, 15);
p.a = 4;
p.b = 23;
EXPECT_EQ(p.a, 4);
EXPECT_EQ(p.b, 23);
}As a larger example, let's define the IP packet without the preprocessor directives and without the need to call the hton/ntoh functions:
union ThinIpPacket {
public:
constexpr static u32 data_size = 20;
explicit ThinIpPacket(u8* const data) : data_(data) {}
u8 const* data() const { return data_; }
u8* data() { return data_; }
template <int bit_start, int bit_end>
using BitField = ThinBitField<ThinIpPacket, bit_start, bit_end>;
// IP version
BitField<0, 3> version;
// internet header length in 4-byte words (4 bits, length up to 15*4 = 60
// bytes)
BitField<4, 7> ihl;
// differentiated services code point
BitField<8, 13> dscp;
// explicit congestion notification
BitField<14, 15> ecn;
// total length
BitField<16, 31> len;
// identification
BitField<32, 47> id;
// flags
BitField<48, 50> flags;
// fragment offset, given in units of 8 bytes (all but the last fragment have
// a length that's divisible by 8), 13 bits, max fragment offset therefore
// 65528
BitField<51, 63> fo;
// time to live (hop count)
BitField<64, 71> ttl;
// protocol
BitField<72, 79> protocol;
// header checksum
BitField<80, 95> chk;
// source address
BitField<96, 127> src;
// destination address
BitField<128, 159> dst;
private:
u8* const data_;
};
static_assert(DataHolder<ThinIpPacket>);
TEST(ThinIpPacketTest, AccessesBitFields) {
// clang-format off
//
// see https://pressbooks.howardcc.edu/cmsy164/chapter/packet-analysis-ip-headers-tools-and-notes/
u8 data[]{
/* 00 */ 0x45, 0x00, 0x01, 0x23, 0x50, 0x6a, 0x00, 0x00,
/* 08 */ 0x40, 0x11, 0x8d, 0x0f, 0xc0, 0xa8, 0x0d, 0x01,
/* 10 */ 0xc0, 0xa8, 0x0d, 0xff, 0xd6, 0x83, 0xd6, 0x83,
/* 18 */ 0x01, 0x0f, 0x63, 0x13, 0x00, 0x73, 0x79, 0x73,
/* 20 */ 0x61, 0x64, 0x6d, 0x69, 0x6e, 0x73, 0x2d, 0x4d,
/* 28 */ 0x61, 0x63, 0x2d, 0x50, 0x72, 0x6f, 0x2e, 0x6c,
/* 30 */ 0x6f, 0xf3, 0x61, 0x6c, 0x00, 0x31, 0xef, 0x6d,
/* 38 */ 0xff, 0x7f, 0x00, 0x00, 0x10, 0xf2, 0x87, 0xa7,
/* 40 */ 0x91, 0x7f, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00,
/* 40 */ 0x00, 0x00, 0x00, 0x00, 0x10, 0x37, 0xc3, 0x01,
/* 50 */ 0x00, 0x70,
};
// clang-format on
ThinIpPacket p(data);
EXPECT_EQ(p.version, 4);
EXPECT_EQ(p.ihl, 5);
EXPECT_EQ(p.dscp, 0);
EXPECT_EQ(p.len, 0x0123);
EXPECT_EQ(p.ttl, 64);
EXPECT_EQ(p.protocol, 17); // UDP
EXPECT_EQ(p.chk, 0x8d0f);
EXPECT_EQ(p.src, 0xc0a80d01);
EXPECT_EQ(p.dst, 0xc0a80dff);
}I'm not claiming that this is the best way to deal with bit-level protocols, but the relatively straightforward code shows that it's possible to access bit fields in C++ in a way that looks like C bit fields but avoids their endianness and alignment issues.