Recent content from Rambling Comments - Len Holgate's blog
Back in 2004 I started a series of blog posts, called “Practical Testing”, about testing real-world, multi-threaded code. In 2023 I did a brief precis of how we got here and, since the code is still in regular use and still evolving, it’s time to talk about some of the recent changes. This release of the code makes no functional changes to the code that we’re testing in…
Today marks the start of the 30th year of JetByte Limited. Where did the time go? We started as a ‘bum on seat’ contracting company so that Len could work for Credit Suisse Financial Products as a C++ developer back in 1997. This worked well, with interesting work, interesting people and lots of new skills to learn. When UK tax rules were changed we adjusted our working practices as…
Another year, 2026. We had an exciting 2025, with lots of interesting work on complicated projects. Our focus remains the same as ever, helping clients with high performance, reliable, server systems on Windows and Linux. This year we will begin our 30th year of software development consultancy and, it seems that the more things change, the more they stay the same. The highlight of the year was…
A few days ago one of my build machines started to have unexpected test failures during regular integration builds of one of my client’s codebases. My other build machines ran the tests fine and the failures were intermittent and spread over a worryingly large array of tests. TL;DR Don’t use CLOCK_REALTIME to measure wait or delay times as it can be changed and the time reported can…
I have been using a home-grown continuous integration system based on a hacked version of CruiseControl.Net since 2007. Whilst it’s not perfect, it works, and I have quite a bit of custom tooling that builds configurations for it for various client projects and builds of various versions of The Server Framework. TL;DR By exporting the configuration of Visual Studio as a vsconfig file you can…
We’re pleased to have resumed work with our nameless, large, American postal company, who, back in 2019 took our mail-sorting server software into an extended pilot programme. Things went well, they have over 300 servers running the software and are processing over 554 million pieces of mail a day, each one of which results in multiple messages processed by our server software. They now want…
I’ve been playing around with the low-level access to the Windows networking stack that is provided by \Device\Afd. This provides a ‘readiness’ interface rather than the ‘completion’ interface provided by traditional IOCP designs. I’ve been meaning to do some comparative performance tests, much in the same way that I did for my investigations of the RIO API, but…
Welcome to 2025. 2024 was a good year for us, we continued to work with the secretive Online Gaming Company adding to and improving the cloud gaming server that we wrote for them. It’s quite nice to have a client that uses our server code to support over 1.4 billion monthly players and to continue to extend and improve the code that we’ve now been a part of for over 16 years.
I’ve been cleaning up a continuous integration build system for my cloud gaming client’s server system. The codebase is pretty big, and my build system on Windows is pretty rubbish but works… It has been getting slower and slower due to how I build the system, and I wanted to make it faster. Whilst doing this, I ran into a Visual Studio error message that I hadn’t seen…
Last time I looked at way of using \Device\Afd to perform individual socket polling for readiness. This differed from the previous approach to using \Device\Afd, which batched up the sockets and issued a single poll for multiple sockets. The individual socket polling approach appeals to me as it would appear to scale more easily, and putting together an echo server that supports multiple…
Recently, I’be been exploring socket readiness notifications on Windows using the \Device\Afd interface. My initial forays into this were from a Linux epoll direction as the use of the \Device\Afd API provides a similar interface to the epoll API and makes it possible to build something almost the same on Windows which can help when writing cross platform code. I’d got to the point…
I’ve just had to deal with a situation where Windows 11 Explorer crashes and restarts every time I browse to a particular directory. The directory contained the source to my AFD code and I somehow managed to guess at the cause and fix the offending file… The problem was that I had some Visual Studio sln files checked in to a git repository and at some point git had changed the line…
For a while now I’ve been playing around with the low-level access to the Windows networking stack that is provided by \Device\Afd. Recently I had a comment from a reader, Matt James, who pointed me in the direction of his work over at https://github.com/Emjayen/napi which does a lot more with the \Device\Afd interface than I have. It’s well worth checking out. This code includes, as…
There’s a bug in the TCP OOB (out of band) data handling code in all versions of The Server Framework from 6.6 through to 7.4. This issue manifests as a purecall during socket destruction and is due to a race condition during the release of the last two references held on a socket. If you don’t use OOB then a simple work around is to comment out the following in TAsyncSocket.
Recently I spent quite a long time debugging a heap corruption bug. When I eventually found the issue, it was such a classic buffer overrun bug that it was actually quite laughable how long it had taken to track it down. It was just SO obvious that I immediately spotted it as soon as I looked at the code. The problem was that it took me so long to actually look at this piece of code.
There’s a lot of noise about how “Joe Biden’s government wants you to move away from using C++”. This is as a result of the US government releasing a report, Back to the building blocks: A path toward secure and measurable software which suggests: Uncle Sam says 'no!' “Memory safety vulnerabilities are a class of vulnerability affecting how memory can be accessed,…
I just broken my .htaccess file for about an hour and this resulting in /blog/ 301 redirecting to / which wasn’t intended… Fixed now.
I’ve been using the MiniDumpWriteDump() API from DbgHelp.dll for 20 years or so. It has proven to be a useful diagnostic tool, and I use it in all manner of places, including many where others may simply use an assert(). It’s a heavy-weight debugging tool, but it has proved useful over the years; rather than just throwing an exception because things that shouldn’t happen have…
Back in 2004 I wrote this, I wonder why I used to make some people unhappy… I’m starting to believe that, at 37, I must now be old and cranky because to be quite honest with you; if you’re not writing code in such a way that you define concepts and abstractions in such a way that the actual main line business logic code that you write is clear and easy to understand then…
The only difference between a client and a server is the way in which the connection is established. For a client, you create a socket and call “connect” on it. For a server, we have a socket that is “listening” for connections, “accepts” new incoming connections and returns a socket that is then indistinguishable from a client connection. In the past,…
Now that I have a reasonably easy to use event-driven socket class I can start to use it. The first thing to build is a simple client. This will connect to a server and expect that server to echo anything that is sent to it. The client will send data and read it back and compare it with what it sent. A simple, single-threaded, client Last time, I put together a tcp_socket object that made it…
I’ve been investigating the ‘sparsely documented’ \Device\Afd interface that lies below the Winsock2 layer. Today I use a test-driven method for building some code to make using this API a little easier. Using the API In my previous posts on the \Device\Afd interface I focused on exploring and understanding the API itself. Since it’s ‘sparsely documented’ I used…
Back in 2009 I started playing around with embedded assembly language on simple hardware with the intention of building a robot spider. This was hosted on a separate website, L’Hexapod.com It was a lot of fun but once my children came along the project took up too much time for me to focus on it. Back in 2010 I thought that the birth of my first son, Scott, would be just a small blip in my…
2023 was a good year. We finished off a project for our Industrial Control Client and everyone seems happy, which is always nice. Our work with the secretive Online Gaming Company is taking up most of our time as we continue to enhance the cloud gaming server that we wrote for them. They continue to go from strength to strength, which is good. We have been rolling out a series of changes into…
We’ve now reached the end of the recent embedded development project for our Industrial Control Client. The final phase was made more complicated by the difficulty in debugging the changes. The embedded hardware had no screen, and the network debug facility that it supported was unreliable; it sometimes just lost messages. So the first step was to work around this issue with some debug…
I had an internet outage this morning. This shouldn’t have been much of a problem for me as all of the code that I needed to be working on is on my local git server and all of the dependencies are local. Or so I thought. The client has a c# shim layer that builds as part of the C++ server build. They rely on NuGet to grab some components from somewhere for some reason.
The second of our “reanimation” projects has reached a significant milestone. We ran the whole new system on the real hardware last week, and it mostly works. This is a big step for us and the client as the project has been quite complex in terms of how the development has been done. As I said, our secret Industrial Control Client has had us working on a program that compiles in Visual…
When your machine has multiple NUMA nodes it’s often useful to restrict a process to using just one for performance reasons. It’s sometimes hard to fully utilize multiple NUMA nodes and, if you get it wrong, it can cost in performance as the nodes need to keep their caches consistent and potentially access memory over a slower link than the memory that is closer to the node, these…
We recently had an old client contact us with an unusual request. We last worked with VEXIS Systems Inc. back in 2010 when we extended the telephony server we’d built for them to support CLR hosting, using The Server Framework’s CLR Hosting Option. We then built a managed plugin system that integrated with the existing unmanaged system so that they could write their business logic in…
I’ve always found testing multi-threaded code in C++ a humbling experience. It’s just so easy to make stupid mistakes and for those mistakes to lurk in code until the circumstances are just right for them to show themselves. This weekend a unit test that has run thousands of times for many years started to fail. The reason for the failure was a fairly obvious race condition in the test…
I’ve had a sick PC for several weeks now. It has cost me a surprising amount of time and thought. It started with my main work machine randomly hanging. This is Windows 11 with a Ryzen 9 5900X, and it has previously run faultlessly for two years or so. The hangs were, at first, annoying and I assumed that it was some driver that had been updated and was playing up, and initially I hoped that…
Previously published This article was previously published on len-learns-rust.com. A full index of these articles can be found here. In addition to managing the lifetimes of references to variables, the Rust compiler’s borrow checker also deals with enforcing Rust’s guarantees about mutability and so helps to prevent data races. Basically, you can have any number of immutable…
Previously published This article was previously published on len-learns-rust.com. A full index of these articles can be found here. One of the jobs of the Rust compiler’s “borrow checker” is to track the life of each reference to a variable so that it can prevent dangling references. To do this, it annotates each variable and reference with details of the scope in which it is…
One of my long-term clients has hundreds of cloud machines running instances of their server, each server maintains thousands of reliable UDP connections using a custom protocol that we’ve developed over the years. When things go wrong it’s often hard to work out why. Even though we have reasonable unit test coverage of the code that runs the UDP protocol, it’s hard to build…
Previously published This article was previously published on len-learns-rust.com. A full index of these articles can be found here. One reason that access shared data using locks is a bad idea is that, in complex code, it may be possible to deadlock. At their simplest, deadlocks are caused when one thread (thread a) obtains and holds a lock which another thread (thread b) requires whilst itself…
Previously published This article was previously published on len-learns-rust.com. A full index of these articles can be found here. Since I now understand a little about how to share data between threads I can try and use my Id Manager from multiple threads. Following the same pattern as I’ve been using with the other threading code, something like this might work… #[test] fn…
Previously published This article was previously published on len-learns-rust.com. A full index of these articles can be found here. Now that we can send messages to threads I want to see how we can access shared data from those threads. This isn’t the best design choice as shared data needs to be protected by locks so that it is accessed in an atomic fashion and the various threads involved…
The manual process around updating broken links is due to be replaced by a simple link checker that I’ve been writing in Rust. It’s not quite ready yet but it’s nearly there… I was updating a few broken links today and came across this from 2004; “Software development is about discipline and detail; code quality starts to decay as soon as developers forget this. All…
Previously published This article was previously published on len-learns-rust.com. A full index of these articles can be found here. The simplest threading is already covered by most Rust books. Starting up a thread, passing stuff to it, letting it run and waiting for it to finish. Something like this is the basic thread example in Rust. This work’s and is easy to understand and reason…
Previously published This article was previously published on len-learns-rust.com. A full index of these articles can be found here. My threading background in C++ on Windows and Linux goes back a long way and that means that I have some set ways of doing things that may not map directly to the Rust way of doing things. I tend to use threads with the following patterns at present: A thread starts…
This blog has been around a long time and the internet tends to rot. This means that quite a lot of the links on old posts are broken. I’m slowly fixing these broken links to use “The Wayback Machine” but it’s complicated to automate as the resulting URLs need to include a timestamp of a valid snapshot and can’t just include a ‘rough idea of the date’. So…
On the 3rd of May 2003 I posted the first entry on this blog. I then proceeded to “back fill” the blog with various things that had either been posted before in other places or had been laying around waiting for me to have somewhere to put them. This is why although the blog began in 2003 the archives go back to 1992. What I said on the 10th anniversary of this blog is still apt:
I’ve been investigating the ‘sparsely documented’ \Device\Afd interface that lies below the Winsock2 layer. Today I use a test driven method for understanding and documenting the API. TDU - Test Driven Understanding When trying to understand a new API I always like to end up with executable documentation in the form of tests that show the behaviour of the API. I write these tests…
Yesterday I was bemoaning encapsulation and how it was hiding what was going on inside my objects (and quite right too, what good would it be otherwise?). The issue is that the object I was interested in, and each of the objects that formed it, were allocating more memory that expected. It wasn’t so much that the object was bigger than expected, just that there were more allocations than I…
I’m debugging performance issues with a C++ server that has been stalling and then failing to recover. I’ve reached a point where we can generate the problem using a network interruption that causes multiple connections to disconnect at the same time. The fixed sized thread pool that services these connections becomes overloaded with work that requires it to clean up connection objects…
Back in August 2012 I shared my scripts for building OpenSSL on Windows. These have changed a little since the ones I had for the 1.0.x and 0.9.x releases of OpenSSL. The main idea is the same, the scripts build the OpenSSL code as both static libs and DLLs for both x86 and x64 and allow you to have all of the files next to each other in the same directory by adding various ‘warts’ to…
I’ve been writing a series of blog posts, called “Practical Testing”, about testing real-world, multi-threaded code. Up until now I’ve used my own, home grown, unit testing framework. When I started out with this series back in 2004 there wasn’t much in the way of mature testing frameworks for C++ and, over the years I haven’t really found a need to switch from…
Nineteen years ago I began a series of blog posts, called “Practical Testing”, about testing real-world, multi-threaded code. As with most code that works well, and is used by lots of people, we’re still changing it and improving it and using it. I’ve just done a precis of how we got here and now it’s time to continue the journey. As I hinted at the end of the last…
I’m in the process of investigating GoogleTest and the experience has been interesting. I’ve been unit testing code and doing Test Driven Development for a long time now; almost 20 years and I’m still learning. I’ve had my own test code, a simple test framework, that has served me well but it’s not always appropriate so I’m looking for alternatives. I’ll…
So, this morning I’m back from my Easter break and working on some code for a client and the first thing I do is kick off my CI build and things start failing. It seems that my “cunning plan” to have my CI build use the preview version of Visual Studio 2022 whilst the client uses earlier versions has paid off again… We build with all warnings enabled and treat warnings as…