Recently I gave a talk at the NYAN Conference called ‘From ‘dotnet run’ to ‘hello world’ : In the talk I demonstrate how you can use PerfView to analyse where the .NET Runtime is spending it’s time during start-up : From 'dotnet run' to 'hello world' from Matt Warren This post is a step-by-step guide to that demo. Code Sample For this exercise I delibrately only look at what…
Background ‘Default Interface Methods’ (DIM) sometimes referred to as ‘Default Implementations in Interfaces’, appeared in C# 8. In case you’ve never heard of the feature, here’s some links to get you started: Default implementations in interfaces (official announcement ) Default Interface Methods (C# Language Proposal), here’s some notable sections: Diamond inheritance and classes Interface…
Over the last few years, I’ve come across more and more research papers based, in some way, on the ‘Common Language Runtime’ (CLR). So armed with Google Scholar and ably assisted by Semantic Scholar , I put together the list below. Note: I put the papers into the following categories to make them easier to navigate (papers in each category are sorted by date, newest -> oldest): Using the .NET…
As the saying goes: “All problems in computer science can be solved by another level of indirection” - David Wheeler and it certainly seems like the ‘.NET Runtime’ Engineers took this advice to heart! ‘Stubs’ , as they’re known in the runtime (sometimes ‘Thunks’ ), provide a level of indirection throughout the source code, there’s almost 500 mentions of them ! This post will explore what they are,…
Who doesn’t like a nice bit of ‘ASCII Art’? I know I certainly do! To see what Matt’s CLR was all about you can watch the recording of my talk ‘From ‘dotnet run’ to ‘Hello World!’’ (from about ~24:30 in) So armed with a trusty regex /\*(.*?)\*/|//(.*?)\r?\n|"((\\[^\n]|[^"\n])*)"|@("[^"]*")+ , I set out to find all the interesting ASCII Art used in source code comments in the following .NET related…
I’m a massive fan of everything Fabien Sanglard does, I love his blog and I’ve read both his books cover-to-cover (for more info on his books, check out the recent Hansleminutes podcast ). Recently he wrote an excellent post where he deciphered a postcard sized raytracer , un-packing the obfuscated code and providing a fantastic explanation of the maths involved. I really recommend you take the…
What is ‘stack walking’, well as always the ‘Book of the Runtime’ (BotR) helps us, from the relevant page : The CLR makes heavy use of a technique known as stack walking (or stack crawling). This involves iterating the sequence of call frames for a particular thread , from the most recent (the thread’s current function) back down to the base of the stack. The runtime uses stack walks for a number…
It seems like this time of year anyone with a blog is doing some sort of ‘advent calendar’, i.e. 24 posts leading up to Christmas. For instance there’s a F# one which inspired a C# one ( C# copying from F#, that never happens 😉) However, that’s a bit of a problem for me, I struggled to write 24 posts in my most productive year , let alone a single month! Also, I mostly blog about ‘.NET Internals’…
A little over 4 years ago Microsoft announced that they were open sourcing large parts of the .NET framework and as this slide from New Features in .NET Core and ASP.NET Core 2.1 shows, the community has been contributing in a significant way: Side-note : This post forms part of an on-going series, if you want to see how things have changed over time you can check out the previous ones: Open…
Recently I was fortunate enough to chat with Chris Bacon who wrote DotNetAnywhere ( an alternative .NET Runtime ) and I quipped with him: .. you’re probably one of only a select group (*) of people who’ve written a .NET runtime, that’s pretty cool! * if you exclude people who were paid to work on one, i.e. Microsoft/Mono/Xamarin engineers, it’s a very select group. But it got me thinking, how many…
I recently came across the excellent ‘Fuzzlyn’ project , created as part of the ‘Language-Based Security’ course at Aarhus University . As per the project description Fuzzlyn is a: … fuzzer which utilizes Roslyn to generate random C# programs And what is a ‘fuzzer’, from the Wikipedia page for ‘ fuzzing ’ : Fuzzing or fuzz testing is an automated software testing technique that involves providing…
.NET is a managed runtime , which means that it provides high-level features that ‘manage’ your program for you, from Introduction to the Common Language Runtime (CLR) (written in 2007): The runtime has many features, so it is useful to categorize them as follows: Fundamental features – Features that have broad impact on the design of other features. These include: Garbage Collection Memory Safety…
I’m constantly surprised at just how popular resources related to ‘.NET Internals’ are, for instance take this tweet and the thread that followed: If you like learning about '.NET Internals' here's a few talks/presentations I've watched that you might also like. First 'Writing High Performance Code in .NET' by Bart de Smet https://t.co/L5S9BsBlWe — Matt Warren (@matthewwarren) July 9, 2018…
I’ve been digging into .NET Internals for a while now, but never really looked closely at how the ‘ Just-in-Time ’ (JIT) compiler works. In my mind, the interaction between the .NET Runtime and the JIT has always looked like this: Nice and straight-forward, the CLR asks the JIT to compile some ‘ Intermediate Language ’ (IL) code into machine code and the JIT hands back the bytes when it’s done.…
Whether you want to look at what your code is doing ‘ under-the-hood ’ or you’re trying to see what the ‘ internals ’ of the CLR look like, there is a whole range of tools that can help you out. To give ‘ credit where credit is due ’, this post is based on a tweet , so thanks to everyone who contributed to the list and if I’ve missed out any tools, please let me know in the comments below . While…
Firstly, what exactly is CoreRT ? From its GitHub repo : .. a .NET Core runtime optimized for AOT (ahead of time compilation) scenarios, with the accompanying .NET native compiler toolchain The rest of this post will look at what that actually means. Contents Existing .NET ‘AOT’ Implementations High-Level Overview The Compiler The Runtime ‘Hello World’ Program Limitations Further Reading Existing…
It turns out that the .NET Runtime has a technical standard (or specification ), known by its full name ECMA-335 - Common Language Infrastructure (CLI) (not to be confused with ECMA-334 which is the ‘C# Language Specification’ ). The latest update is the 6th edition from June 2012 . The specification or standard was written before .NET Core existed, so only applies to the .NET Framework , I’d be…
I recently appeared on Herding Code and Stackify ‘Developer Things’ podcasts and in both cases, the first question asked was ‘ how do you figure out the internals of the .NET runtime ’? This post is an attempt to articulate that process, in the hope that it might be useful to others. Here are my suggested steps: Decide what you want to investigate See if someone else has already figured it out…
Discuss this post on HackerNews and /r/programming Before we dive into the technical details, let’s start with a quick history lesson, courtesy of Don Syme who worked on adding generics to .NET and then went on to design and implement F# , which is a pretty impressive set of achievements!! Background and History 1999 Initial research, design and planning .NET/C# Generics History: Some Photos From…
It all started with a tweet, which seemed to resonate with people: If you like reading my posts on .NET internals, you'll like all these other blogs. So I've put them together in a thread for you!! — Matt Warren (@matthewwarren) January 12, 2018 The aim was to list blogs that specifically cover .NET internals at a low-level or to put it another way, blogs that answer the question how does…
I’ve now been blogging consistently for over 2 years (~2 times per/month) and I decided it was time for my first ‘retrospective’ post. Warning this post contains a large amount of humble brags , if you’ve come here to read about ‘ .NET internals ’ you’d better check back in a few weeks, when normal service will be resumed! Overall Stats Firstly, lets looks at my Google Analytics stats for 2017,…
A little over 3 years ago Microsoft announced that they were open sourcing large parts of the .NET framework and as Scott Hanselman said in his Connect 2016 keynote , the community has been contributing in a significant way: This post forms part of an on-going series, if you want to see how things have changed over time you can check out the previous ones: Open Source .NET – 2 years later Open…
The .NET runtime (CLR) has predominantly used a just-in-time (JIT) compiler to convert your executable into machine code (leaving aside ahead-of-time (AOT) scenarios for the time being), as the official Microsoft docs say : At execution time, a just-in-time (JIT) compiler translates the MSIL into native code . During this compilation, code must pass a verification process that examines the MSIL…
If you grew up in the UK and went to school during the 1980’s or 1990’s there’s a good chance that this picture brings back fond memories: (image courtesy of Classic Acorn ) I’d imagine that for a large amount of computer programmers (currently in their 30’s) the BBC Micro was their first experience of programming. If this applies to you and you want a trip down memory lane, have a read of…
Recently I was fortunate enough to be invited to the CORESTART 2.0 conference to give a talk on Microsoft & Open Source a ‘Brave New World’ . It was a great conference, well organised by Tomáš Herceg and the teams from .NET College and Riganti and I had a great time. I encourage you to attend next years ‘Update’ conference if you can and as bonus you’ll get to see the sights of Prague! Including…
Generics in C# are certainly very useful and I find it amazing that we almost didn’t get them : What would the cost of inaction have been? What would the cost of failure have been? No generics in C# 2.0? No LINQ in C# 3.0? No TPL in C# 4.0? No Async in C# 5.0? No F#? Ultimately, an erasure model of generics would have been adopted, as for Java, since the CLR team would never have pursued a…
Recently I was listening to the excellent DotNetRocks podcast and they had Steven Sanderson (of Knockout.js fame ) talking about ‘WebAssembly and Blazor’ . In case you haven’t heard about it, Blazor is an attempt to bring .NET to the browser, using the magic of WebAssembly . If you want more info, Scott Hanselmen has done a nice write-up of the various .NET/WebAssembly projects . However, as much…
Just over a year ago Google made all the open source code on GitHub available for querying within BigQuery and as if that wasn’t enough you can run a terabyte of queries each month for free ! So in this post I am going to be looking at all the C# source code on GitHub and what we can find out from it. Handily a smaller, C# only, dataset has been made available (in BigQuery you are charged per byte…
It’s a fundamental part of .NET and can often happen without you knowing , but how does it actually work ? What is the .NET Runtime doing to make boxing possible? Note : this post won’t be discussing how to detect boxing, how it can affect performance or how to remove it (speak to Ben Adams about that!). It will only be talking about how it works . As an aside, if you like reading about CLR…
Have you ever wondered where and why the .NET Runtime (CLR) allocates memory? I don’t mean the ‘ managed ’ memory that your code allocates, e.g. via new MyClass(..) and the Garbage Collector (GC) then cleans up. I mean the memory that the CLR itself allocates, all the internal data structures that it needs to make is possible for your code to run. Note just to clarify, this post will not be…
It is something we take for granted every time we run a .NET program, but it turns out that loading a Type or class is a fairly complex process. So how does the .NET Runtime (CLR) actually load a Type? If you want the tl;dr it’s done carefully , cautiously and step-by-step Ensuring Type Safety One of the key requirements of a ‘Managed Runtime’ is providing Type Safety, but what does it actually…
Turns out that what I’d always thought of as “ Compiler magic ” or “ Syntactic sugar ” is actually known by the technical term ‘ Lowering ’ and the C# compiler (a.k.a Roslyn ) uses it extensively. But what is it? Well this quote from So You Want To Write Your Own Language? gives us some idea: Lowering One semantic technique that is obvious in hindsight (but took Andrei Alexandrescu to point out to…
Now that the CoreCLR is open-source we can do fun things, for instance find out if it’s possible to add new IL (Intermediate Language) instruction to the runtime. TL;DR it turns out that it’s easier than you might think!! Here are the steps you need to go through: Step 0 - Introduction and Background Step 1 - Add the new IL instruction to the runtime Step 2 - Make the Interpreter work Step 3 -…
A while ago I wrote about the ‘special relationship’ that exists between Strings and the CLR , well it turns out that Arrays and the CLR have an even deeper one, the type of closeness where you hold hands on your first meeting As an aside, if you like reading about CLR internals you may find these other posts interesting: The CLR Thread Pool ‘Thread Injection’ Algorithm The 68 things the CLR does…
If you’re near London at the end of April, I’ll be speaking at ProgSCon 2017 on Microsoft and Open-Source – A ‘Brave New World’ . ProgSCon is 1-day conference, with talks covering an eclectic range of topics , you’ll learn lots!! As part of a never-ending quest to explore the CoreCLR source code I stumbled across the intriguing titled ‘HillClimbing.cpp’ source file. This post explains what it does…
Whilst writing a previous blog post I stumbled across the .NET Interpreter, tucked away in the source code. Although, it I’d made even the smallest amount of effort to look for it, I’d have easily found it via the GitHub ‘magic’ file search : Usage Scenarios Before we look at how to use it and what it does, it’s worth pointing out that the Interpreter is not really meant for production code. As…
photo by Alan O’Rourke Just over 2 years ago Microsoft open-sourced the entire .NET framework, this posts attempts to provide a ‘Hitchhikers Guide’ to the source-code found in the CoreCLR GitHub repository . To make it easier for you to get to the information you’re interested in, this post is split into several parts Overall Stats ‘Top 10’ lists High-level Overview Deep Dive into Individual Areas…
Because the CLR is a managed environment there are several components within the runtime that need to be initialised before any of your code can be executed. This post will take a look at the EE (Execution Engine) start-up routine and examine the initialisation process in detail. (*) 68 is only a rough guide, it depends on which version of the runtime you are using, which features are enabled and…
Delegates are a fundamental part of the .NET runtime and whilst you rarely create them directly, they are there under-the-hood every time you use a lambda in LINQ ( => ) or a Func<T> / Action<T> to make your code more functional . But how do they actually work and what’s going in the CLR when you use them? IL of delegates and/or lambdas Let’s start with a small code sample like this: public…
Over the last few months there have been several blog posts looking at GC pauses in different programming languages or runtimes. It all started with a post looking at the latency of the Haskell GC , next came a follow-up that compared Haskell, OCaml and Racket , followed by Go GC in Theory and Practice , before a final post looking at the situation in Erlang . After reading all these posts I…
According to the NASA ‘Near Earth Object Program’ asteroid ‘ 101955 Bennu (1999 RQ36) ’ has a Cumulative Impact Probability of 3.7e-04, i.e. there is a 1 in 2,700 (0.0370%) chance of Earth impact, but more reassuringly there is a 99.9630% chance the asteroid will miss the Earth completely! But how does this relate to exceptions in the .NET runtime, well let’s take a look at the official .NET…
It’s common knowledge that reflection in .NET is slow , but why is that the case? This post aims to figure that out by looking at what reflection does under-the-hood . CLR Type System Design Goals But first it’s worth pointing out that part of the reason reflection isn’t fast is that it was never designed to have high-performance as one of its goals, from Type System Overview - ‘Design Goals and…
This post is completely inspired by (or ‘copied from’ depending on your point of view) a recent post titled JAVA PAPERS (also see the HackerNews discussion ). However, instead of looking at Java and the JVM, I’ll be looking at references to research papers in the .NET language, runtime and compiler source code . If I’ve missed any that you know of, please leave a comment below! Note: I’ve…
A little over 2 years ago Microsoft announced that they were open sourcing large parts of the .NET framework and as Scott Hanselman said in his recent Connect keynote , the community has been contributing in a significant way: You can see some more detail on this number in the talk ‘What’s New in the .NET Platform’ by Scott Hunter: This post aims to give more context to those numbers and allow you…
Well it turns out that it’s a really nice example of collaboration between the main parts of the .NET runtime, here’s a list of all the components involved: Compiler JITter CLR Garbage Collector (GC) Now you could argue that all of these are required to execute any C# code, but what’s interesting about the fixed keyword is that they all have a specific part to play. Compiler To start with let’s…
The dotnet CLI tooling comes with several built-in cmds such as build , run and test , but it turns out it’s possible to add your own verb to that list. Arbitrary cmds From Intro to .NET Core CLI - Design The way the dotnet driver finds the command it is instructed to run using dotnet {command} is via a convention; any executable that is placed in the PATH and is named dotnet-{command} will be…
What’s the problem with LINQ? As outlined by Joe Duffy , LINQ introduces inefficiencies in the form of hidden allocations , from The ‘premature optimization is evil’ myth : To take an example of a technology that I am quite supportive of, but that makes writing inefficient code very easy, let’s look at LINQ-to-Objects. Quick, how many inefficiencies are introduced by this code? int [] Scale ( int…
In the CLR strings are stored as a sequence of UTF-16 code units, i.e. an array of char items. So if we have the string ‘testing’, in memory it looks like this: But look at all those zero’s, wouldn’t it be more efficient if it could be stored like this instead? Now this is a contrived example, clearly not all strings are simple ASCII text that can be compacted this way. Also, even though I’m an…
In which we use System.Runtime.CompilerServices.Unsafe a generic API (“type-safe” but still “unsafe”) and mess with the C# Type System! The post covers the following topics: What it is and why it’s useful How it works Code samples Tricks you can do with it Using it safely What it is and why it’s useful The XML documentation comments for System.Runtime.CompilerServices.Unsafe state that it:…
If you’ve ever spent time debugging .NET memory dumps in WinDBG you will be familiar with the commands shown below, which aren’t always the most straight-forward to work with! However back in May 2013 Microsoft released the CLR MD library , describing it as: … a set of advanced APIs for programmatically inspecting a crash dump of a .NET program much in the same way as the SOS Debugging Extensions…