In November 2025 at Pycon France (at Lyon), I watched the talk "Build a frozendict type (immutable dictionary)" by Antoine Rozo ( video in French). It was a funny talk building a frozendict type in pure Python exploring different implementations such as a frozen dataclass with slots, frozenset , and super . See …
I'm writing an article series on Free Threading internals to learn more about Free Threading, explain how it works, and explain how it solved the "remove the GIL" issue where previous attempts failed. Reference counting Deferred reference counting PyMutex (this article) PEP 703 replaces the unique Global Interpreter Lock (GIL …
I'm writing an article series on Free Threading internals to learn more about Free Threading, explain how it works, and explain how it solved the "remove the GIL" issue where previous attempts failed. Reference counting Deferred reference counting (this article) PyMutex In the previous article , we have seen how the …
I'm writing an article series on Free Threading internals to learn more about Free Threading, explain how it works, and explain how it solved the "remove the GIL" issue where previous attempts failed. Reference counting (this article) Deferred reference counting PyMutex This first article explores the problem of tracking objects …
In the Python C API, I dislike APIs modifying immutable objects such as _PyBytes_Resize() . I designed a whole new PyBytesWriter API to replace this _PyBytes_Resize() function. As usual in Python, it took multiple iterations and one year to design the API and to reach an agreement. Picture: The Secret World …
Design an API can take time. This article describes the design of the C API to import and export Python integers. It takes place between August 2023 and December 2024. In total, the discussions got more than 448 messages! The API is a thin abstraction on top of CPython implementation …
Here is a report on my 18 commits merged into Python in February 2025: Reorganize C API tests Use PyErr_FormatUnraisable() Reorganize includes C API: Remove PySequence_Fast() C API: Fix function signatures C API: Deprecate private _PyUnicodeWriter Documentation Misc changes Painting: Le Déjeuner des Canotiers (1881) by Auguste Renoir . gh-93649: Reorganize …
PEP 741 story Sometimes, writing a PEP can be a wild ride. It took two whole years between the early discussions and getting PEP 741 eventually accepted by the Steering Council. The API is only made of 18 functions, but it took more than 200 messages to design properly these …
In May, I designed a new C API to build a Python str object: the PyUnicodeWriter API . Many people were involved in the design and the discussion was quite long. The C API Working Group helped to design a better and more convenient API. It took me basically a whole …
In Python 3.13, I made multiple enhancements to make the limited C API more usable: Add 14 functions to the limited C API. Make the special debug build Py_TRACE_REFS compatible with the limited C API. Enhance Argument Clinic to generate C code using the limited C API. Add an …
In Python 3.13 alpha 1, I removed more than 300 private C API functions. Even if I announced my plan early in July, users didn't "embrace" my plan and didn't agree with the rationale. I reverted 50 functions in the alpha 2 release to calm down the situation and …
Last June, I proposed adding a new PyDict_GetItemRef() function to Python 3.13 C API. Every aspect of the API design was discussed in length. I will explain how the API was designed, to finish with the future creation of C API Working Group. Photo: Psyche Revived by Cupid's Kiss …
In 2023, between May 4 and July 8, I made 144 commits in the Python main branch. In this article, I describe the most important Python contributions that I made to Python 3.12 and Python 3.13 in these months. Drawing: Foxes in Love: Cuddle Summary Add PyImport_AddModuleRef() and …
Drawing: "L'oeil du cyclone" by Théo Grosjean. Convert macros to functions For 4 years, between Python 3.7 (2018) and Python 3.12 (2022), I made many changes on macros in the Python C API to make the API less error prone (avoid macro pitfalls ) and better define the API …
This morning, I got this email from the buildbot-status mailing list: The Buildbot has detected a new failure on builder PPC64LE Fedora Rawhide Refleaks 3.x while building Python. I get many of buildbot failures per month (by email), but I like to debug reference leaks: they are more challenging …
The PyObject structure prevents indirectly to optimize CPython. We will see why and how I prepared the C API to make this structure opaque. It took me 1 year and a half to add functions and to introduce incompatible C API changes (fear!). In February 2020, I started by adding …
I'm trying to enhance and to fix the Python C API for 5 years. My first goal was to shrink the C API without breaking third party C extensions. I hid many private functions from the public functions: I moved them to the "internal C API". I also deprecated and …
In 2020, I created a new pythoncapi_compat project to add Python 3.10 support to C extensions without losing support for old Python versions. It supports Python 2.7-3.10 and PyPy 2.7-3.7. The project is made of two parts: pythoncapi_compat.h : Header file providing new C API …
This article is about changes that I made, with the help other developers, in the Python C API in Python 3.8, 3.9 and 3.10 to avoid accessing structures members: prepare the C API to make structures opaque . These changes are related to my PEP 620 "Hide implementation …
This article is about the work done in Python in 2019 and 2020 to better isolate subinterpreters. Static types are converted to heap types, extension modules are converted to use the new multiphase initialization API (PEP 489), caches, states, singletons and free lists are made per-interpreter, many bugs have been …
This article is the history of Python C API discussions over the last 4 years, and the creation of C API projects: pythoncapi website , pythoncapi_compat.h header file and HPy (new clean C API) . More and more people are aware of issues caused by the C API and are working …
This article is about old reference leaks discovered or caused by the work on isolating subinterpreters: leaks in 6 different modules (gc, _weakref, _abc, _signal, _ast and _thread). Refleaks buildbot failures With my work on isolating subinterpreters, old bugs about Python objects leaked at Python exit are suddenly becoming blocker …
My previous article Daemon threads and the Python finalization in Python 3.2 and 3.3 introduces issues caused by daemon threads in the Python finalization and past changes to make them work. This article is about bugfixes of the infamous GIL (Global Interpreter Lock) in Python 3.9, between …
This article is about a race condition in threading shutdown that I fixed in Python 3.9 in March 2019. I also forbid spawning daemon threads in subinterpreters to fix another related bug. Drawing: #CoronaMaison by Julien Neel . Race condition in threading shutdown Random test failure noticed on FreeBSD buildbot …
At exit, the Python finalization calls Python objects finalizers (the __del__() method) and deallocates memory. The daemon threads are a special kind of threads which continue to run during and after the Python finalization. They are causing race conditions and tricky bugs in the Python finalization. This article covers bugs …
This article describes the discussion on the design of the development mode (-X dev) that I added to Python 3.7 and how it has been implemented. The development mode enables runtime checks which are too expensive to be enabled by default. It can be enabled by python3 -X dev …
Keeping Python competitive I'm trying to find ways to make Python more efficient for many years, see for example my discussion at the Language Summit during Pycon US 2017: Keeping Python competitive (LWN article); slides . At EuroPython 2019 (Basel), I gave the keynote "Python Performance: Past, Present and Future": slides …
After explaining how to Debug Hybrid Graphics issues on Linux , here is the story of four graphics bugs that I had in GNOME and Firefox on my Fedora 30 between May 2018 and September 2019: bugs in gnome-shell, Gtk, Firefox and mutter. gnome-shell freezes In May 2018, six months after …
Hybrid Graphics is a complex hardware and software solution to achieve longer laptop battery life: an integrated graphics device is used by default, and a discrete graphics device with higher graphics performances is enabled on demand. If it is designed and implemented carefully, users should not notice that a laptop …
In September 2017, during the CPython sprint at Facebook, I proposed my idea to create A New C API for CPython . I'm still working on the Python C API at: pythoncapi.readthedocs.io . My analysis is that the C API leaks too many implementation details which prevent to optimize Python …
I added a new sys.unraisablehook function to allow to set a custom hook to control how "unraisable exceptions" are handled. It is already testable in Python 3.8 beta1 , released last week! An "unraisable exception" is an error which happens when Python cannot report it to the caller. Examples …
I fixed multiple bugs in asyncio ProactorEventLoop previously. But test_asyncio still failed sometimes. I noticed a memory leak in test_asyncio which will haunt me for 1 year in 2018... Yet another example of a test failure which looks harmless but hides a critical bug. The bug is that sending a …
In December 2017, Yury Selivanov pushed the long awaited start_tls() function. A newly added test failed on Windows. Later, the test started to fail randomly on Linux as well. In fact, it was a well hidden race condition in the asynchronous handshake of SSLProtocol which will take 5 months of …
Between December 2014 and January 2015, once I succeeded to fix the root issue of the random asyncio crashes on Windows ( Proactor Cancellation From Hell ), I fixed more race conditions and bugs in ProactorEventLoop : ConnectPipe() Race Condition Race Condition in BaseSubprocessTransport._try_finish() Close the transport on failure: ResourceWarning Cleanup code …
Between 2014 and 2015, I was working on the new shiny asyncio module (module added to Python 3.4 released in March 2014). I helped to stabilize the Windows implementation because... well, nobody else was paying attention to it, and I was worried that test_asyncio randomly crashed on Windows. One …
This article describes a few locales bugs that I fixed in Python 3 between 2012 (Python 3.3) and 2018 (Python 3.7): Support non-ASCII decimal point and thousands separator Crash with non-ASCII decimal point LC_NUMERIC encoding different than LC_CTYPE encoding LC_MONETARY encoding different than LC_CTYPE encoding Tests non-ASCII locales …
Recently, I worked on a change which looked simple: move the code to initialize the sys.stdout encoding before Py_Initialize() . While I was on it, I also decided to move the code which selects the Python "filesystem encoding". I didn't expect that I would spend 2 weeks on these issues …
Since Python 3.0 was released in 2008, each time an user reported an encoding issue, someone showed up and asked why Python does not "simply" always use UTF-8. Well, it's not that easy. UTF-8 is the best encoding in most cases, but it is still not the best encoding …
During the childhood of Python 3, encodings issues were common, even on well configured systems. Python used UTF-8 rather than the locale encoding, and so commonly produced mojibake . For these reasons, when users complained about the Python behaviour with the POSIX locale, bug reports were closed with a message like …
September 2016, a few days before the CPython core dev sprint, Steve Dower proposed two major backward incompatible changes for Python 3.6 on Windows: PEP 528: Change Windows console encoding to UTF-8 and PEP 529: Change Windows filesystem encoding to UTF-8 . At the first read, I was sure that …
Between Python 3.0 released in 2008 and Python 3.4 released in 2014, the Python filesystem encoding changed multiple times. It took 6 years to choose the best Python filesystem encoding on each platform. I have been officially promoted as a core developer in January 2010 by Martin von …
In my previous article, I wrote that os.listdir(str) ignored silently undecodable filenames in Python 3.0 and that lying on the real content of a directory looks like a very bad idea. Martin v. Löwis found a very smart solution to this problem: the surrogateescape error handler. This …
Ten years ago, when Python 3.0 final was released, os.listdir(str) ignored silently undecodable filenames : $ python3.0 >>> os.mkdir(b'x') >>> open(b'x/nonascii\xff', 'w').close() >>> os.listdir('x') [] You had to use bytes to see all filenames: >>> os.listdir(b'x') [b'nonascii\xff'] If the locale is POSIX …
It took me 4 years to fix a nasty bug in the famous Python GIL (Global Interpreter Lock), one of the most critical part of Python. I had to dig the Git history to find a change made 26 years ago by Guido van Rossum : at this time, threads were …
Thanks to my latest change on time.perf_counter() , all Python 3.7 clocks now use nanoseconds as integer internally. It became possible to propose again my old idea of getting time as nanoseconds at Python level and so I wrote a new PEP 564 "Add new time functions with nanosecond …
Since 2012, I have been trying to convert all Python clocks to use internally nanoseconds. The last clock which still used floating point internally was time.perf_counter() . INADA Naoki's new importtime tool was an opportunity for me to have a new look on a tricky integer overflow issue. Modify importtime …
My contributions to CPython during 2017 Q3 (july, august, september), Part 3 (funny bugs). Previous report: My contributions to CPython during 2017 Q3: Part 2 (dangling threads) . Summary: FreeBSD bug: minor() device regression regrtest snowball effect when hunting memory leaks Bugfixes Other Changes FreeBSD bug: minor() device regression bpo-31044 : The …
My contributions to CPython during 2017 Q3 (july, august, september), Part 2: "Dangling threads". Previous report: My contributions to CPython during 2017 Q3: Part 1 . Next reports: My contributions to CPython during 2017 Q3: Part 3 (funny bugs) . Summary: Bugfixes: Reference cycles socketserver leaking threads and processes test_logging random bug …
My contributions to CPython during 2017 Q3 (july, august, september), Part 1. Previous report: My contributions to CPython during 2017 Q2 (part1) . Next reports: My contributions to CPython during 2017 Q3: Part 2 (dangling threads) . My contributions to CPython during 2017 Q3: Part 3 (funny bugs) . Summary: Statistics Security fixes …
I am working on the Python security for years, but I never wrote anything about that. Let's fix this! PSRT I am part of the Python Security Response Team (PSRT): I get emails sent to security@python.org . I try to analyze each report to validate that the bug is …