Part one and some articles in between. Transactions and transaction lifecycles are complicated. Not to go into the craziness of payment card systems, we can say that all transactions consist of multiple transfers for fees, increasing and checking various limits, and may have reversals. Here, a transaction is a collection of individual Transfers . Similar to accounts, transfers can be linked…
Of course, TigerBeetle does not provide anything for locking, but I don’t want to introduce a control-plane database or add Redis to the mix, so let’s build some new nails for the TigerBeetle hammer. First, we will need a dummy account to use as the second leg for the double-entry accounting. account = tb . Account ( id = 43 , ledger = 42 , code = 42 , ) errors = client . create_accounts ([…
Yes, it fills your stomach with calories quickly and cheaply. No, a few fast food meals will not kill you. And yes, you can stay healthy longer by hand-picking ingredients and declining each “Would you like to supersize that?” But pushing it so much is a bit strange: Look, I got my food in 3 minutes and didn’t even have to leave the car. The foodservice industry is cooked now.
Could not keep it under the rug until April Fool’s Day TigerBeetle is a reliable, fast, and highly available database for financial accounting. It tracks financial transactions or anything else that can be expressed as double-entry bookkeeping , providing three orders of magnitude more performance and guaranteeing durability even in the face of network, machine, and storage faults. Continuing my…
TigerBeetle is a database built for financial accounting, and the only record types available are Accounts and Transfers . That might be enough for the simplest accounting setup, but not for any realistic financial product. The way TigerBeetle solves that is by requiring an Online General Purpose (OLGP) database in the control plane that stores metadata and mapping between TigerBeetle’s…
I am a huge fan of System V Inter Process Communication primitives . There is some rawness and UNIX spirit to them. There is a newer and kinda “improved” version of those primitives named POSIX IPC. While there are a few things in POSIX IPC that can’t be done with System V IPC, most of the time it’s the other way around. Primarily due to the rawness of System V IPC. Let’s check the POSIX…
I had a great experience speaking about contactless payment cards at BSides Krakow . For those who want to get their hands dirty: Slides are here and here are the code snippets .
We had a requirement to display the last N transactions on the ATM screen (”mini-statement”). The simplest solution is to keep the list of transactions, order them by date, and take the newest N transactions. But it gets tricky once you realize there are active customers making several transactions per day and inactive ones who use the card occasionally and might make a transaction every couple of…
I was investigating performance issues of a Django application running with Gunicorn behind a Nginx server. First, I added more timing information to Nginx access.log: log_format timing '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" rt="$request_time" uct="$upstream_connect_time" uht="$upstream_header_time"…
Many go to Martin Fowler for microservice architecture, distributed systems, micro frontends, event sourcing, and other fancy ideas about architecture, but a few have noticed the advice to do a monolith first : As I hear stories about teams using a microservices architecture, I’ve noticed a common pattern. Almost all the successful microservice stories have started with a monolith that got too big…
We have given up on distributed transactions (2PC) but have not given up working with multiple resources like the database, message brokers, and queues. Instead, everybody tries to build their own atomic operations over multiple resources. Some do code&pray, and others try to have a database as a source of truth with the transactional outbox pattern . django-taskq uses the Django database as the…
In a world full of Uncles Bob be John Ousterhout More of method length, comment and TDD discussion and A Philosophy of Software Design should be on every bookshelf. It has some critcism but so does everything.
While on the topic of On-Line Transaction Processing Benchmarks , it’s interesting to observe the strategies companies employ to achieve optimal results. All code for both the transaction monitor and database is available in the PDF report. Let’s look at the Oracle one that uses Oracle Tuxedo and the Oracle database There’s a lot of cryptic C code using the OCI interface and PL/SQL code blocks.…
When examining the On-Line Transaction Processing Benchmark , most people focus on the performance numbers and the database software. But there is another column named “TP Monitor” that lists the transaction monitor software. Before cloud-scale systems took over, the best performance numbers were achieved with Oracle Tuxedo (or BEA Tuxedo, before Oracle acquired it). The good results of Oracle…
Casey Muratori had this to say : You should never take library design advice from anyone who hasn’t had to make a living selling a library in a competitive arena. I will rephrase it slightly and put it on the wall: “You should never take software design advice from anyone who hasn’t had to make a living selling software in a competitive arena.” “Never take” might be too strong but at least be…
There are several JSON library benchmarks like this one where the built-in Python json library is not the worst performer but there are several libraries that have even up to 4x better result. Some of that might no longer be true for the latest versions of Python. But since I know a bit of Python JSON encoder, I decided to find out why there is a difference. One of the answers is that the Python…
When you do need thousands and tens of thousands of transactions per second, TigerBeetle is there for you. A few hundred per second you can do it yourself in ~25 lines of code . And you can reach thousands but it requires an understanding of databases and sometimes creative approaches. A double-entry entry transaction can be done with 2 SQL statements in a single database transaction: You insert…
I was listening to a talk about finding a PAN in a haystack and this slide came up with performance numbers in seconds: This topic is close to my heart because it involves FinTech, databases, and performance investigation/tuning. A card number or PAN or Primary Account Number is a series of 12 to 19 digits. Most popular VISA and MasterCard cards use 16 digits but card systems must handle the…
Sometimes I think the most important paper of our times is Scalability! But at what COST? We build “web-scale” systems and get excited about how scalable they are and how we solve the problems of scale, but we do not measure the COST: Configuration that Outperforms a Single Thread. We lack reasonable, well-done single-threaded implementations for different domains. One of which is ledger and…
Once again I am playing with eBPF and bpftrace. This time I am trying to trace all file access. Whenever a file is open, created, or deleted I want to print the filename, the process ID, and the user who did it. bpftrace has the username built in to get the username. However, I noticed I was missing a lot of file creations and by trial and error, I discovered many applications use the openat…
Wikipedia says that O stands for: Open–closed principle: “Software entities … should be open for extension, but closed for modification.” Once again, let’s go to the actual article written by Robert C. Martin The Open-Closed Principle . This time the article does not have any redefinitions and cites a principle described in another book: Object Oriented Software Construction, Bertrand Meyer,…
So many people throw SOLID around as the unquestionable truth that I decided to dig a bit deeper and look into its origins. I will start with S. Wikipedia says that S stands for: Single-responsibility principle (SRP): “There should never be more than one reason for a class to change.” In other words, every class should have only one responsibility. From there I went to the linked SRP article…
The correct way to list the queues on the Linux machine is by using the ipcs -q command. However, I have not seen anything in the system calls that would provide such a capability. But Today I Learned… The ipcs utility has two ways of retrieving the list of queues. The first and main one is to read and parse /proc/sysvipc/msg . If the file is not present, it does a fallback by using a strange…
I have built several scripts for monitoring Oracle Tuxedo over the years. Most relied on the tmtrace and tputrace(3c) features provided by Tuxedo itself. I also tried to collect statistics on the IPC queues using standard system calls and tools. But all these years I have been thinking about how to measure the time a message spends in a queue. Messages in IPC queues are just an array of bytes,…
I was going through some Docker images of large applications and looking at the layer sizes with dive . A lot of space was used by installed dependencies under /usr/local/lib/python3.10/dist-packages/ and virtual environments. Yes, there were a lot of .py and .pyc files as you would expect but the largest files were the .so files. In one of the cases, there were 269M of compiled C and C++ code in…
I have been looking at some FinTech talks about Ledgers and Accounting and then I found one about Ledger implementation at Modern Treasury . It had some good ideas like delaying balance updates until they are needed. I scanned the documentation and at least from the API side, it had the distinction between a relative update by checking the remaining balance and optimistic locking by checking the…
TigerBeetle is an interesting project to follow. It links to interesting papers, it challenges assumptions, algorithms, and architecture of past systems. It satisfies my cravings for low-level programming. In one of the presentations , it says “Accounting is the Language of Business” and “SQL is the Language of OLTP” so we get “OLTP Impedance Mismatch”. As an example of impedance mismatch, one of…
I was reading about offset retention and was mentally prepared for losing consumer offsets when the consumer has been offline for 7+ days. But I realized that it’s 7+ days since the last update of offsets. This means if a topic did not receive any new message for 7+ days, there is no reason to update the offsets and they will be lost as well. WTF, Kafka? How about the Principle of least…
Open your conftest.py and type the following lines: import gc # Like gc.disable() but overrides your dependencies that do # gc.disable() and gc.enable() gc . set_threshold ( 0 ) Depending on the project size and number of tests there is a speedup from a few percent in smaller projects up to 10% in a project with 30,000 tests. Your experience may vary. I went down the Python GC rabbit hole and…
My journey to understand the performance and concurrency of FastAPI services lead me to the Python json.dumps function producing JSON string out of object tree. And I ended up making it a bit faster: _PyAccu vs _PyUnicodeWriter How do you create the JSON string out of string representation of all objects in the tree? json.dumps C implementation used _PyAccu for that . _PyAccu maintains two lists…
After finding a good enough solution for FastAPI and cooperative multi-threading issues, a part of me was still not happy with the results. There was a significant drop in the number of concurrent requests: 1643620388 309 1643620389 5 1643620390 3 1643620391 6 1643620392 5 1643620393 322 It’s not this bad in practice or for any realistic response size. The numbers above were obtained by running…
Both the HTTP client (Scala) and HTTP server (Python) are running in Kubernetes. Depending on the input, the service may take even more than 10 minutes to produce a response. Message queues would be a better solution instead of long HTTP connections but that is not an option at this point. Everything works for short requests of a couple of minutes. Once a request takes 5-6 or more minutes, strange…
Cal Paterson wrote a great article comparing and describing synchronous and asynchronous Python frameworks and explaining why asynchronous frameworks go a bit wobbly under load. This is a story of how we experienced wobbliness in a recent project. We are using FastAPI, Pydantic, and Kubernetes to build microservices. One of them is a query service that returns a paginated result containing a list…
TL;DR Don’t forget to set socket.nagle.disable=True to disable Nagle’s algorithm The code I am working with uses Confluent Kafka Python library that calls librdkafka C++ library underneath. The code does synchronous messaging: it produces events one-by-one and ensures event is persisted in Kafka topic by waiting for acknowledgments from all replicas. This code is sensitive to latency and not so…
I have a piece of C++ code that calls user-defined functions implemented in Python. Instead of requiring all functions to have the same signature with 6 arguments, the C++ code inspects the function signature and passes only the arguments function accepts - 1, 3, or all 6 of them. I use the inspect module and getargspec function for that but it feels a bit wrong and bloated. So let’s see how we…
Oracle Tuxedo documentation of the tpcall function describes (mostly) the high-level behavior of the function. But in addition to that, I am interested in how the messages will be sent through the IPC queues and what happens to the transactions as that has an impact on the performance and the behavior of the system under load. TPNOFLAGS First, let us see how the tpcall behaves without any flags,…
ltrace is a tool for tracing dynamic library calls. I use it from time to time and this time I needed to trace the msgrcv system call. Usually, you would use strace to trace the system calls, but I need an output of both library calls and system calls. One interesting feature is that I don’t need to trace the actual system calls using the -S flag and can trace the libc system call wrappers…
FML32 is Oracle Tuxedo buffer type similar to std::multimap in C++ or multidict in Python where the key might occur more than once in the container. Fchg32 is the function for changing a value for a specific key and occurence. But carefully reading the documentation reveals more interesting details. First, if there are only two occurances and you try to change the value of the fifth occurance, a…
I described the SIGFPE bomb of Boolean Expressions before. Going through the list of the C functions I was reminded of the Fboolpr32 function that prints the expression tree as it was parsed. Ten minutes later I had added it to the Python Tuxedo library . So let us look at the SIGFPE bomb again: >>> import tuxedo as t >>> t . Fboolev32 ({ "TA_STATUS" : "OK123" }, "TA_STATUS %! 'OK.*'" ) Floating -…
Time accounting is strange in Oracle Tuxedo. First, there is the SCANUNIT parameter which must be a value between 0 and 60 seconds and also a multiple of 2 or 5 ( 10 by default). Then the SANITYSCAN to perform system health checks and BLOCKTIME for blocking timeouts are expressed as the multipliers of that SCANUNIT . One of the side-effects of that is timeouts are rarely exact. Most often they are…
Servers in Oracle Tuxedo can be configured either in Multiple Servers - Single Queue (MSSQ) setup or Single Server - Single Queue (SSSQ) mode. More in-detail information about SSSQ and MSSQ setup can be found in my book “Modernizing Oracle Tuxedo Applications with Python” . When it comes to the performance aspect of these setups, Tuxedo documentation recommends using the MSSQ setup when: You have…
Oracle Tuxedo has the Fboolev32 function for evaluating Boolean expressions in which the “variables” are the values of fields. The expressions are a subset of the C programming language with a nice addition of regular expression match operators: expression %% expression yields a 1 if the first expression is fully matched by the second expression (the regular expression). expression !% expression…
Oracle Tuxedo uses System V IPC message queues for sending messages between processes. These queues live inside the OS kernel and are limited in size: sudo sysctl -a | grep kernel.msgm Unless you have modified these parameters, you should see the following result on Linux: kernel.msgmax = 8192 kernel.msgmnb = 16384 kernel.msgmni = 32000 kernel.msgmax is the maximum size of an individual message…
Now that my book about Oracle Tuxedo is completed, I can work on some code that did not make it into the book. Here is the first by-product: a Prometheus exporter of Tuxedo metrics . Chapter 6 (among other topics) showed how to retrieve statistics from the Tuxedo core system, but there was no space to include export to Application Performance Management tool.
I did investigate tpacall() before and you can find more details there. But this time I had to prepare internal presentation so I developed a small Tuxedo app for simulation and scripts for visualizing the results. Just 2 Tuxedo servers, one calling the other, and a client program injecting the events. The application can process 10 events per second, 20 events per second are sent to the…
A server waits for a new incoming service call by using msgrcv() system call on IPC queue. The call blocks until a message is available or a signal interrupts the system call. So how can Oracle Tuxedo command tmshutdown stop and shut down the server? Sending a signal might do more harm to the server code so it’s not an option. tmshutdown could delete the IPC queue and that should make msgrcv() to…
Oracle Tuxedo allows us to develop transactional service-oriented (or microservice) applications easily and performance is quite good. So easy and performant that I would not care about implementing sagas or compensating transactions most of the time. And then there is tpacall() function call that allows to parallelize execution by calling some service in asynchronous mode, doing work in parallel…
There’s already a collection of articles describing problems others have with Go: Go is not good I have this saying that life is too short to write everything in C. Sadly it’s true for Go as well. I could get used to other aspects but every second line checking and propagating errors is just too much.
Today I Learned one more thing for “tabs versus spaces” debate: Java hates tabs. Turns out Java compiler counts each tab symbol as 8 spaces while parsing the source code. And when it reports errors it uses the 8-space version of column number. You don’t get this normally when using javac from the command line, but all IDEs call Java compiler programmatically to access diagnostic information. So…
I wrote before about tpadvertisex() and how it did not work for me It does! The documentation of tpadvertisex() mentions “flags” argument but does not explain it’s usage. I assumed that it’s just a placeholder like some other Tuxedo functions have. So I was working on some other project creating Python bindings for Oracle Tuxedo and needed a list of all error codes. And there it was in atmi.h :…