Blog Posts from Martin Heinz
Whether you're developer, DevOps engineer, SysAdmin, QA or in any other technical role, you're surely familiar with cURL - _the command line tool and library for transferring data with URLs_ (as described in docs). Most of the time however, we all really only use `curl` for simple tasks, such downloading a file or checking if website is accessible, yet there's some much more `curl` can do! And in…
Chances are that you never touched and maybe haven't even heard about Python's `weakref` module. While it might not be commonly used in your code, it's fundamental to inner workings of many libraries, frameworks and even Python itself. So, in this article we will explore what it is, how is it helpful, and how you could incorporate it into your code as well.
Learn how to use BuildKit - the improved builder backend for Docker - that adds many new features to Docker, including new Dockerfile syntax, built-in debugger and more...
If you work in shell/terminal often enough, then over time the history will become your personal knowledge vault, documentation and command reference. Being able to use this personal documentation efficiently can hugely boost your productivity. So, here are a couple of tips on how to optimize your shell history configuration and usage to get the most out of it.
All of us - software engineers - use git every day, however most people only ever touch the most basic of commands, such as "add", "commit", "push" or "pull", like it's still 2005. Git however, introduced many features since then, and using them can make your life so much easier, so let's explore some of the recently added, modern "git" commands, that you should know about.
Python has many options for formatting strings and text, including f-strings, `format()` function, templates and more. There's however one module that few people know about and it's called `textwrap`. This module is specifically built to help you with line-wrapping, indentation, trimming and more, and in this article we will look at all the things you can use it for.
Low indoor air quality - or high CO2 - negatively impacts cognitive performance, causes headaches, drowsiness and more. It's easy to fix though, just use a CO2 sensor and open a window from time-to-time. But why stop there, when you can set up complete air quality monitoring solution using CO2 sensor, Prometheus, Grafana dashboards and alerts?
While Python's `bisect` module is very simple - containing really just 2 functions - there's a lot one can do with it, including searching data efficiently, keeping any data sorted, and much more - and in this article we will explore all of it!
PostgreSQL can be more than just an SQL database. Why maintain a Redis or Memcached instance when you can just use PostgreSQL as a cache instead?
Whenever I troubleshoot anything container-related, I look for a good container image that contains all the right tools to troubleshoot and/or solve the problem. However, finding such an image, or assembling my own is time-consuming and honestly just way too much effort. So, to save both you and me the hassle of finding or building such image(s), here's a list of container images that will satisfy…
From time to time, when coding, we all run into weird behaviours of the programming language. Sometimes it's a "feature" we weren't aware of, sometimes it's just quirky behaviour of the language, and sometimes it's borderline bug. Python - as any other programming language - has these eyebrows-raising quirks, so here's a list of weird Python "features" that might catch you off-guard.
Couple days ago, I published my 100th article, so I feel like it’s time for reflections — looking at how I got there, what I learned along the way and whether it was actually worth the time and effort. As well as some thoughts on whether you should try writing too.
There are certain bugs and issues that are very hard to troubleshoot. Just ask yourself, _"How would I debug deadlock, segmentation fault, crashing application, or a hanging process?" _ Now there's a tool to help you will all of that - _PyStack_ is a powerful new debugger that - according to docs - _"uses forbidden magic to let you inspect the stack frames of a running Python process or a Python…
etcd is the brain of every Kubernetes cluster, the key-value storage keeping track of all the objects in a cluster. It's intertwined and tightly coupled with Kubernetes, and it might seem like an inseparable part of a cluster, or is it? In this article we will explore how we could replace etcd with PostgreSQL database, as well as why and when it might make sense to do so.
Let's imagine a situation - you have multiple Python applications running on Kubernetes that interact with each other. There's bug that you can't reproduce locally, but it surfaces everytime you hit a particular API endpoint. If only you could attach to the remote running application processes, set breakpoints and debug them live... how easy it would be to troubleshoot the bug... But you can! In…
Python is a popular choice for automating anything and everything, that includes automating system administration tasks or tasks that require running other programs or interacting with operating system. There are however, many ways to achieve this in Python, most of which are arguably bad, though. So, lets take a look at all the options you have in Python for running other processes - the bad; the…
Python is 32 years old language, yet it still doesn't have proper, true parallelism/concurrency. This is going to change soon, thanks to introduction of a "Per-Interpreter GIL" (Global Interpreter Lock) which will land in Python 3.12. While release of Python 3.12 is some months away, the code is already there, so let's take an early peek at how we can use it to write truly concurrent Python code…
Python is known to come with "batteries included", thanks to its very extensive standard library, which includes many modules and functions that you would not expect to be there. However, there are many more "essential" Python libraries out there that you should know about and be using in all of your Python projects, and here's the list.
When it comes to synthetic testing, engineers often rely on 3rd party platforms such as Datadog or New Relic that provide this type of monitoring. If you're running your applications and services on Kubernetes though, you can spin up synthetic monitoring platform yourself using Kuberhealthy, and in this article we will take a look at how to deploy it, configure it, create synthetic checks and set…
Running live demos can be stressful. You know what you want to say and show. You prepare the CLI commands you want to run to best showcase what you've built, but then you waste time typing long commands; you make typos; the commands fail or take way too long to complete. Maybe they depend on external system (network, APIs, ...) and of course it's not cooperating while you're running your live…
While Python is not a pure functional programming language, you still can do a lot of functional programming in it. In fact, just one function - `reduce` - can do most of it and in this article I will show you all the things one can do just with `reduce`.
Nowadays, Alpine Linux is one of the most popular options for container base images. Many people (maybe including you) use it for anything and everything. Some people use it because of its small size, some because of habit and some, just because they copy-pasted a Dockefile from some tutorial. Yet, there are plenty of reasons why you **should not** use Alpine for your container images, some of…
There's a lot of "magic" that happens behind the scenes to make whole Kubernetes work. One of those is resource management and resource allocation done by Linux cgroups. In this article we will take a deep dive into what cgroups are, how Kubernetes uses them to manage Node resources, and how we can take advantage of them beyond setting resource requests and limits on Pods.
Have you ever written a long chain of if/else statements or a huge match/case block, with all statements just matching against a list of values, and wondered how could you make it more concise and readable? If so, then _dictionary dispatch_ pattern might be a tool for you. With dictionary dispatch we can replace any block of conditionals with a simple lookup into Python's dict and in this article…
Continuous profiling is a great tool for optimizing the performance of applications. It allows us to continuously monitor and analyze application's resource usage, identify bottlenecks, and use resources more efficiently. In this article, we will take a look at how to set up and use _Grafana Phlare_, a powerful new open-source tool by _Grafana Labs_, to perform continuous profiling of Python…
We all are familiar with Python's generators and all their benefits. But, what if I told you that we can make them even better by combining them with recursion? So, let's see how we can use them to implement _"lazy recursion"_ and supercharge what we already do with generators in Python!
Python's magic methods - also known as _dunder_ (double underscore) methods - can be used to implement a lot of cool things. Most of the time we use them for simple stuff, such as constructors (`__init__`), string representation (`__str__`, `__repr__`) or arithmetic operators (`__add__`/`__mul__`). There are however many more magic methods which you probably haven't heard about and in this article…
With what's happening at Twitter, many people are considering moving to Mastodon. Like Twitter, Mastodon also offers an API that can be used to create many useful application, bots, to analyze data, respond to notification or simply post some statuses, and in this crash course you will learn how to do all of the above using Python.
Kubernetes v1.25 introduced Container Checkpointing API as an alpha feature. This provides a way to backup-and-restore containers running in Pods, without ever stopping them. This feature is primarily aimed at forensic analysis, but general backup-and-restore is something any Kubernetes user can take advantage of. So, let's take a look at this brand-new feature and see how we can enable it in our…
Google has literally hundreds of APIs, including ones for Gmail, Drive, Maps, Translation, Analytics and more. All of these share the same concepts like authorization, pagination or media uploads/downloads. In this article we will explore all of these concepts and also get our hands dirty with some of the above-mentioned APIs to learn about all the cool things you can do with any and all of…
Out-of-the-box, Python standard library ships with many great libraries some of which provide commandline interface (CLI), allowing us to do many cool things directly from terminal without needing to even open a `.py` file. This includes things like starting a webserver, opening a browser, parsing JSON files, benchmarking programs and many more, all of which we will explore in this article.
Python provides a lot of ways to ask questions about your code. Whether it's basic things like `help()` function, builtin functions like `dir()` or more advanced methods in `inspect` module - the tools are there to help you find the answers to your questions. Let's find out what kinds of questions about our own code can Python answer for us and how it can help us during debugging sessions, dealing…
Every Python developer is familiar with the `self` argument, which is present in every* method signature of every class. We all know how to use it, but do you _really_ know what it is, why it's there and how it works under the hood?
Python's list comprehensions (and generators) are an awesome feature that can greatly simplify your code. Most of the time however, we only use them to write a single `for` loop, maybe with addition of one `if` conditional and that's it. If you start poking around a bit though, you will find out that there are many more features of Python's comprehensions that you don't know about, but can learn a…
The assignment operator - or "walrus operator" as we all know it - is a feature that's been in Python for a while now (since 3.8), yet it's still somewhat controversial and many people have unfounded hate for it. In this article I will try to convince you that the walrus operator really is a good addition to the language and that if you use it properly, then it can help you make your code more…
The `match`/`case` syntax that got introduced to Python in 3.10 looks like basic `switch` statement which we all know from other languages - in Python however - it's much more than just an alternative `if` syntax. So, let's explore advanced features of `match`/`case` syntax - or as it's properly called - _structural pattern matching_. As well as tips and tricks for using it effectively, including…
With every Python release, there are new modules being added and new and better ways of doing things get introduced. We all get used to using the good old Python libraries, but it's time say goodbye to `os.path`, `random`, `pytz`, `namedtuple` and many more obsolete Python libraries and start using the latest and greatest ones instead.
Scaling application on Kubernetes using _Horizontal Pod Autoscaler (HPA)_ based on their CPU or memory usage is pretty simple. There are however many more features of HPA that you can use to customize scaling behaviour of your applications using external/custom metrics or alpha/beta features like _"scaling to zero"_ or container metrics scaling. So, let's explore all of these options so that we…
As files, datasets and configurations grow, it gets increasingly difficult to navigate them. There are however many tools out there, that can help you to be more productive when dealing with large JSON and YAML files, complicated regular expressions, confusing SQL database relationships, complex development environments and many others. So, let's take a look at some of those!
We've all been there - it's frustrating seeing deletion of Kubernetes resource getting stuck, hang or take a very long time. You might have _"solved"_ this using the terrible advice of removing finalizers or running `kubectl delete ... --force --grace-period=0` to force immediate deletion. 99% of the time this is a horrible idea and in this article I will show you why.
In this article we take look at how you can leverage Kubernetes Python Client library to automate whatever annoying Kubernetes task you might be dealing with, whether it's creating/patching resources, watching events, accessing containers or anything else.
Monitoring is usually at the end of a checklist when building an application, yet it's crucial for making sure that it's running smoothly and that any problems are found and resolved quickly. Building complete monitoring - including aggregated logging, metrics, tracing, alerting or synthetic probes - requires a lot of effort and time though, not to mention building and managing the infrastructure…
With the recent events relating to _Google Analytics_ platform, it's becoming very clear that the time has come for many of us to migrate from Google Analytics to different platforms. In this article we will go over both the _"Why?"_, so that you can make an informed decision whether you need to migrate of not, as well as the _"How?"_ of migrating from Google Analytics - that is quickly and easily…
Formatted string literals - also called _f-strings_ - have been around since Python 3.6, so we all know what they are and how to use them. There are however some facts and handy features of f-string that you might not know about. So, let's take a tour of some awesome f-string features that you'll want to use in your everyday coding.
Every project can benefit from a robust continuous integration pipeline that builds your application, runs tests, lints code, verifies code quality, runs vulnerability analysis and more. However, building such pipeline takes a significant amount of time, which doesn't really provide any benefit on its own. So, if you want a fully featured, customizable CI pipeline for your Python project based on…
When it comes to performance optimization, people usually focus only on speed and CPU usage. Rarely is anyone concerned with memory consumption, well, until they run out of RAM. There are many reasons to try to limit memory usage, not just avoiding having your application crash because of out-of-memory errors. In this article we will explore techniques for finding which parts of your Python…
Before any new feature, change or improvement makes it into Python, there needs to be a _Python Enhancement Proposal_, also knows as PEP, outlining the proposed change. These PEPs are a great way of getting the freshest info about what might be included in the upcoming Python releases. So, in this article we will go over all the proposals that are going to bring some exciting new Python features…
We all spend a good chuck of our time debugging, sifting through logs or reading tracebacks. Each of these can be difficult and time-consuming and in this article we will focus on making the last one - dealing with tracebacks and exceptions - as easy and efficient as possible. To achieve this we will learn how to implement and use custom _Exception Hooks_ that will remove all the noise from…
If you're using GitHub as your version control system of choice then GitHub Apps can be incredibly useful for many tasks including building CI/CD, managing repositories, querying statistical data and much more. In this article we will walk through the process of building such an app in Go including setting up the GitHub integration, authenticating with GitHub, listening to webhooks, querying…
Profiling is integral to any code and performance optimization. Any experience and skill in performance optimization that you might already have will not be very useful if you don't know where to apply it. So, in this article we will look at the tools and techniques that can help us narrow down our focus and find bottlenecks both for CPU and memory consumption, as well as how to implement easy…