Recent content in Posts on Blog of Julian Andres Klode
Recently a person reported a bug in APT saying that TLS is failing on FIPS systems with MD5 errors, and suggested we call ERR_clear_error() around TLS operations. Like any serious software engineer would do, I said No. Just because one component failed to handle its errors does not mean I can go around and discard all errors in another place - the program should have failed earlier (or discarded…
Problem statement Currently if you have an automatically installed package A (= 1) where A (= 1) Depends B (= 1) A (= 2) Depends B (= 2) and you upgrade B from 1 to 2; then you can: Remove A (= 1) Upgrade A to version 2 If A was installed by a chain initiated by Recommends (say X Rec Y, Y Depends A), the solver sometimes preferred removing A (and anything depending on it until it got).
As I was shopping groceries I had a shocking realization: The active dependencies of packages in a solver actually form a trie (a dependency A|B - “A or B” - of a package X is considered active if we marked X for install). Consider the dependencies A|B|C , A|B , B|X . In most package managers these just express alternatives, that is, the “or” relationship, but in Debian…
As you may recall from previous posts and elsewhere I have been busy writing a new solver for APT. Today I want to share some of the latest changes in how to approach solving. The idea for the solver was that manually installed packages are always protected from removals – in terms of SAT solving, they are facts. Automatically installed packages become optional unit clauses. Optional clauses…
In my previous blog, I explored The New APT 3.0 solver . Since then I have been at work in the test suite making tests pass and fixing some bugs. You see for all intents and purposes, the new solver is a very stupid naive DPLL SAT solver (it just so happens we don’t actually have any pure literals in there). We can control it in a bunch of ways: We can mark packages as “install”…
APT 2.9.3 introduces the first iteration of the new solver codenamed solver3, and now available with the –solver 3.0 option. The new solver works fundamentally different from the old one. How does it work? Solver3 is a fully backtracking dependency solving algorithm that defers choices to as late as possible. It starts with an empty set of packages, then adds the manually installed packages,…
APT currently knows about three types of upgrades: upgrade without new packages (apt-get upgrade) upgrade with new packages (apt upgrade) upgrade with new packages and deletions (apt{,-get} {dist,full}-upgrade) All of these upgrade types are necessary to deal with upgrades within a distribution release. Yes, sometimes even removals may be needed because bug fixes require adding a Conflicts…
This is the story of the currently progressing changes to secure boot on Ubuntu and the history of how we got to where we are. taking a step back: how does secure boot on Ubuntu work? Booting on Ubuntu involves three components after the firmware: shim grub linux Each of these is a PE binary signed with a key. The shim is signed by Microsoft’s 3rd party key and embeds a self-signed Canonical…
Z3 is a theorem prover developed at Microsoft research and available as a dynamically linked C++ library in Debian-based distributions. While the library is a whopping 16 MB, and the solver is a tad slow, it’s permissive licensing, and number of tactics offered give it a huge potential for use in solving dependencies in a wide variety of applications. Z3 does not need normalized formulas,…
This is an edited copy of an email I sent to provide guidance to users of apt-key as to how to handle things in a post apt-key world. The manual page already provides all you need to know for replacing apt-key add usage: Note: Instead of using this command a keyring should be placed directly in the /etc/apt/trusted.gpg.d/ directory with a descriptive name and either “gpg” or…
APT 2.2.0 marks the freeze of the 2.1 development series and the start of the 2.2 stable series. Let’s have a look at what changed compared to 2.2. Many of you who run Debian testing or unstable, or Ubuntu groovy or hirsute will already have seen most of those changes. New features Various patterns related to dependencies, such as ?depends are now available (2.1.16) The Protected field is…
Yesterday I got a fresh new Pixel 4a, to replace my dying OnePlus 6. The OnePlus had developed some faults over time: It repeatedly loses connection to the AP and the network, and it got a bunch of scratches and scuffs from falling on various surfaces without any protection over the past year. Why get a Pixel? Camera : OnePlus focuses on stuffing as many sensors as it can into a phone, rather than…
Sporting a beautiful 10.1" 1920x1200 display, the Lenovo IdeaPad Duet Chromebook or Duet Chromebook, is one of the latest Chromebooks released, and one of the few slate-style tablets, and it’s only about 300 EUR (300 USD). I’ve had one for about 2 weeks now, and here are my thoughts. Build & Accessories The tablet is a fairly Pixel-style affair, in that the back has two components, one…
Ubuntu “Focal Fossa” 20.04 was released two days ago, so I took the opportunity yesterday and this morning to upgrade my VPS from Ubuntu 18.04 to 20.04. The VPS provides: SMTP via Postfix Spam filtering via rspamd HTTP(S) via nginx and letsencrypt (certbot) Weechat relay OpenVPN server Shadowsocks proxy Unbound recursive DNS resolver, for the spam filtering I rebooted one more time…
After brewing in experimental for a while, and getting a first outing in the Ubuntu 19.10 release; both as 1.9, APT 2.0 is now landing in unstable. 1.10 would be a boring, weird number, eh? Compared to the 1.8 series, the APT 2.0 series features several new features, as well as improvements in performance, hardening. A lot of code has been removed as well, reducing the size of the library.
If you have ever used aptitude a bit more extensively on the command-line, you’ll probably have come across its patterns. This week I spent some time implementing (some) patterns for apt, so you do not need aptitude for that, and I want to let you in on the details of this merge request !74 . so, what are patterns? Patterns allow you to specify complex search queries to select the packages…
In the previous post about setting up a email server, I explained how I setup a forwarder using Postfix. This post will look at setting up Dovecot to store emails (and provide IMAP and authentication) on the server using GPG encryption to make sure intruders can’t read our precious data! Architecture The basic architecture chosen for encrypted storage is that every incoming email is…
This week, I’ve been working on rolling out mail services on my server. I started working on a mail server setup at the end of November, while the server was not yet in use, but only for about two days, and then let it rest. As my old shared hosting account expired on January 1, I had to move mail forwarding duties over to the new server. Yes forwarding - I do plan to move hosting the actual…
(What follows is an excerpt from my master’s thesis, almost all of section 2.1, quickly introducing Go to people familiar with CS) Go is an imperative programming language for concurrent programming created at and mainly developed by Google, initially mostly by Robert Griesemer, Rob Pike, and Ken Thompson. Design of the language started in 2007, and an initial version was released in 2009;…
As of today, I migrated various services from shared hosting on uberspace.de to a VPS hosted by hetzner . This includes my weechat client, this blog, and the following other websites: jak-linux.org dep.debian.net redirector mirror.fail Rationale Uberspace runs CentOS 6. This was causing more and more issues for me, as I was trying to run up-to-date weechat binaries. In the final stages, I ran…
So, I’ve been using ikiwiki for my website since 2011. At the time, I was hosting the website on a tiny hosting package included in a DSL contract - nothing dynamic possible, so a static site generator seemed like a good idea. ikiwiki was a good social fit at the time, as it was packaged in Debian and developed by a Debian Developer. Today, I finished converting it to Hugo . Why? I did not…
I’m back to blogging, after shutting down my wordpress.com hosted blog in spring. This time, fully privacy aware, self hosted, and integrated with mastodon. Let’s talk details: In spring, I shutdown my wordpress.com hosted blog, due to concerns about GDPR implications with comment hosting and ads and stuff. I’d like to apologize for using that, back when I did this (in 2007 ), it…
2017 is ending. It’s been a rather uneventful year, I’d say. About 6 months ago I started working on my master’s thesis - it plays with adding linear types to Go - and I handed that in about 1.5 weeks ago. It’s not really complete, though - you cannot actually use it on a complete Go program. The source code is of course available on GitHub , it’s a bunch of Go code…
I just uploaded APT 1.6 alpha 1, introducing a very scary thing: Seccomp sandboxing for methods, the programs downloading files from the internet and decompressing or compressing stuff. With seccomp I reduced the number of system calls these methods can use to 149 from 430. Specifically we excluded most ways of IPC, xattrs, and most importantly, the ability for methods to clone(2) , fork(2) ,…
APT 1.5 is out, after almost 3 months the release of 1.5 alpha 1, and almost six months since the release of 1.4 on April 1st. This release cycle was unusually short, as 1.4 was the stretch release series and the zesty release series, and we waited for the latter of these releases before we started 1.5. In related news, 1.4.8 hit stretch-proposed-updates today, and is waiting in the unapproved…
In DebConf17 there was a talk about The Update Framework, short TUF. TUF claims to be a plug-in solution to software updates, but while it has the same practical level of security as apt, it also has the same shortcomings, including no way to effectively revoke keys. TUF divides signing responsibilities into roles: A root role, a targets rule (signing stuff to download), a snapshots rule (signing…
In the past two days, I moved my main web site jak-linux.org (and jak-software.de) from a very old contract at STRATO over to something else: The domains are registered with INWX and the hosting is handled by uberspace.de . Encryption is provided by Let’s Encrypt . I requested the domain transfer from STRATO on Monday at 16:23, received the auth codes at 20:10 and the .de domain was…
We just released the first beta of APT 1.4 to Debian unstable (beta here means that we don’t know any other big stuff to add to it, but are still open to further extensions). This is the release series that will be released with Debian stretch, Ubuntu zesty, and possibly Ubuntu zesty+1 (if the Debian freeze takes a very long time, even zesty+2 is possible). It should reach the master archive…
I’m proud (yes, really) to announce DNS66 , my host/ad blocker for Android 5.0 and newer. It’s been around since last Thursday on F-Droid, but it never really got a formal announcement. DNS66 creates a local VPN service on your Android device, and diverts all DNS traffic to it, possibly adding new DNS servers you can configure in its UI. It can use hosts files for blocking whole sets…
Abstract I introduce TrieHash an algorithm for constructing perfect hash functions from tries. The generated hash functions are pure C code, minimal, order-preserving and outperform existing alternatives. Together with the generated header files,they can also be used as a generic string to enumeration mapper (enums are created by the tool). Introduction APT (and dpkg) spend a lot of time in…
Today, I wrote sicherboot , a tool to integrate systemd-boot into a Linux distribution in an entirely new way: With secure boot support. To be precise: The use case here is to only run trusted code which then unmounts an otherwise fully encrypted disk, as in my setup: If you want, sicherboot automatically creates db, KEK, and PK keys, and puts the public keys on your EFI System Partition (ESP)…
Did that ever happen to you: You run apt update, it fetches a Release file, then starts fetching DEP-11 metadata, then any pdiff index stuff, and then applies them; all after another? Or this: You don’t see any update progress until very near the end? Worry no more: I tweaked things a bit in 1.3~rc4 ( git commit ). Prior to 1.3~rc4, acquiring the files for an update worked like this: We…
Ever since it’s creation back in the dark ages, APT shipped with it’s own build system consisting of autoconf and a bunch of makefiles. In 2009, I felt like replacing that with something more standard, and because nobody really liked autotools, decided to go with CMake. Well, the bazaar branch was never really merged back in 2009. Fast forward 7 years to 2016. A few months ago, we…
I recently found out that I have access to a 1 TB cloud storage drive by 1&1, so I decided to start taking off-site backups of my $HOME (well, backups at all, previously I only mirrored the latest version from my SSD to an HDD). I initially tried obnam. Obnam seems like a good tool, but is insanely slow. Unencrypted it can write about 3 MB/s, which is somewhat OK, but even then it can spend hours…
The APT 1.2.7 release is out now. Despite of what I wrote earlier, we now print warnings for Release files signed with signatures using SHA1 as the digest algorithm. This involved extending the protocol APT uses to communicate with the methods a bit, by adding a new 104 Warning message type. W: gpgv:/var/lib/apt/lists/apt.example.com_debian_dists_sid_InRelease: The repository is insufficiently…
Tomorrow is the anniversary of Caesar’s assassination APT will see a new release, turning of support for SHA-1 checksums in Debian unstable and in Ubuntu xenial, the upcoming LTS release. While I have no knowledge of an imminent attack on our use of SHA1, Xenial (Ubuntu 16.04 LTS) will be supported for 5 years, and the landscape may change a lot in the next 5 years. As disabling the SHA1…
Not only do I keep incrementing version numbers faster than ever before, APT also keeps getting faster. But not only that, it also has some bugs fixed and the cache is now checked with a hash when opening. Important fix for 1.1.6 regression Since APT 1.1.6, APT uses the configured xz compression level. Unfortunately, the default was set to 9, which requires 674 MiB of RAM, compared to the 94 MiB…
APT’s performance in applying the Pdiffs files, which are the diff format used for Packages, Sources, and other files in the archive has been slow. Improving performance for uncompressed files The reason for this is that our I/O is unbuffered, and we were reading one byte at a time in order to read lines. This changed on December 24, by adding read buffering for reading lines, vastly…
I started transitioning from 1024D to 4096R. The new key is available at: https://people.debian.org/~jak/pubkey.gpg and the keys.gnupg.net key server. A very short transition statement is available at: https://people.debian.org/~jak/transition-statement.txt and included below (the http version might get extended over time if needed). The key consists of one master key and 3 sub keys (signing,…
I spent the weekend using almost exclusively my Chromebook 13, on a single charge Saturday and Sunday. Keyboard I think I like the keyboard better now than I used to when I first tried it. It gets nowhere near the ThinkPad X230 one, though; appart from the coating, which my (backlit) X230 unfortunately does not have. Screen While the screen appeared very grainy to me on first sight, having only…
Today, I received my Acer Chromebook 13, in the glorious FullHD variant with 4GB RAM. For those of you who don’t know it, the Acer Chromebook 13 is a 13.3 inch chromebook powered by a Tegra K1 cpu. This version cannot be ordered currently, only pre-orders were shipped yesterday (at least here in Germany). I cannot even review it on Amazon (despite having it bought there), as they have not…
Today I not only submitted my bachelor thesis to the printing company, I also released a new version of hardlink, my file deduplication tool. hardlink 0.3 now features support for xattr support, contributed by Tom Keel at Intel. If this does not work correctly, please blame him. I also added support for a –minimum-size option. Most of the other code has been tested since the upload of RC1 to…
Today, we worked, with the help of ioerror on IRC, on reducing the attack surface in our fetcher methods. There are three things that we looked at: 1. Reducing privileges by setting a new user and group 2. chroot() 3. seccomp-bpf sandbox Today, we implemented the first of them. Starting with 1.1~exp3, the APT directories /var/cache/apt/archives and /var/lib/apt/lists are owned by the…
Since some time, those crappy Fritz!Box devices do not support WDS anymore, but rather a proprietary solution created by AVM. Now what happens if you have devices in another room that need/want wired access (like TVs, Playstations) or if you want to extend the range of your network? Buying another Fritz!Box is not very cost efficient - What I did was to buy a cheap TP-Link TL-WR841N (can be bought…
Today, I decided to set my X230 back to UEFI-only boot, after having changed that for a bios upgrade recently (to fix a resume bug). I then choose to save the settings and received several error messages telling me that the system ran out of resources (probably storage space for UEFI variables). I rebooted my machine, and saw no logo appearing. Just something like an underscore on a text console.…
Today I made an important change to the python-apt code: It is now native Python 3 code (but also works under Python 2). The previous versions all run 2to3 during the build process to create a Python 3 version. This is no longer needed, as the code is now identical. As part of that change, python-apt now only supports Python 2.7, Python 3.3, and newer. I’m using some features only present in…
I released python-apt 0.9. This completely removes support for the old API from the code base (it was disabled for the entirety of 0.8 in Debian, and in Ubuntu since saucy). Highlights: * Cleanup: Complete removal of old-api support code * Bug fix: Various coverty bug fixes by Michael Vogt * Bug fix: Correctly handles multi-arch dependencies in apt.debfile, so packagekit and gdebi can now install…
This week, a few hours after Lenovo announced the X240, I bought an X230. Normally, the X230 model I bought comes with a Core i5-3320M CPU, 4GB RAM, 500GB HDD. My model was a special set including a second 4GB RAM stick and a 128 GB mSATA Plextor SSD. It came without Windows; and the ThinkVantage button is black instead of blue and has no label. I put a fresh installation of unstable on it and…
The script apt-show-versions is developed by another Debian Developer called Christoph Martin in Perl. Recently, it turned out that apt-show-versions is too slow for some users; so I decided to rewrite his program using APT’s C++ API. I expect this to be part of a future APT release, rendering the original apt-show-versions obsolete. The rewrite is sadly not 100% backwards compatible to the…
Writing recursive descent parsers is easy, especially in Python (just like everything is easy in Python). But Python defines a low limit on the number of recursive calls that can be made, and recursive descent parsers are prone to exceed this limit. We should thus write our parser without real recursion. Fortunately, Python offers us a way out: Coroutines, introduced in Python 2.5 as per PEP 342.…