This blog tracks development of the open source accounting and ERP software LedgerSMB. I also offer some perspectives on PostgreSQL including new features which we may find useful. Brought to you by Metatron Technology Consulting.
In my recent trip to PgConf Russia, a friend brought up the reaction of a Linux kernel maintainer to a patch submission by Baikal Electronics. The reaction has apparently shocked open source developers throughout Russia and I think perfectly encapsulates the dangers open source projects, including PostgreSQL, have to navigate today in the changing geopolitical context. I decided to write a longer…
I have decided to do a series of mini-articles on human factors in database operations. This is the first, and covers fatigue. In my talk at the PostgreSQL devroom of Fosdem, I asked a few questions: 1. How many of you have seen someone work on a production database while drunk? About half the audience. 2. How many times does this cause a major incident? No hands. 3. How many of you have seen…
At PGConf India, I watched Bruce Momjian's excellent talk Future Postgres Challenges . This talk discusses technical, technological, and project-related challenges Postgres has faced and continues to face. This immediately lead me to ask a question about efforts at geopolitical disentanglement and how we should try to avoid them in the Postgres community. While this question immediately gets…
Table inheritance is one of the most misunderstood -- and powerful -- features of PostgreSQL. With it, certain kinds of hard problems become easy. While many folks who have been bitten by table inheritance tend to avoid the feature, this blog post is intended to provide a framework for reasoning about when table inheritance is actually the right tool for the job. Table inheritance is, to be sure,…
It came out a while ago but I haven't promoted it much yet. This is the recorded version of the PostgreSQL at 20TB and Beyond talk. It covers a large, 500TB analytics pipeline and how we manage data. For those wondering how well PostgreSQL actually scales, this talk is worth watching.
My overall perspective here is that the PostgreSQL community needs a code of conduct, and one which allows the committee to act in some cases for off-infrastructure activity, but that the current code of conduct has some problems which could have been fixed if efforts had been better taken ensure that feedback was gathered when it was actionable. This piece discusses what I feel was done poorly…
I saw an interesting presentation recorded and delivered on LinkedIn on contempt culture by Aurynn Shaw, delivered this year at PyCon. I had worked with Aurynn on projects back when she used to work for Command Prompt. You can watch the video below: Unfortunately comments on a social media network are not sufficient for discussing nuance so I decided to put this blog post together. In my view she…
The PostgreSQL at 10 TB And Beyond talk has now been released on Youtube. Feel free to watch. For the folks seeing this on Planet Perl Iron Man, there is a short function which extends SQL written in Perl that runs in PostgreSQL in the final 10 minutes or so of the lecture. This lecture discusses human and technical approaches to solving volume, velocity, and variety problems on PostgreSQL in the…
One of the topics discussed in the large database talk is the way we used PL/Perl to solve some data variety problems in terms of extracting data from structured text documents. It is certainly possible to use other languages to do the same, but PL/Perl has an edge in a number of important ways. PL/Perl is light-weight, flexible and fills this particular need better than any other language I have…
I have been invited to give a talk on PostgreSQL at 10TB and above in Malmo, Sweden. The seminar is free to attend. I expect to be talking for about 45 minutes with some time for questions and answers. I also have been invited to give the talk at PG Conf Russia in March. I do not know whether either will be recorded. But for those in the Copenhagen/Malmo area, you can register for the seminar at…
Having been using various vcs solutions for a while, I think it is worth noting that my least favorite from a user experience perspective is git. To be sure, git has better handling of whitespace-only merge conflicts and a few other major features than any other vcs that I have used. And the data structure and model are solid. But even understanding what is going on behind the scenes, I find git…
So one of the clients I do work with is moving a large database from PostgreSQL to Hadoop . The reasons are sound -- volume and velocity are major issues for them, and PostgreSQL is not going away in their data center and in their industry there is a lot more Hadoop usage and tooling than there is PostgreSQL tooling for life science analytics (Hadoop is likely to replace both PostgreSQL and,…
So for those of you who know, I now spend most of my time doing more general PostgreSQL consulting and a fair bit of time still on LedgerSMB . One of my major projects lately has been on a large scientific computing platform currently run on PostgreSQL, but due to volume and velocity of data being moved to Hadoop (the client maintains other fairly large PostgreSQL instances with no intention of…
Which the discussion of CVE-2016-1238, a quick and easy fix for broken code that has been suggested is to add the following line to the top of broken Perl scripts: Note this applies to Perl as run anywhere, whether pl/perlU, plain perl, or something else. use lib '.'; In some corners, this has become the goto solution for the problem (pun quite deliberate). It works, gets the job done, and…
This post is about the dangers in writing user defined functions in untrusted languages, but it is also specifically about how to avoid CVE-2016-1238-related problems when writing PL/PerlU functions. The fix is simple and straight-forward and it is important for it to be in pl/perlU stored procedures and user defined functions for reasons I will discuss. This discusses actual exploits and the…
Because I have lost confidence in the approach taken by those in charge of fixing this problem, I have decided to do a full disclosure series on CVE-2016-1238. As mentioned in a previous post, the proposed fixes for the minor versions don't even remotely fix the problem and at most can provide a false sense of security. For this reason it is extremely important that sysadmins understand how these…
A cardinal rule of software security is that, when faced with a problem, make sure you fully understand it before implementing a fix. A cardinal rule of general library design is that heuristic approaches to deciding whether something is a problem really should be disfavored. These lessons were driven home when I ended up spending a lot of time debugging problems caused by a recent Debian fix for…
LedgerSMB 1.5 rc1 is around the corner. I figure it is time for a very short list of major improvements: A one-page app design which provides better responsiveness and testability Speaking of testability, we now have selenium-based bdd-tests as well (and a test coverage starting to approach reasonable -- remember, we started with no tests). We have spun off our database access framework as…
The largest database I have ever worked on will eventually, it looks like, be moved off PostgreSQL. The reason is that PostgreSQL doesn't scale well enough. I am writing here however because the limitations are so extreme that it ought to give plenty of ammunition for those who think databases don't scale. The current database size is 10TB and doubling every year. The main portions of the…
In one of my projects we had an issue where a large table that was under huge transactional load was having trouble with autovacuum not keeping up. The problem was that the table sometimes held over half a billion records, added and deleted millions of records a day, and that since most of these occurred at the heads of various indexes, autovacuum was just not fast enough. So we decided to…
This will likely be my last post on this topic. I would like to revive this blog on technical rather than ideological issues but there seems like a real effort to force ideology in some cases. I don't address this in terms of specific rights, but in terms of community function and I have a few more things to say on this topic before I return to purely technical questions. I am also going to say at…
The PostgreSQL, Ruby, and PHP communities have all been considering codes of conduct for contributors. The LedgerSMB community already uses the Ubuntu Code of Conduct. Because this addresses many projects, I am syndicating this further than where there are current issues. This is not a technical post and it covers a wide range of very divisive issues for a very diverse audience. I can only hope…
Preface This article demonstrates the simplest cases regarding autoserialization and deserialization to the database of objects in PGObject. It also demonstrates a minimal subset of the problems that three valued logic introduces and the most general solutions to those problems. The next article in this series will address more specific solutions and more complex scenarios. The Problems Often…
Preface I have decided to put together a PGObject Cookbook, showing the power of this framework. If anyone is interested in porting the db-looking sides to other languages, please let me know. I would be glad to provide whatever help my time and skills allow. The PGObject framework is a framework for integrated intelligent PostgreSQL databases into Perl applications. It addresses some of the same…
15 September 2014, London . The LedgerSMB project - all-volunteer developers and contributors - today announced LedgerSMB 1.4.0. Based on an open source code base first released in 1999, the LedgerSMB project was formed in 2006 and saw it's 1.0 release in the same year. It has now seen continuous development for over eight years and that shows no signs of slowing down. "LedgerSMB 1.4 brings major…