Process control, data science, AI and life
With the important leaps being made in the world of AI generative art I've been returning in my mind to a short story by Spider Robinson called Melancholy Elephants . It had a significant impact on how I think about creativity. Most of what I say will be about music, two-dimensional visual art and the written word since these are the fields most affected by progress made in the last year or so.…
Introduction I have been investing in a solar system at home, since South African load shedding is really getting me down. One of the key elements of going off the grid is to reduce your electricity consumption. I've focused on reducing demand by scaling down our geysers from 3 to 1 big one and 1 small undercounter unit. I also managed to find a second hand heat pump, so that geyser is going to be…
Introduction ¶ Note 1: This is an edited version of a full Jupyter Notebook which you can use to follow along with the calculations. Note 2: All the timings are for CPython 3.9.7 on Intel MacBook Pro (16-inch, 2019) There are several ways to look up a value in a dictionary in Python. dictionary = { 'known' : 'value' } The most obvious way is straight indexing: : key = 'known' value = dictionary […
4 short months ago I collected some data on the causes of death in South Africa , placing the deaths attributed to COVID-19 in context, mostly for my own edification. Even then, when they numbered less than half of what they are now, it was clear that this was a serious concern. At the time, I neglected some analysis due to a lack of time. Notably I didn't normalise by population, even though the…
I was curious this morning, so I spent some time digging for context to the numbers we're constantly seeing ticking up on our reports these days. The following chart shows this context. As always, it's hard to get recent statistics for South Africa. I started with the top 10 natural causes of death in South Africa from StatsSA [1]. I've used their attribution of the underlying causes † , Table…
I spend a great deal of time in my terminal when I am using my computer. According to RescueTime I've spent more than 40 hours in my terminal this year. I won't go into a great deal of detail in this post about the things that I do, but I think it is useful to put all of the things I find really useful in one place. Non-commands Before you can start using the terminal you need a good terminal…
Every now and again I will come across the reasoning that it is useless to measure a noisy signal frequently. My background as a control engineer tells me in fact a noisy signal should be sampled more frequently so that we can more accurately filter out the noise and in general more frequent sampling is better than less frequent sampling, all other things being equal. How often should I weigh…
I recently read an old article by Joel Spolsky which is actually a review of The art of Unix Programming by Eric S Raymond. Its title, “Biculturalism” made me think immediately of The Two Cultures by CP Snow . Unix culture Joel is a developer who understands the Windows culture very well. He worked on Microsoft Excel and is one of the founders of Stack Overflow. He really understands the needs of…
The wrong way You've started writing some code which reads from data files, and does some expensive computation which is written to some different data files. Perhaps you're doing this in a Jupyter notebook which contains the following cell: import pandas input_data = pandas.read_csv('input.csv') results_data = expensive_operation(input_data) results_data.to_csv('results.csv') Now, you want to put…
Gautrain API reverse engineering The iOS 12 update contained a very cool feature: Siri Shortcuts. This replaces the Workflows app. I was using Workflows to send my wife an SMS with my estimated time of arrival at home when I left work. With the new Shortcuts app I could have this all happen automatically, and it also supports reading from Web APIs. The first part, however, is to figure out how to…
The problem An old friend e-mailed me a challenge today. He was working with a data file which contains measurements of compounds from groundwater taps. The file looks like this (when opened in a spreadsheet program): In other words, it contains measurements for a particular date for a particular tap point and compound. He was using the following R code to plot how the Iron levels varied over time…
This post overlaps a little bit with the the previous one , recapping my programming history. We ended the previous part in 2002 when I finished my Masters and started teaching, but I want to go into a little more detail on my history with scripting languages. I encountered Matlab for the first time in my second year (in 1998) in a course called CRV220. I still have the study guide so I can say…
Original challenge: Standard library only I set myself a little challenge to write a simple spreadsheet-like program in Python in less than 100 lines, using only the standard library. The code in this repository manages it in 100 non-blank lines, including comments. 88 if you ignore the comments. The full code is also at the end of this post. Note: I have also redone the code in PyQT5. This is…
The problem One of my postgraduate students is writing a simulation code in Python. Some of the requirements have involved us generating files containing Python programs to be executed later. But while I was going through the code today I spotted this: dist_code = 'def {0}(random_number):\n values = {1}\n cum_integrate = {2}\n return numpy.interp(' \ 'random_number, cum_integrate,…
Disclaimer: This post is unresearched and contains no links as I'm just working from an idea I had this morning. I'd love to hear if I got something wrong, but I'm going fast to stop myself from getting stuck on this. In the beginning was the word Language developed long before we wrote it down. But I think as soon as we started speaking to one another we had the problem that it is possible to say…
I recently posted this tweet: The world would be better if we had one language in common, drove on the same side of the road and used the same units. #standards — Carl Sandrock (@chthonicdaemon) January 9, 2017 I was surprised by the number of people who protested on Facebook. Most of the problems were with the language rather than the other elements I mentioned. There were a couple of different…
People often maintain that they don't need a special tool like git for version management. I believe that most people are using some kind of version management strategy when working on projects, but that they use ad-hoc strategies which have specific problems. So this is not a pure "This is why you need git" post. Rather it is an exploration of the difficulties of version management and how a tool…
As our department's go-to guy for computer-related queries, I have been involved in ushering many people from Word to LaTeX and more recently Markdown (via Jupyter Notebooks ). I find myself repeating some thoughts over and over again, so I thought I would take the time to write them down this time rather than have another discussion about it. Formatting and meaning Many modern computer users use…
I use Python for most of my programming needs, and since the CPython interpreter is not particularly efficient, it is relatively common for people to write the CPU-intensive bits of their code in other languages like C, C++ or (for numerical work) Fortran. Python allows for a great deal of flexibility in defining modules in other languages and using their compiled routines with little overhead. I…
I've been seeing a lot of people complaining about the nuclear deal. There seems to be an especially vocal group who feel that we should be investing in renewables rather than nuclear. Some of these people hold up Fukushima as a warning about how bad things can go with nuclear power. There's a lot to say about this topic, and I'm not going to take the time to write a book about it, just drop some…
It's interesting how your tools affect your choices and how you can end up in a place which works for your current needs but doesn't make the next step you're trying to get to easy. To understand my current situation regarding managing my files, it's probably best to go into a little bit of history. Early computer use: no syncing I enrolled as an undergraduate in 1997. When I arrived on campus I…
A student was having difficulty using a class I had created to represent transfer function objects. I discovered that they were actually placing these objects in a numpy.matrix and trying to multiply things by one another. To my surprise, this actually worked! I had to investigate. The code here is all in Python. I created a quick tracing object which just kind of logs the operations that are done…
With the recent bout of load shedding, everyone's been writing about electricity. The problem is that many people get things wrong, or apply conventions which are less useful than they think. The basics Electricity is fundamentally about the flow of electrons. You may say that it is also about magnetic fields, but most of the terminology you'll be seeing in the articles about load shedding is…
Last week I spent a day going over the basics of the Unix shell, version control using git and some Python with engineering students. Here are some of the things I said, links to interesting tools and my insights from crowd feedback. Motivation Every year, new project students arrive on campus and are asked to do larger projects than they are used to. The methods they have used for data processing…
I spoke at a recent symposium on the difficulties in managing data interchange with industry. Just for some context, "industry" here stands for personnel involved in process control and modelling activities. My direct experience is in the petrochemical, mining and paper and pulp industries. I also show an example of data from a piece of analytical equipment. I've uploaded the source on GitHub here…