Ocassional Notes
Some Anti-virus software will monitor that certain key files, for example important binaries or the kernel, have not been modified. The way to do this is with file integrity monitoring . The general idea is to store hashes of files, and continually re-hash the files to make sure nothing has changed. Let’s look at how this might look in python. def monitor_files (path, monitor_log_file=…
Programming could be described as the process of giving a computer instructions. There are many ways to write programs that have within some tolerance the same function. Because of this arguments and philosophies arise as to what way code should be written. When starting out a lot of folks tend try to “golf” their code, make it as short as possible, while others may aim for best performance. I’m…
Dask distributed is a python package / distributed processing framework that gives the ability to parallelize code execution across a cluster. For a simple example, observe: def double (x) : x * 2 scheduler_host = "whatever" client = Client( "tcp://{scheduler_host}:8786" ) scattered_inputs = client.scatter(list(range( 20 ))) futures = client.map(double, scattered_inputs) results =…
Unit testing your code is the ultimate route to developer sanity. A well written test suite is a patchwork proof of the correctness of your system. The more complete your patchwork, the more you can rest easy knowing your code will do what you intend it to do. The typical measure of the completeness of a test suite is line coverage. A profiler running while the test suite is run checks every line…
Using the BLEMidi.h library, you can get your esp32, the goat of dev boards, to be a midi server (instrument), or client (it’s complicated). It’s pretty easy to get your esp32 to play notes: # include <BLEMidi.h> void setup () { Serial . begin ( 115200 ); BLEMidiServer. begin ( "Basic MIDI device" ); } void loop () { if (!BLEMidiServer.isConnected()) { return ; } BLEMidiServer.noteOn( 0 , 80 , 127…
Cookie clicker is a video game where the goal is to click a cookie many times. Each click increments a click counter, and as the number of clicks increases power-ups are unlocked that accelerate the counter incrementing rate. Clearly this is a silly and futile endeavor. But not all silly and futile endeavors are entirely pointless. Enter the Arduino leonardo , an IC controller that can act as a…
Like everyone else, I do my best to read many books, and like many, I find it to be a challenge, not just in finding the time, but also I found after you’ve read enough books, especially non-fiction, a lot of the book seems like filler. Different books on the same subject will cover the same material, or the author will be needlessly verbose in covering a topic. I thought ‘if I just want to read…
DBT has support for a wide variety of databases. So far, I’ve been using Google’s bigquery as the database for all of my DBT blog posts, so let’s try something different. For this demo I’ll use Apache Hive Development Environment Setup Apache hive has a quickstart guide that doesn’t quite work, so I will provide the steps I took to get it going in podman . First, a very important step (I could…
DBT supports hooks , which are a mechanism for running operations at different points of the execution of your DBT DAG. This is useful for running operations like adding a record to an audit table, running specific reports, and more. Here I will show an example based on my “enriched movies” example from my previous blog post. Consider the use case where you want to export a table to your data lake…
DBT is a tool to simplify populating the relationships between different tables. With DBT, you can specify the queries used to create your tables, as well as parameterizing portions of those queries. You can also add data tests. Introduction: Getting to Know Data Build Tool (dbt) Ever wished for a smoother way to handle your data without the headaches of complex ETL processes? Enter Data Build…
I had a visit with my mother recently where she introduced me to the idea of Sator Squares . It’s a five letter acrostic, popular in ancient Rome, and originally rediscovered during the excavation of Pompei and Herculaneum. It has the interesting property that transposition of the matrix is an identity operation. Image from wikipedia The drawing she wrote of it has been sitting on my counter for…
Just a quick note about an uninformative error I saw in bigquery the other day and was having trouble finding on google. If you see the error: MaterializedView is required for DerivationSpec When trying to create a materialized view in bigquery on google cloud platform (GCP), it means you didn’t specify the query, or including an empty query. It’s an easy mistake to make. Maybe this will help…
Beam has support for working with collections of data that conforms to a schema, and you can use SQL Transforms to transform this data. This feels a bit more like working with data in spark, but beam does not have the same level of optimization. Here’s some imports. I’ll leave these in because beam’s docs are not great and I had to copy these out of a youtube video when I was learning to do this:…
I’ve written many times about solving wordle. I still play wordle most days, but recently I’ve moved on to playing other daily games available on the NYT site, spellling bee and letter boxed . So let’s solve those too. Spelling bee is just about spelling as many words as you can, but they all have to inclue the central letter. The ideal is to get a pangram , i.e. a word that includes all the…
In a chain of method calls, what happens when one of the calls returns a null? The next chained call will throw some kind of reference error. But what if you don’t want to deal with a reference error? What if you want the null value to be the answer? That’s where a safe navigation operator is useful. Ruby has a safe navigation operator &. . This is really useful in database applications. For…
Ages ago I was working as a contractor on a data engineering team, working on a knowledge graph platform that was built on a service architecture. The platform had dozens of services. Each service had its code in its own repository. We used containers for deployment, testing, and so on. This meant we had dozens of containers to migrate over. Several of the containers were built on top of other…
I was preparing for something I needed to do at work with beam, namely running a shell command for something that isn’t possible to run natively in the beam runtime, and did not find much documentation for it. It’s relatively straight-forward but I have not “blogged” in a while so here goes. Cowsay is a notoriously difficult algorithm, which no one outside of the original author has been able to…
After using spotify for many years, I decided to try switching my music service a few months ago. I was finding that any time I put on shuffle, spotify would eventually direct me back to funk and soul music from the seventies. I generally love those styles of music, but for a service that is supposed to provide discovery, it wasn’t really working for me. I compared the various services available…
At my new job our focus is to uncover scientific insights to accelerate and improve scientific research. I’m a data engineer by trade, but the science side of the business bleeds in to what I do on a fairly regular basis. Without the scientific and research background that many of my peers have, I feel the need to do some self improvement in those areas. So I was happy to find a series of videos…
I’ve blogged several times about wordle before. It’s a great game for someone like me who loves words. I don’t think I’m a great wordle player, but I love thinking about the intricacies of the game. One bugaboo in wordle, which I’ll call “diabolical answers” is when you reach a point where you have four letters correct and there are many possible answers. The worst of these would have more than…
Apache Beam is a distributed programming framework, mostly designed as a counterpart to the DataFlow service in Google Cloud Platform. In the past I’ve done a fair bit of work on pipelines in Spark and with a service architecture, but I’ll be needing Beam for my new job, so Ive been playing around a bit with that. I’ve may have written a bit about it before, but I think of calculating π using the…
Since late 2012, I’ve been a parent. My oldest is an absolutely avid coder. He loves the scratch programming language, and has made an incredibly bevy of spectacular games, originally by following griffpatch tutorials, and then moving on to his own stuff. As a way of sharing time with him, I do some scratch of my own. It’s a fun way of spending time together and encouraging his interest. Of…
Since 2007 I’ve been keeping up my own personal word of the day blog. I started it back in the heyday of google reader. In those days I was subscribed to several word of the day blogs and wanted to do one of my own. I’ve gone through phases of posting more and less. Some years I posted every day, and toward the late teens I had single digits of posts for several years running. It’s fun. My…
Happy 2s Day. I was chatting with my brother the other day, and he opined that I may live in Canada’s largest municipality that doesn’t have a municipal bus transit system. Of course I was interested to find out which is the biggest Canadian city without buses, so here we go . I was very pleased to find that pandas will read an html page and output a list of dataframes that it was able to parse…
One of the paper handouts my child used to do at school was the “word ladder.” The exercise is a starting word and a series of instructions, with a hint/definition of the word that will be created. For example, given the starting word “jump”, a sequence like this could be created: Replace first letter: get rid of Replace last letter: bereft of speech. Replace first vowel, add er at end: wetter.…
I had access to udemy courses back when I was working at Deloitte Digital. One interesting course I did was the introduction to deep q learning from Pieran Data . Deep Q learning is a way to use neural nets to do reinforcement learning. It was an interesting course and I learned a lot, so I’ll give a recap here. Classical Q learning is a reinforcement learning algorithm that revolves around making…
My old work ( TWG ) had a 3D printer for employees to play with and use. In the summer of 2019 my son was getting into model rocket launching, but we kept losing them in trees. So I decided to try 3d printing some rockets. It never really worked out, but in my new spirit of blogging failures, I’m posting about it. My goal was to make my own easily produced reusable rocket. I am not a rocket…
Wordle is a popular word deduction game. The aim is to guess a word given six guesses. You can only guess valid English words that are in wordle’s dictionary. Each wrong guess will yield hints. A letter highlighted in yellow indicates it matches the target word somewhere. A letter marked in green indicates the correct letter in the correct position. I feel like a branching strategy (preferably…
When I was a child, they had a plotter at my mom’s office. I don’t remember the model, but it was something like a HP 7550A . I was fascinated by it, and I’m sure I watched it go for many hours. From that, I’ve always loved plotter technology. I’ve always kind of wanted to make my own. Last year, the urge became too great and I tried out the branchiograph project. I didn’t get the best results for…
Non-negative matrix factorization is a fast technique for generating embeddings from a dataset. More concretely, given a matrix you can decompose it into two matrices that are approximattely multiplicands of the matrix. I.e. given matrix M , it finds matrices A and B such that M ≈ AB . A very cool thing about embedding matrices is that the rows and columns that comprise these matrices naturally…
The 100 emoji is iconic; it’s a symbol recognized across cultures and it has a semiotic impact globally. In a world where most societies have standardized around a base-10 (or decimal) number system, a doubly-underlined 100 is a shining beacon of success, or total mastery. The forthright and striking symbol of flawless execution, total knowledge, and success lends itself naturally, in this remix…
I’ve been messing about with blender a bit since the pandemic started. I originally thought I could use it for visualizing data, which is certainly true, but I’ve found more that I’ve been getting in blender-world. I was watching a “coding adevntures” video over the weekend where the author described using fibonacci spheres as a way of evenly distibuting points on a sphere. But I couldn’t find an…
I wrote a while back about a script I wrote to run on my android phone that would sort and archive any photos I had on the device into dated folders I have on a network drive. Privacy and de-googling concerns led me to get an iphone when the time came to upgrade (essential, my old phone’s manufacturer shut down, and faced with an aging battery and a looming lack of support I decided to upgrade).…
I was super excited to find that I can install bash on my phone, then scp and ruby. It opens up a lot of possibility of what a phone should really be able to do. Too often I feel constrained by limited options in apps. An example of this is taking old photos and putting them into my long term storage on my home computer. I backup to google photos, but I like to keep a local copy too. My old…
I was reading an article on Fizzbuzz . Across the various languages used, the article has two recurring techniques. For each technique listed here, I will give it a rating on a scale of my choosing. The two techniques listed in the article are: 1. conditional logic This is the standard technique that everyone, regardless of experience, would probably think of first. The following example will run…
Here’s a talk I gave at work that gives a run-down of some fundamental concepts in deep learning, and I finished the talk with an example of a convolutional neural network in keras.
Javascript is going to get a pipeline operator any day now, and I love working with it in elixir, so I thought I’d try to make one in ruby. So here’s an implementation class Object def pipe arg arg.( self ) end end And it works like this: plus1 = -> (x) {x+ 1 } 1 .pipe(plus1) # 2 1 .pipe(plus1).pipe(plus1) # 3 And you can do things like add a logging aspect: putit = ->(x) {puts x; x} 1…
I tried using j for some of my advent of code puzzles. I found it wasn’t totally suitable for a lot of the challenges, but I had fun learning about J nonetheless. J is a declarative and functional array programming language. I like this quote from J’s new user guide : J isn’t just another way to declare variables and write loops. J is a way of thinking big: describing an algorithm by looking at it…
I had an issue at work where some database writes were throwing errors within a thread, and the error was being silently eaten when the thread died. The solution is to use a queue for inter-thread communication, like this: q = Queue.new Thread.new do i = 1 loop do begin sleep i i += 1 raise 'whatever' rescue StandardError => e q << e end end end loop do puts q.size end
Bayes’ theorem is a way of determining the likelihood of an event A given that another event B has occurred. It’s a way of making an educated guess without much information to go on. It’s a way of going from knowing the likelihood of evidence, given a known outcome to knowing the likelihood of a known outcome given existing evidence. E.g. What’s the likelihood tacos are for dinner? If we know the…
rein is a gem for adding database constraints in rails migrations. It’s always been possible to set these up using execute calls in the migration, but rein makes it look rails-y. Letting your database manage constraints is often a great idea. I’m not a huge fan of the rails-way of letting the application layer manage all relationships between data. My main concerns with this are: A developer…
I am still working on picking up some elixir , so when a coworker mentioned that writing a Slack bot is cool, I decided to give that a try. Slack is a bit easier to work with than IRC because the communication is done in JSON so you can skip a step in message parsing. The login process for a slack bot is: Authenticate over HTTP using your bot’s token. Open a websocket connection using the socket…
I was talking to a colleague who was expounding about the coolness of elixir . So I thought I should give it a try. In the spirit of doing something non-trivial to learn a new tool, I decided to port over the IRC bot I’ve been working on in go and ruby (which passes my bar for non-triviality). You can check it out here . It does the same super-simple (but easily extensible) bot behaviour of…
I saw an article on twitter about writing an IRC bot in ruby . It piqued my interest because of nostalgia and because I’m a slack curmudgeon. I believe that IRC has value. I never actually got around to trying the cinch gem that the article recommends, although if I ever need to do anything serious in this realm it wil be the first tool I turn to. Instead I found an article about writing a really…
My work ( TWG ) has a 3D printer for employees to play with and use. Sometime last summer I downloaded some train tracks from thingiverse and printed them for my son to play with. He loves the custom tracks dearly. He’s always excited to visit my work to see what a coworker called “the toy factory”. And often he will ask me to print him something while I’m at work. So I get a lot of emotional…
A coworker recently filled my ear with praise for vue.js . I’m always eager to try new frameworks, and the benefits over angular (namely performance and simplicity) sound really promising. I haven’t had a chance to develop anything serious with this, but my initial attempt was exhilirating. I saw a link to the xkcd api in a tweet the other day, and decided to try it out with vue.js. I don’t really…
Last month a workmate and I flew down to San Francisco to go to elasticon. I attended an interesting talk given two employees of Giant Oak . Giant Oak does contracting for government agencies to try to solve social problems. I am not sure if this is their motto but one of the speakers say that they “see the people behind the data,” which sounds really cool. Their work involves things like trying…
In irb , fork Now typing exit will not work because your keystrokes might go to either the parent process or the forked child process. I found it to be impossible to get the characters exit to coherently all go to the same process.
For my “I had trouble finding this on google” series, here’s a solution I found to getting a total duration from a collection of objects with durations stored as Time objects. total_duration = durations.reduce(Time.gm( 2000 )) do |total, time| total += time.to_i end There are two little tricks happening here. The first is that durations are stored as a time of day on January 1st, 2000, so we use…
Every programmer has some level of interest in knowing how to write a language. I’ve always wanted to tinker with this level of software development, but even after years of development experience you never really get exposed to this stuff because what’s already there generally just works. On top of that, it’s a somewhat complex process involving several steps with intimidating terminology. But…