Recent content on Explorations - computing, research, etc.
Some examples of where i get limited by a specific programming language/framework’s syntax/particular way of doing things, and how I navigate it by trying to see if i can solve it myself based on what i know or good old internet search, (i.e. manually looking at links/posts) and learning a bit more in the process. I know there is research now showing the impact of using LLMs on our brain,…
Some suggestions that i offered to someone else, looking for some general guidance around setting up their DevOps processes. Overall, the idea is that you run as many automated checks/processes as possible before the code is merged into your main/master repository and deployed to production so that you can: Prevent bad code from being merged - bringing down your systems post deployment Ensure code…
In what can only be described as sign of the times, one of the books I wrote, arguably, the most impactful one, Doing Math With Python is one of the works in the Anthropic Copyright Settlement. This is just a post to commemorate this time in history and in a bizarre twist, me being part of it. Off late, I have realized, I don’t like an “agent” doing my software work.
In this blog post, I am going to desribe how we can calculate sepsis onset time using sepsis-3 criteria for the MIMIC-III patient data. Reading the relevant tables Sepsis-3 Criteria Operationalization in code Identifying the onset of suspected infection Time of drawing of the first culture Time of administering of antibiotics Identifying infection onset Identifying organ failure/dsyfunction…
I remember the days of reading API documentation, and figuring things out and experiencing a joy having done so. The experience was enriching and that’s really what enabled me keep doing it, because of a sense of joy I derive from it. I remember the days, when that was just how things happened. Now, my first call is to open my favorite LLM powered tool and ask it to do what I need to be…
I recently downloaded MIMIC-CXR-JPG, from Google cloud storage. The reason I had to eventually use Google cloud storage download was because of the suggestion that was offered for working around the bandwidth constrained physionet.org servers. These are some notes that might help someone else. Before you get started: Right off the bat, the first thing you want to keep in mind is the bucket’s…
In this post, we are going to discuss an implementation detail about how we can use a custom HTTP client to use the Go GitHub SDK and the https://pkg.go.dev/golang.org/x/oauth2 package. Introduction Using a custom HTTP client with golang.org/x/oauth2 Using the custom HTTP client with Go Github SDK Summary References Introduction Let’s create an example *oauth2.Config for communicating with…
In this post, I am going to share what I have learned about writing HTTP client middleware in Go. Let’s get started! HTTP client and Transport Writing your own RoundTripper implementation Returning static responses Summary HTTP client and Transport Go’s http.Client defines a default value for the Transport field when one is not specified: type Client struct { // Transport specifies the…
I delivered a talk at the recently concluded Kiwi PyCon XI. Checkout some photos of the event from Kristina DC Hoeppner here. I attended it virtually as I was unable to travel due to last minute interruptions from a certain virus. The organizers were of course very well prepared for that. Massive kudos to the Next day video team and everyone else involved in the process, including the paper chair…
I delivered a talk at the recently concluded PyCon US 2022 conference. Amazing conference, great work by the organizers and everybody involved. This was my third time at PyCon US and my first time participating in the post conference sprints. I contributed to the Pyodide project during the sprints, and was a great experience to work with the core developers during the sprints and for a couple of…
At PyCon US 2022, Anaconda announced PyScript: Python in the Browser. So far my understanding is that it builds on Pyodide and makes it magically easy to bridge the world of the Browser - the Document Object Model (DOM) and Python. It’s so magical that you can simply copy scripts that you were running using Python installed on a computer and they just run in the browser. Check out the blog…
I saw a computer for the first time at school when I was 10 years old (~1995/96). I failed horribly in the first few years to actually get programming. Then i had this teacher with whom i took group classes, who taught me computer programming, properly. GW Basic, then some Java, some C and C++. I picked up Linux, installed Red Hat Linux 6/7 i think on a PC I managed to acquire citing that it will…
I have finally gotten around to reading (listening) “The Code Breaker: Jennifer Doudna, Gene Editing, and the Future of the Human Race” by Walter Isaacson. And in one of the first chapters, the writer talks about Jennifer Doundna’s thoughts (I think) on how often it’s the practical application of science or in solving a problem, we improve/revise our understanding of the…
Update: Please note that my usage of “Call by reference” is not correct here as pointed out by a Reddit community member here. Read it as “Call by passing pointer values”. While trying to understand this a bit more, I came across a few more links to the topic. You can find them here in my reply. I will do a follow up post on this topic perhaps. I am grateful that the reddit…
While working on the solutions to exercises for my soon to be published book Practical Go, I needed to implement a way to implement an option in my command line application which could be specified multiple times. The result would be that all the values specified would form a list of the values. To make that concrete, consider that you are writing a command line HTTP client application. You want…
While working on the solutions to exercises for my soon to be published book Practical Go, I wanted to write a test function which would simulate a user not providing an interactive input when one was asked for. However, I noticed that the test function would not wait for me to provide the input and just continue the execution. Consider the following test function: import ( 'bufio' 'fmt' 'os'…
I always get confused between “serialization” and “deserialization”. Perhaps, that is because I am trying to memorize what they are, and then trying to recall from memory. Of course, it’s sufficient to remember only one of them correctly. So, here’s my trick that I am going to use from now on. It is derived from this Wikipedia article on the subject, in the…
One word that has always baffled me in the context of software/computing is “transparency”. “This change is completely transparent to the application” - or some version of it is often used to tell consumers of an application/library that a certain change won’t be noticeable by the consumer at all. If I were to put in my English language hat on (a non-native at that), i would think that means “so,…
I delivered a talk at the recently concluded PyCon Australia conference. Amazing conference, great work by the organizers and everybody involved. Fantastic experience! My talk My talk was in the DevOops track titled “Planning for Failure using Chaos Engineering”. Since I joined Atlassian, I have come to realize the value chaos engineering/wargaming brings to the table when it comes to assess the…
In Go, there are a couple of ways to return values from a function. Non-named return values Until today, I had been exclusively using the following style of what i am going to refer to as “Non-named return values”: func myFunc() (int, error) { return 1, errors.New('An error') } You declare in the function signature that you will be returning an int and and an error. Then in your code,…
The Go module path enforces certain restrictions as expected on what constitutes a valid path. Try running go mod init https://foo.bar/baz for example. Now, what if you as a Go programmer needed to run this check yourself? That’s where the golang.org/x/mod/ package comes in. It has a number of functions, one of them being the CheckPath function, which you can use as follows: // Using Go 1.16…
Using a file for persistent storage (and not a database - datastore/object store) sounds like an academic exercise. For me, it brings back memories of writing a structure in C (programming language) to a file to simulate a student record database. However, there may be situations where you may just get by using it especially when you just want to run a single copy of your application. Let’s…
Demo - Embedding a template Demo - Serving files from a directory Learn more The most exciting feature for me in the Go 1.16 release is the new “embed” package which allows you to embed a file contents as part of the Go application binary. This ability so far was most easily available via using various third party packages and they worked great. You could also use go generate to roll…
In my latest article for the folks at learnk8s, I write about establishing authentication between services deployed in Kubernetes. Specifically, we discuss how you can use the Kubernetes primitives - Service accounts with a new feature - Service Account Token Volume Projection to setup authentication between two HTTP services. You can find the article here with the accompanying code repository…
My two recent articles, Validating Kubernetes YAML for best practice and policies and Enforcing policies and governance for Kubernetes workloads looks at the topic of enforcing policies for your Kubernetes workloads. Check them out and let me know if you have any comments.
Introduction Gatekeeper allows a Kubernetes administrator to implement policies for ensuring compliance and best practices in their cluster. It makes use of Open Policy Agent (OPA) and is a validating admission controller. The policies are written in the Rego language. Gatekeeper embraces Kubernetes native concepts such as Custom Resource Definitions (CRDs) and hence the policies are managed as…
Introduction In this post, I will share some of my learnings and explorations on plugins in Golang. We will write a “driver” program which will load two plugins and execute a certain function which are present in both of them. The driver program will feed an integer into the first plugin, which will run some processing on it. The result of the first plugin is fed into the second plugin…
Welcome to this new blog post! Introduction Enforcing policies Using kustomize to manage policies Rolling the policy changes out Multiple matching policies Conclusion Introduction Pod security policies are cluster level resources. The Google cloud docs has some basic human friendly docs. A psp is a way to enforce certain policies that pod needs to comply with before it’s allowed to be…
Metadata I have posted this article on dev.to where I welcome comments and discussions. Introduction Log forwarding is an essential ingredient of a production logging pipeline in any organization. As an application author, you don’t want to be bothered with the responsibility of ensuring the application logs are being processed a certain way and then stored in a central log storage. As an…
This blog is managed as a git repository and I use Hugo as the framework for managing it. I am using the Hugo classic theme which I have tweaked slightly and store it along with the blog source. I wanted to modify the sorting of the blog post titles on the index page so that the most recently modified page was displayed first. The support was added to Hugo 4 years back, so the following template…
Introduction In this post, I will describe my experiments with using Cloud Custodian to perform various tasks usually falling into the bucket of compliance and sometimes convention. I encourage you to take a look at some of the example policies. Some of the areas I will cover are resource tagging and unused resources across multiple AWS accounts. Installation and setup Cloud Custodian is a Python…
A reader of my book “Doing Math with Python” wrote to me a few weeks back about a strange problem they were having. They were trying to create an animated projectile motion from the code listing in the book. However, they were not seeing the expected results. Worse, there were no errors. They figured the issue on their own eventually since I didn’t get the time to reply back and…
Introduction There are two broad discussion points in this post: Managing the lifecycle of Kubernetes YAML manifests Static guarantees/best practices enforcements around Kubernetes YAML files before they are applied to a cluster Prior art and background Please read this article to get a more holistic view of this space. What follows is my summary of what I think is the state at this stage and how…
Please note this document is currently in progress. You may read this other post instead which illustrates what I wanted to discuss in this post with an implementation in Go. Introduction In a service oriented architecture, more popular these days as a microservice oriented architecture it’s notoriously difficult and time consuming to find out whether a certain error observed in a particular…
Since around 1.5 years ago, I have been working in roles which has required to automate various things usually classified as “infrastructure and operations” work. Sometimes they have involved working with cloud infrastructure and at other times they have involved interacting with databases and other services via HTTP APIs. During this period, to solve such tasks I have started using…
Introduction The docker workflow plugin enables leveraging Docker containers for CI/CD workflows in Jenkins. There are two broad patterns one would generally use containers in their CI/CD environment. The first would be as “side car” containers - these are containers which run alongside your tests/other workflow and provide services such as a database server, memory store and such. The…
Monday was just beginning to roll on as Monday does, I had managed to work out the VPN issues and had just started to do some planned work. Then, slack tells me that new deployment had just been pushed out successfully, but the service was actually down. Now, we had HTTP healthchecks which was hitting a specific endpoint but apparently that was successful, but functionally the service was down. So…
Introduction When implementing a solution for allowing users other than the cluster creator to access the cluster resources we are faced with two fairly old generic problems - authentication and authorization. There are various ways one can solve these problems. I will discuss one such solution in this post. It makes use of AWS Identity and access management (IAM) features. This in my humble…
Repeating the same argument to Printf If we wanted to repeat the same argument to a call to fmt.Printf(), we can make use of “indexed” arguments. That is, instead of writing fmt.Printf("%s %s", "Hello", "Hello"), we can write fmt.Printf("%[1]s %[1]s", "Hello"). Learn about it in the docs. Multi-line strings Things are hassle free on the multi-line strings front: package main import (…
Introduction This in-progress page lists some of my findings while working with Kubernetes. EKS cluster setup You may also find this guide from spacelift.io useful. This section will have findings that are relevant when working with an AWS EKS cluster. Terraform configuration for master This is based on the tutorial from the Terraform folks here. Unlike the tutorial though, I assume that you…
Introduction I wanted a Nginx configuration which would satisfy the following requirements: Any example.com requests should be redirected to www.example.com The above should happen for http and https http://example.com should redirect directly to https://www.example.com Solution We will need four server blocks: http - example.com (listen on 80) http - www.example.com (listen on 80) https -…
Welcome to this new blog post! Introduction Setting up DIY bash completion Data provided to completion handlers Single <TAB> and double <TAB><TAB> Getting good old BASH completion back Magic of bash-completion package compgen built-in command Learning more Bash completion for applications cobra (Golang), click (Python) and clap (Rust) complete (Golang) and shell_completion (Rust) Conclusion…
Some notes on PostgreSQL which you may find useful. Thanks to all those numerous StackOverflow answers that helped me do my job at hand. Schemas and Database The database is the highest level of organization. A database can have one or more schemas. The public schema is present by default and all tables created are created in this schema, if not otherwise specified. Learn more about it in the…
I was debugging a issue where we were getting truncated logs in ElasticSearch in the context of a setup as follows: Application Logs -> Fluentd (logging) -> Nginx -> ElasticSearch The original problem turned out to be on the application side, but my first point of investigation was what are we getting on the nginx side? Do we get the entire message that we are expecting and something is going on…
I wanted to setup Nginx logging so that it would perform GeoIP lookup on the IPv4 address in the X-Forwarded-For header. Here’s how I went about doing it on CentOS 7. This nginx module integrates Maxmind GeoIP2 database with the RPMs being available by getpagespeed.com. Once I had installed the module, the hard part for me was how to get the data I wanted - city, timezone information and…
In this post, we will see how we can use Golang to generate Terraform configuration from a TOML specification. That is, given a TOML file, like: subnet_name = 'SubnetA' rules = [ {rule_no=101, egress = false, protocol = 'tcp', rule_action = 'allow', cidr_block = '127.0.0.1/32', from_port = 22, to_port = 30}, ] We will generate: # This is a generated file, do not hand edit. See README at the # root…
docker logs by default shows the container’s stdout and stderr logs. However, what I discovered was that the stderr logs from the container are output to the host system’s stderr as well. I was expecting everything from the container to be on the host’s stdout. Let’s see a demo. Consider the Dockerfile: FROM alpine:3.7 CMD echo 'I echoed to stdout' && >&2 echo 'I echoed to…
I love working in software. Mostly things work as expected, but at times no. I changed A, how can Z be affected - after all they are all miles apart. Right? Wrong. Z can be affected. Today’s story is the latest - totally unexpected, but not surprising. Background We run ASP.NET compilation on our code base as part of every build in our Continuous Integration (CI) pipeline. I changed some…
Recently, I wrote two articles about using traefik as a reverse proxy. The first article discussed deploying a ASP.NET framework application and the second discussed deploying ASP.NET core applications. In both cases, I demonstrated the following: Docker native integration In-built support for LetsEncrypt SSL certificates One of the things I didn’t discuss was how we could setup an…
Windows docker images can be bulky and on a server that you are deploying your application as docker images, the free disk space becomes a metric to watch out for. The following script will setup a Scheduled tasks to be run at a 7.0 PM UTC which will prune all unused images: # Scheduled tasks if (-Not (Test-Path 'C:\ScheduledScripts')) { mkdir C:\ScheduledScripts } $command='docker image prune…