Recent content in Vignesh Ravichandran on Home
The last line of defense for those OOPS moments — so even DBAs can sleep peacefully. Have you ever accidentally run a DELETE without a WHERE clause? Or typed DROP TABLE thinking you were in staging, only to realize a half-second too late that you were in prod? If yes — keep reading, this is for you. If no — keep reading anyway. Nobody who uses Postgres long enough stays immune.
Seven attempts, seven different wrong answers — lessons from setting up a local LLM. It’s tax season, which means I’ve been staring at a notes file full of stock sales — 23 transactions across the year that I needed to total up. The kind of data I’d rather not paste into a chat window I don’t control. I’d been meaning to set up a local LLM anyway, and this seemed like…
The analytics databases optimized for queries. The real leverage was always where data gets written. Your application writes to Postgres. So does almost everyone else’s. Postgres is the most widely used database in the world, topping Stack Overflow’s developer survey two years running. It didn’t win as a data warehouse. It won as the place applications write. For years, the data…
I was writing an article for Infoq on the topic less spoken costs of managed databases and one question that the editor asked is how much margin RDS makes compared to running a Postgres instance on EC2? That intrigued me and honestly, I never did the math so far. I have been using AWS-managed databases since 2016 and I thought what’s a better time to do an analysis of Cloud cost than today ?
I was trying to play with Phil’s pgtam and the first step is that to install Postgres version 16. Sounds fairly innocous. When I ran configure, I was getting the below error: checking for icu-uc icu-i18n... no configure: error: ICU library not found If you have ICU already installed, see config.log for details on the failure. It is possible the compiler isn't looking in the proper directory.…
Abstract On a high level, the talk addressed following questions: What is Edge and why are the benefits of edge? What are the challenges of implementing Postgres at the edge? How we implemented it at Cloudflare? Where Postgres at the edge is heading? Slides from the presentation Link from Developerweek Cloudx Link from Developerweek Cloudx virtual
Abstract We talked about the importance of having a good toolset, of practicing incidents and of internally advocating database best practices to a large engineering organization. Recordings of the presentation
Abstract With the advent of cloud/managed offerings, running Postgres on their own hardware is becoming rare. It’s great that a lot of the Ops work is taken care of by the provider, however, understanding a layer or two beneath these abstractions will be useful for anyone in strengthening their knowledge. For eg. When you can’t connect to an instance how do you find where the problem…
Abstract This talk explores how Cloudflare uses HAProxy for health checks, load balancing and reading traffic among nodes set up with Postgres streaming replication in hot standby mode. Cloudflare operates multiple Postgres Clusters across four data centers, and all of these clusters are made up of six nodes. During a primary failure, Cloudflare’s high availability system promotes a replica to…
I kept hearing about the term wire protocol especially Postgres wire protocol in the recent days (Looking at you cockroachdb, yugabytedb - in a good way) but never really quite understood it. Decided to implement something simple in Go to understand it better. As always, if you find anything wrong or I misunderstood please correct me. In simple terms, “wire” - something over network…
I use pg_rewind for rewind a Postgres cluster which is ahead of the primary. Yesterday, after the rewind Postgres didn’t start. It failed with the below error. This is using 9.6 version pg_rewind client. Sep 2 20:06:14.371503 productiondb[1]: [1-1] time=2022-09-02 20:06:14.338 GMT,pid=78197,user=admin,db=postgres,client=[local],appname=[unknown],vid=,xid=0 FATAL: the database system is…
Citus: Distributed PostgreSQL for Data-Intensive Applications paper can be downloaded here. Recently, our team got a request to provide a solution to shard Postgres. One of the solutions that we discussed was Citus. I have heard about the product and seen their blogs related to Postgres in the past but never used it. I thought it would be fun to read about its internal workings. If you find…
TL;DR: Connect with people whom you admire. Things can take longer so stay in touch. At the right moment, don’t hesitate to ask. I recently became an Engineering Manager after being an Individual Contributor for almost 8 years. If you are interested to know more about the reasoning behind the change. As like many other jobs in Tech, there is no formal training for an Engineering Manager.…
Capturing thoughts from https://twitter.com/viggy28/status/1530800893842444289 When you are doing major DML changes, other than locks one more thing to keep in mind is replication lag. Especially if you use your replicas in hot standby mode. When you need to delete most of the records in a massive table, its better to create a new table and just copy the records that you need to preserve. When you…
Before I forget let me write this down here. $$ log_stament - none - all log_min_duration_statment - millisecond value When you set log_statement=none and log_min_duration_statement=1 then any statement which takes longer than 1 millisecond will be logged. When you set log_statement=all and log_min_duration_statement=1 then all statements are logged; however it only shows duration on statements…
Last week, I got promoted to Engineering Manager (EM) of the Database team at Cloudflare. I joined the team almost 3 years ago. A senior team member and myself were hired to form a then dedicated database team. I have been enjoying it and the team is amazing to work with. Great people. Opportunities came to either advance on the Individual Contributor (IC) or the leadership side. I picked the…
I saw this pattern of a function taking optional parameters (variadic arguments) and I never understood why they do that until today. There is a bit of a background on why I need to use functional pattern. Let’s say I don’t use functional option. Then, all the arguments need to be passed. Also, difficult to set default values and implement logic for default values. package main import…
double pointers A pointer (whose value is another variable’s address) which points to another pointer copying a pointer to another pointer copies the address copying a de-referenced pointer to another de-referenced pointer updates the value of the one to another https://gist.github.com/viggy28/604cda186ff1b872d562b31ca3976732 package main import "fmt" type DB struct { Name string } // double…
context is a standard package. Context is an interface in context package. type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} } There are 4 methods in Context interface. Most of the times, we don’t define our own types which satisfy the interface (which off course we can do), but we mostly use the factory…
go run command go run command is a quick way to run go program. However, what it does behind the screen is it compiles the program and runs an executable and removes it once the program is stopped. byte is a built-in alias of uint8. rune is a built-in alias of int32. The semicolon insertion rule In Go, I thought there is no need for semicolon at the end of the statement.
closure is a function inside another function where you can reference variable defined in the outer function. I have seen them useful in dealing with packages where a function expects an type func() as an argument where the func() doesn’t take any argument. example from package github.com/cenkalti/backoff signature of backoff.Retry is as below backoff.Retry(o backoff.Operation, b…
From the wiki, In computing, data storage, and data transmission, character encoding is used to represent a repertoire of characters by some kind of encoding system that assigns a number to each character for digital representation. Okay, they mean converting character to some number for storing and transmitting data. There are two popular character sets 1. ASCII 1. American Standard Code for…
There are two types of variable declaration. recursively expanded variables - foo is assigned with bar which inturns assigned to ugh whose value is Huh? foo = $(bar) bar = $(ugh) ugh = Huh? all:;echo $(foo) $ make $ Huh? simply expanded variables - foo is assigned with bar however bar is not defined yet. so it outputs to nothing. foo := $(bar) bar := $(ugh) ugh := Huh?
I have know about the postgres wire protocol, but first I ran into streaming replication protocol. I was looking at this code in stolon replConnParams["replication"] = "1" db, err := sql.Open("postgres", replConnParams.ConnString()) if err != nil { return nil, err } defer db.Close() rows, err := query(ctx, db, "IDENTIFY_SYSTEM") if err != nil { return nil, err } I was wondering what is this…
defer is one of the unique keywords in Go. What does defer do? As the name suggests, it defers something. It defers function calls. A deferred function gets invoked right after the surrounding function returns. func outer() { defer inner() log.Println("outer got called") } func inner() { log.Println("inner got called") } outer got called prints first then inner got called. What the main use of…
I came across this piece of code from The Go Programming Language book. loop: for { select { case size, ok := <-fileSizes: if !ok { break loop // fileSizes was closed } nfiles++ nbytes += size case <-tick: printDiskUsage(nfiles, nbytes) } } The complete program is available in Github Named break statement allows to break perhaps multiple control statements. In this example, break loop breaks the…
I was going through The Go Programming Language and in chapter 5 there was a section Caveat: Capturing Iteration Variables. The example is like this: var rmdirs []func() for _, dir := range tempDirs() { // 1 os.MkdirAll(dir, 0755) // 2 rmdirs = append(rmdirs, func() { os.RemoveAll(dir) // NOTE: incorrect! // 3 }) } dir is the same variable. i.e. the address of the variable is same dir’s…
program1 package main import ( "log" ) func main() { done := make(chan struct{}) go func() { log.Println("done") }() <-done //wait for background goroutine to finish } program2 run nc -l 8000 on another terminal package main import ( "log" "net" ) func main() { conn, err := net.Dial("tcp", "localhost:8000") if err != nil { log.Fatal(err) } conn.Close() done := make(chan struct{}) go func() {…
Initializing embedded field: The field name of an anonymous field during a struct instantiation is the name of the struct (Time) type Event struct { ID int time.Time } event := Event{ ID: 1234, Time: time.Now(), } If an embedded field type implements an interface, the struct containing the embedded field will also implement this interface
Concurrency channels are like typed pipes, Where you can send something from one side and receive from other side. using channel direction operator <-. Depends on the direction of the variables around the operator, its either sending or receiving. x := <-c // receive from c c <- sum // send sum to c fmt.Println(“channel receive happening”, <-c) channels are reference type. similar to…
In Go, copying generally does a copy by value. So modifying the destination, doesn’t modify the source. However, that holds good only if you are copying value types not reference types (i.e. pointer, slice, map etc.) package main import ( "fmt" ) func main() { type Cat struct { age int name string Friends []string //reference type } wilson := Cat{7, "Wilson", []string{"Tom", "Tabata",…
In v6, /48 is the minimum prefix that can be advertised by the routers in the Internet. I believe in v4, its /24. Until yesterday I wasn’t aware of that. I might have thought you could even advertise one IP (ie. /128 or /32) but that’s not possible. Ref: https://blog.apnic.net/2020/06/01/why-is-a-48-the-recommended-minimum-prefix-size-for-routing
This is something new for this site. In fact, First of its kind. Mostly, I write about Tech, Finance, books etc. Recently, I thought about writing/journaling more of my personal life and experiences. Also, interactions with the professional people whom I met. If you have known Sri or myself personally, we moved to San Francisco from Boston (Sri) and Los Angeles (myself) in 2019. We don’t…
namespaces is a Linux Kernel feature. There are different types of namespaces - Partitions Kernel resources such that one set of Processes see a different resources while others different - Apartment complex analogy - 7 different namespaces - PID namespace - Watching TV show - Net namespace - each namespace gets an unique IP address s list of port. - apartment address analogy - Uts namespace -…
I have seen this term here and there. I remember fiddling with this in pain of setting up port forwarding in xfinity/. DHCP - Dynamic Host Configuration Protocol (DHCP) is a network management protocol used on Internet Protocol (IP) networks for automatically assigning IP addresses and other communication parameters to devices connected to the network using a client–server architecture (from wiki…
I have used many signals with kill like SIGHUP, SIGKILL etc. Today I came across USR2. Did a Google search and found the manual page from GNU. According to it, These signals are used for various other purposes. In general, they will not affect your program unless it explicitly uses them for something. Another one from the BSD mailing list USR2 is a "user defined signal" (from "man signal") It…
There are tools like https://tehnoblog.org/ip-tools/ip-address-in-cidr-range/ which does the job. It doesn’t support IPv6. One of my colleagues today taught me how to do it by hand quickly. Say we have an IP 2b06:4600:1101:0:abcd:efa:dbd:ea60:f5a6 is in the CIDR 2b06:4600:1101::/64 First step is to check what’s the mask bits. Here it is 64. So you take the address, take the first 64…
Disclaimer: Whatever I write here is my learnings. So, don’t take these words as granted or reference. Also, please lmk if there are any errors. I have often come across this term JWT. I knew the definition - Json Web Token, but honestly that’s all I knew for a veryyyy longgg time. Recently, I came across a problem where I have been told that JWT might be a good solution for it.
This is part of my Today I Learnt series where I share whatever I am learning something new. After having issues with port forwarding in Xfinity I decided to look for alternative solutions. I have used Cloudflare tunnel (used to be called Argo Tunnel) in the past to expose websites running on my laptop to the Internet. So, I decided to try it out for Ghost blogging site which I am setting up for…
This article is part of my Today I Learnt. I have been trying to share whatever I learnt newly on the day. Some of them are basic, but still I want to write it down for two reasons. 1. To help me understand the concepts better 2. Potential help other people who running in to same issue. Alright, It’s all started when I bought a new domain (synergy.net) for my Dad.
This is a crosspost from my TIL What the heck is .PHONY in makefile? I have seen this for so long but I always shrugged off. Whatever that is. Well, finally I decided to learn about it. A bit of introduction to Makefile syntax. target: prerequisites <TAB> recipe The first target in a Makefile will be executed by default when we call make. blah: blah.o cc blah.o -o blah # Runs third blah.
Libpq-Postgres-Go After I started working on Postgres, I have heard this term libpq enough times, but never had a good grasp of it. After digging around this topic for a couple of days, here is my understanding. From the Postgres doc https://www.postgresql.org/docs/9.5/libpq.html, libpq is the C application programmer’s interface to PostgreSQL. libpq is a set of library functions that allow…
This is a guest post by Suvam Prasad (https://www.linkedin.com/in/suvamprasad/). I am very glad to have his post. This book is very special to me because it helped me to discover new perspectives of web designing mythology. The book is written by Steve Krug(a famous user experience professional). The book contained hardly 216 pages that give you enough knowledge and idea to understand web design…
Understanding context in Golang through Postgres I was trying to learn Go package context especially with respect to Postgres. On a very high level context provides context to the operation. Yeah, I agree, the previous statement doesn’t really add much value, but hold on I don’t really know how to explain it, rather let’s go over some code. Sometimes its easier to understand by…
TLDR; A short book on Product Development and especially explains how B2C (especially social media) companies make their users hooked on their products. Though Product owners, designers, and founders can use this book in improving their products, every user can read this to understand why some of the popular apps do things (like infinite scroll, weekly digest emails, notifications) which we use…
Disclaimer: This book was published in 1989. So, we have to time travel to understand a few things, but overall the concepts are still relevant. Also, read this during COVID-19 (middle of a global pandemic). This review is a little different than other books. Going to share most of the notes as points. Peter Lynch is America’s money manager. He was Vice-Chairman of the Fidelity Management &…
Part 1 is about Master Internal Triggers which talks about that distractions are mainly caused by discomfort. Most people don’t want to acknolwedge the uncomfortable truth that distraction is always an unhealthy escape from reality. Being indistractable means striving to do what you say you will do. We are compelled to reach for things we supposedly need but really don’t. (eg. checking emails) We…
Here is a list of books which I really enjoyed reading. They are mostly around the topics such as Self Development, Management, Finance, Technology. Give and Take by Adam Grant Atomic Habits by James Clear A random walkdown wall street Designing Data Intensive Application surely you’re joking mr. feynman How To Win Friends and Influence People Lean Startup The Go programming languate Make your bed…
Chapter 1 is Good Returns which explains the Dangers and Rewards of Giving more than you get with multiple examples. One of the things which striked me You can’t just ignore someone because you don’t think they’re important enough. I can’t agree more on that. I have seen first class that founders making this mistake. Its so easy to just ignore, but think about the long term or big picture.
This aricle is inspired by Give and Take by Adam Grant. I was wondering how can I help my friends and network. Searching a job is something stressful and we all have been through. Once my wife and I searched for a year and finally we moved to San Francisco. So far its been a challenge for my network to know whether I can refer to a particular company. To solve this, I am publishing a list of…