An amazing website.
I am not completely new to React. A couple of years ago I spent a few weeks struggling with frontend tools to create 49travel . I learnt a lot about tooling, especially Vercel, a little bit about React but I can hardly say I really understood it. Now it is part of my job, so I do want to understand it better. And what better way to understand it than by trying to write a blog post about it. A side…
In Part 1, we talked about the “Why”. Why write a spec? Because English is ambiguous, and debugging design flaws in code is expensive. We looked at Quint and modeled a simple TCP Handshake. We verified that our logic was sound (no safety violations).
Formal methods, sounds very… formal! But I have been trying to explore what they are and so I decided to write down my learnings. Hopefully this proves a good starting point for someone else who is curious about the idea but doesn’t have a good starting point.
The last couple of posts have been about 49travel and the way I built it. This week I am going to be talking about decorators for a bit. It seems very disconnected from the last posts, but I started reflecting on this while writing some Python for 49travel. So there’s still a connection!
Last week I talked about how I created 49travel. I went over broadly on the ingredients and often glossed over many details. This week I want to talk about one particular aspect which was pretty interesting for me. It was a nice introduction to the various projects going on in the LLM world that are being furiously worked on since the entry of ChatGPT.
49travel
In our last couple of posts we looked at how we could optimize pandas functions. This post will be different. We still want to address issues that we face in the PySpark world, but today we will not look at performance at all.
In our last post we saw that we can get much more performance by rewriting Pandas GroupBy - Apply in NumPy. I also mentioned that I could not get Numba working with this code to see if it helps. In this blog, I want to talk about how I got Numba working and what the results were, but first, what’s Numba?
If you use PySpark a lot you would know that the DataFrame API is great. However there are times when it is not sufficient because it does not cover every single piece of functionality we may want. This is where the Pandas UDF functionality comes in. The nice thing about the Pandas UDF functionality is that it uses Arrow for data transfer between Spark and Pandas which minimizes…
There are many differen ways to do machine learning. The most popular methods use neural networks. We are usually given some data and we train the neural network to fit this data. In this post I will review an interesting way to look at this problem that converts the training problem to an ordinary differential equation. We will look at a property of this ODE that indicates a possible way to…