The Rule I Learned to Love When Ariel and I started Ariga, he introduced a rule that I honestly thought was going to drive me insane: no pull requests over 100 lines of code without a very good reason. I came from teams that practiced long-lived feature branches. You know the drill: you’re working on a feature, you create a branch, and it stays open for the duration of development. Medium to large…
Reflections on a weekend project with an AI co-programmer Almost four years into Ariga , I don’t get to write code as much as I’d like, but occasionally I encounter some interesting problem that seems small enough to tackle in a short amount of time and I dive in. This was the case with an issue that came up from some of our lovely customers for Atlas . One of the great things about Atlas is it’s…
In the past year and a half, I’ve been working on Atlas , a database schema management tool that we’re developing at Ariga . As part of this effort, I worked on implementing the infrastructure for the Atlas DDL , a data definition language that is the basis for Atlas’s declarative style workflow for managing database schemas. The Atlas language is based on HCL , a toolkit for creating…
One of the reasons that I enjoy programming in Go so much is that its design is so elegant. Let me try to explain what I mean by that. Many years ago, I was studying Game Design with a group of friends. During that time, I came across a book that I find myself quoting from in many situations to this day, The Art of Game Design , by Jesse Schell. In one of my favorite parts of this book, the author…
Protocol Buffers (Protobufs) are a popular open-source interface definition language (IDL) that was originally developed at Google. Aside from its benefits as a serialization format - messages are fairly compact and fast to serialize and parse - Protobufs really shine due to their code generation facilities. The basic idea of any IDL is that data structures (called “Messages” in Protobuf) are…
Debugging is like being the detective in a crime movie where you are also the murderer. — Filipe Fortes (@fortes) November 10, 2013 Debugging is the process of diagnosing why a software system fails to behave correctly. As with medical diagnostics, oftentimes finding the cause for illness is the hard part, while offering the treatment requires much less work; there can be many reasons why a…
Credits: The solution described in this post is based on Alex Eagle ’s suggestion in: https://github.com/bazelbuild/bazel-gazelle/issues/773 While migrating the Nexar backend mono-repo to use Bazel as our main build-system I encountered a particularly painful challenge of compiling Go packages with C dependencies, such as confluent-go-kafka or libgeos . For others who will hit this issue, this…
Introduction Let's stop fooling ourselves. What we call CI/CD is actually only CI. — Ant(on) Weiss (@antweiss) September 21, 2020 Containers make CI much more manageable: reproducible, isolated build environments that create portable and predictable deployment artifacts. Continuously delivering containers to production turns out to be quite a difficult problem which we call collectively…
A first dip into DIY static code analysis One of the special things about software engineering as a profession is the possibility of ars-poetic work: part of our work is building tools that target our own work; perhaps a few surgeons around the globe can design and meld their own scalpel, but for software engineers building our own tooling is a day to day reality. Recently, I’ve been working on…
The DRY principle is probably the first software design concept you learn when you start coding. It sounds very serious and convincing, after all, it has an acronym! Furthermore, the idea of not repeating ourselves deeply resonates with the reason many of us enjoy programming computers: to liberate us from mind-numbing repetitious work. It is a concept that is very easy to grasp and explain (I…
Apache Spark provides a very convenient abstraction layer for building distributed applications that process massive amounts of data. It is one of the most useful and robust tools in the tool belt of any data engineering team, facilitating both the exploratory analysis of huge datasets, scheduled production batch processing applications as well as streaming and machine learning applications.…
Is there a name for the pattern where you build your own <something> that isn't your core competency or interest because the 3rd party alternatives are lacking features you need... but then the 3rd parties catch up and you are stuck with your hacks? — Gwen (Chen) Shapira (@gwenshap) September 3, 2019 Here’s a common scenario, if you’re a cutting edge shop: you early-adopt new infrastructure…
Introducing TechNex, Nexar’s internal R&D Meetup One of the things I like doing most as part of my job at Nexar is hosting TechNex, our internal, bi-weekly R&D meetup. In this post, I want to tell a bit about it, why I personally put time and effort into running it and why you should consider starting an internal engineering meetup in your company. What is TechNex One hour, every two weeks. All…
It is today easier than ever to generate a lot observability data (metrics, logs and traces) from your applications and forward it to a central location where you can efficiently query it. However, systems can still get pretty hard to reason about. Having an abundance of application specific data generated from your system does not guarantee that an on-call engineer or operator will know what…
As your software-organization grows in: headcount, service count, programming language count it becomes increasingly harder to coordinate work. Let me try to illustrate this. Let’s say you are working on a new workflow that requires you to send push notifications to some clients of your mobile app. Another team is maintaining the OutboundMessagingService , a microservice written in Java that deals…
Reading and writing TLV-encoded messages with Go Recently, I’ve worked on implementing a high-throughput networking component at work. In an effort to conserve CPU time and reduce payload sizes, this service is using a binary encoding as the wire format. An attribute of the protocol we were implementing was that the messages could include optional fields of variable length. This, I learned, is not…
Previously on “Serverless Applications” In this series’ previous parts ( part 1 , part 2 ), we discussed how we test and deploy Lambda functions as a part of the CI/CD pipeline for our serverless configuration microservice. Today, we will discuss the missing link between our code and the internet, API Gateway. As we learned, Lambda functions are a really useful way to run our code without having…
We built a serverless microservice to manage our customers’ pipeline configuration in ironSource atom Recap In the previous part of this series, we began the tale of the serverless microservice we created in ironSource atom to help us manage the configuration of our customers’ big-data pipelines. We discussed the huge benefits we get from working with AWS Lambda and API Gateway in terms of…
At ironSource’s Data Solutions dept., we work on ironSource atom , a data flow management platform to help our clients easily deal with massive amounts of data. We help companies create custom big-data pipelines into which they can stream data which will be safely collected, transformed and loaded into one of many big-data target types such as Amazon Redshift. We recently decided to build an…
You Gotta Love Frontend (YGLF for short) is an annual conference for Frontend Developers held annually in Tel-Aviv. Well, it’s the second… Source: http://yougottalovefrontend.com/#page-about You Gotta Love Frontend (YGLF for short) is an annual conference for Frontend Developers held annually in Tel-Aviv. Well, it’s the second year they’ve been having it, so it might turn out to be a Fibonacci…
Demo: https://kahootclone.firebaseapp.com GitHub repo: https://github.com/rotemtam/kahootClone A little while back I wrote a story on Medium about why local-multiplayer games should be the future of gaming. An interesting (and successful) example of such a game is “ Kahoot! ”, a multiplayer trivia game designed to help teachers add some digital interactive fun to their classes. Questions are shown…
The Zen of teaching your friend how to Code More than a good programming course, more than excellent Googling skills, to climb the mountain of Learning to Code, one needs a guide. This guide need not have official certifications, nor does he need a map. He knows the mountain well because he has climbed it much. He’s climbed it before you, and he will climb it after you. He’s been doing it for…
Not many know this, but a new genre of digital games is slowly emerging. Yes, it is happening as we speak! I’m talking about “local… Not many know this, but a new genre of digital games is slowly emerging. Yes, it is happening as we speak! I’m talking about “local-multiplayer” games. As defined by one website , they are “games that can be played by two or more people in the same physical…
My favorite characters in spy movies were always the gadget-making-mad-scientists building crazy little machines which help the spy-hero to pull off his amazing feats. Q has always been a much more appealing character than Bond to me, as was Donatello over Leonardo as my favorite Ninja-Turtle. Cheap new hardware and the quickly-growing “maker” mindset give us a small opportunity to step into the…