Postgres to Iceberg in 13 minutes: How Supermetal compares to Flink, Kafka Connect, and Spark
I wrote a post benchmarking various tools for Change Data Capture (CDC) from Postgres to Iceberg, including Supermetal, Flink, Kafka Connect, and Spark.
sap1ens.com
Recent content in Hi! 👋 I'm Yaroslav Tkachenko. on sap1ens.com
Latest posts
The Independent Engineer
I’m starting a new newsletter! For a while, I wanted to share my experience building an independent business. Specifically, building a business as a software engineer. A business where you keep building systems, solving problems, writing code - whatever you really enjoy doing. You can call it being a solopreneur or a consultant. I believe any software engineer can do it given enough focus…
Irontools Update
Last year I launched Irontools , a company focused on building extensions for Apache Flink. I’ve not made many updates since then, and today I want to share that Irontools has pivoted to primarily consulting and training. Building Irontools Link to this heading I learned a lot in the past 15 months. I spent weeks reading Apache Flink source code. But also, studying database optimizations…
Irontools
I completely forgot to post this update on my website… Posting it with a month of delay 🙂. I launched a new company called Irontools . It’s focused on making Apache Flink better, faster, cheaper, and more efficient. You can see the full announcement here .
Compiling and running perf for a specific kernel version
Recently I’ve started working on a Rust application. Shortly after the first prototype was ready, I decided to profile it to make sure I didn’t make any horrible mistakes that could affect performance. Due to the nature of the application, I couldn’t just profile it locally (but if it was an option I’d use flamegraph ). It needed to run in production (Kubernetes). So, I thought I should just…
Going Solo
This is my last week at Goldsky as a full-time employee (I’ll still be involved as an advisor and part-time consultant). I’m not looking for a job. Instead, I’ve decided to become a solopreneur. What does it mean? For a while, I’ve been writing down ideas in the data infrastructure space. Things I wish existed. Things I’d like to build. Open-source projects I’d…
Customizing Flink. Part 2: Class Shadowing
This is the second part of the “Customizing Flink” series. In the first part , I explained why you may want to customize Flink and how to approach creating and maintaining a fork. It’s not easy. It requires setting up an additional repo or two, a branching strategy, a CI/CD process, etc. But there is an easier approach that I want to demonstrate in this post. A word of caution: this technique is…
Customizing Flink. Part 1: Forking
UPDATE: see the part 2 here . Apache Flink is a powerful data processing framework, but sometimes you realize that it doesn’t always work the way you want. Maybe you want to adjust an existing feature a bit, and you don’t have time to go through the review process and wait for the next release. Maybe you have a very specific use case. Or maybe you’ve discovered a bug that hasn’t been fixed yet,…
Website Refresh
Exactly 10 years ago I launched sap1ens.com and published my first blog post: All these years I’ve used Octopress, which is a great framework until you need to migrate to a new machine (try sticking to an 8-year-old Ruby version with a bunch of old dependencies). I wanted to move to something for a while and Hugo is the obvious choice nowadays. So, after 10 years and 63 posts I transitioned…
Using Changelogs and Streams to Solve Blockchain Data Challenges
I wrote another blog post for Goldsky .
How Goldsky Democratizes Streaming Data for Web3 Developers
I wrote a post about Goldsky’s data streaming architecture with collaboration from Redpanda.
Apache Avro Decoding Story: Turning a Data Corruption Bug Into a Feature
I wrote a blog post for Goldsky .
Data Streaming Journey Newsletter
Today I’m very excited to announce my newsletter: Data Streaming Journey . I plan to write my thoughts about data streaming, stream processing, and real-time data pipelines. I want to go deep into relevant topics and write about lots of different tech. In my first post , I’ll talk about exactly-once delivery semantics and why you may choose to avoid them.
Heimdall: making operating Flink deployments a bit easier
I want to introduce Heimdall : a dashboard for operating Flink jobs and deployments. I’ve been working on it for the last several weeks, and we’ve been using it in Goldsky to manage 100+ Flink deployments. First of all, why is it needed? Doesn’t Flink come with a built-in UI? It does, and Heimdall doesn’t try to replace it. Flink UI is amazing for managing a single job. It…
Profiling Apache Flink applications with async-profiler
Profiling is one of the most reliable techniques in the arsenal of a software engineer when debugging performance-related issues. And yet, profiling large distributed systems like Flink can still be extremely challenging. JVM Tools Link to this heading JVM comes with a variety of profiling tools, Flink documentation covers some of them . However, this approach has a few downsides: these tools…
Confluent acquired Immerok: what does it mean?
On January 6th, Confluent announced its acquisition of Immerok . Immerok is a brilliant group of core Apache Flink engineers who have been building a managed Apache Flink offering. This is an important acquisition for Confluent. Confluent’s CEO, Jay Kreps, mentioned in the announcement that “Flink is the future of stream processing.” In this post, I’ll try to understand…
Flink CDC for Postgres: Lessons Learned
Change data capture is a powerful technique for consuming data from a database. Modern solutions like Debezium leverage native WAL abstractions like MySQL binlog or Postgres replication slots to get data reliably and fast. CDC Connectors for Apache Flink is an open-source project that provides tools like Debezium in native Flink source APIs, so it can be easily used in any Flink project. This blog…
Why I’m Joining Goldsky, a Web3 Company
Today I join Goldsky as a Principal Software Engineer. Goldsky’s mission is to make it easy to index, process and use blockchain data. This set of tools could become a foundational building block for many Web3 companies. We leverage streaming SQL to make it possible. Web3? Link to this heading I’ve never been a crypto, NFT or web3 enthusiast. Quite the opposite, I was (and I am) very skeptical…
7 Tips For Optimizing Apache Flink Applications
I wrote a blog post for Shopify’s Data Science & Engineering blog: 7 Tips For Optimizing Apache Flink Applications .
Streaming Systems and Global State
Stateful stream processing is not a new concept, but some approaches and best practices are still not straightforward and continuously changing. The state itself can be represented in a variety of different ways. I’ve recently spent quite a bit of time learning and building stream processing pipelines that use a particular type of state, and I’d love to share more thoughts on this…
Kafka Elasticsearch Sink Connector and the Power of Single Message Transformations
I’ve been using Kafka Connect for a few years now, but I’ve never paid much attention to Single Message Transformations (SMTs), until recently. SMTs are simple transforms that are applied to individual messages before they’re delivered to a sink connector. They can drop a field, rename a field, add a timestamp, etc. I always thought that any kind of transformation should be done…
Flow State and Programming
A few months ago, I finished reading Flow: The Psychology of Optimal Experience by Mihaly Csikszentmihalyi. Wow, what an fantastic book! Have you ever did something that made you focus on the subject so much that you forgot about your surroundings, forgot about time and just enjoyed doing what you were doing? This is known as Flow . Flow is a state of optimal experience; a state of joy, creativity…
Kafka Streams Application Patterns
Kafka Streams is an advanced stream-processing library with high-level, intuitive DSL and a great set of features including exactly-once delivery, reliable stateful event-time processing, and more. Naturally, after completing a few basic tutorials and examples, a question arises: how should I structure an application for a real, production use-case? The answer could be very different depending on…
Deploying Kafka Connect Connectors
Kafka Connect is a modern open-source Enterprise Integration Framework that leverages Apache Kafka ecosystem. With Connect you get access to dozens of connectors that can send data between Kafka and various data stores (like S3, JDBC, Elasticsearch, etc.). Kafka Connect provides REST API to manage connectors. The REST API supports various operations like describing, adding, modifying, pausing,…
DStreams vs. DataFrames: Two Flavors of Spark Streaming
I wrote a guest blog post for Qubole .
2018 In Review
Once again I’m writing a “Year in Review” post, mostly focused on professional life & tech stuff. Check 2017 here . Demonware (Activision) -> Activision Link to this heading I really enjoyed my time at Demonware , but at the beginning of the year I had an opportunity to move to another team at Activision, Data Services. My previous team at Demonware, Data Pipeline, is responsible…
"Kafka: The Definitive Guide": Notes
For the last two years I’ve been working with Apache Kafka a lot . Everything including building infrastructure (and running clusters on bare metal, in VMs and containers), improving monitoring and alerting, developing consumers, producers and stream processors, tuning, maintenance, etc., so I consider myself a very proficient user. Still, all these years I didn’t have a chance to read…
Next-generation data pipelines
It’s Q4 of 2018 and it’s really interesting to observe the change in Big Data Landscape , especially around open-source frameworks and tools. Yes, it’s still very fragmented, but the actual solutions and architectures start to slowly converge. Right now I’m in the beginning of a huge platform redesign at work. We always talk about various frameworks and libraries (which is…
API Flexibility ∝ 1 / Data Definition Strictness
An interesting observation based on the recent conversations at work: the more strict the data format used in the API definition, the harder it is to change the API behaviour later. And vice versa, it’s easier to change the APIs that use flexible data formats. Why do so many startups use JSON? Sure, it’s obvious with web applications, but how do you explain sending it to Kafka, using…
The Level Below
Over the years I realized a very simple, but also a fundamental principle about being a better software engineer - understanding what’s happening one level below . With level I mean any level of abstraction you operate, for example HTTP API for a Front-end engineer, JVM and its internals for an enterprise Java developer, etc. It sounds like an obvious suggestion, but it’s actually very…
QCon London 2018
Last week I had a chance to attend and speak at my first QCon conference: QCon London 2018. QCon has been an example of an extraordinary tech conference for me - great organization, amazing lineup, great tracks covering bleeding-edge tech, as well as various best practices around distributed systems. Also, they managed to get a decent vegetarian food right 😉 Keep reading if you’re curious…
Message enrichment with Kafka Streams
I’ve been working with Kafka Streams for a few months and I love it! Here’s the great intro if you’re not familiar with the framework. In the sections below I assume that you understand the basic concepts like KStream , KTable , joins and windowing. Message enrichment is a standard stream processing task and I want to show different options Kafka Streams provides to implement it…
2017 In Review
Another “Year in Review” post, again mostly focused on professional life & tech stuff (since it’s a tech blog 😉). Check 2016 here . Mobify -> Demonware (Activision) Link to this heading So, I’ve left Mobify and joined Demonware (which is a subsidiary of Activision or Activision Blizzard, Inc. to be more precise). Mobify is a great company that builds great products. I…
A letter to you, %software_conference_organizer%
I’ve been a frequent software conference attendee and a speaker for the last few years. I’ve seen great events with amazing speakers, facilities and food. And I’ve seen very poor events with a lot of issues like zero diversity and really weak speakers. And I’ve been recently thinking about some recommendations for the future events and came up with a list below. I emphasize…
6 months at Demonware
You can probably decide I’m busy based on the absence of any new posts since July :) And indeed, I was pretty busy for a while. I started to work at Demonware precisely 6 months ago and it’s being an amazing ride so far. My team is responsible for Demonware/Activision data pipeline, metrics and logs. I’m mostly focused on the data pipeline, but also slowly learning the metrics…
Bash scripting best practices
I was always afraid of writing shell scripts. Bash seemed to be a programming language that doesn’t allow a slightest mistake… Extra space here and there and everything blows up. Like with every skill, persistence and repetition help. I’ve started writing more and more bash scripts a few years ago. But it’s important to remember one simple rule - when things become really…
How to build CLI in Node.js
CLI (Command-line interface) is a very common way to run a program in a terminal. As a software engineer you use different CLIs every day - git, Docker, npm, etc. Today I want to share my experience building CLIs using Node.js and a few helpful packages. Goal Link to this heading I’m going to only focus on npm in this article, simply because I don’t have enough experience with other…
Craft Conf 2017
Last week I attended Craft Conference in Budapest, Hungary. Here’s the description of the event: CRAFT is about software craftsmanship, presenting which tools, methods, practices should be part of the toolbox of a modern developer and company, and serving as compass on new technologies, trends. You can learn from the best speakers and practitioners in our community. It was definitely one of…
About being a polyglot software engineer
Recently I’ve watched a technical talk about some tool, let’s call it X. This tool is written in Java. Presenter was really excited and energetic and he obviously knew his stuff. And then, there were two questions after the presentation: Are there any similar solutions in languages other than Java? What do you think about the concept Y? The concept Y is very similar to what presenter…
Should I use HTTP2 everywhere by default?
Yes, you definitely should.
Microxchg 2017
Last week I had a pleasure visiting Microxchg conference in Berlin. I was really excited, because I also conducted my first workshop there - Building Microservices with Scala and Akka . I think the event was truly successful and I’d like to share my thoughts and notes below for the most interesting talks I attended. The conference is dedicated to Microservices, as you can guess from its…
2016 In Review
Here’s my first attempt to summarize most important things happened in my professional life (mostly programming) this year, for future self. May be it’ll be interesting for someone else too. Bench -> Mobify Link to this heading Probably one of the hardest decisions I made was to leave Bench Accounting. I wrote all details here , so I’m not going to repeat myself. I do want to…
Intro To RAML 1.0
Pretty much every web developer has built HTTP APIs. How does it usually happen? If you have good practices established in your team you probably start with a whiteboarding session. Good one-hour exercise with your colleagues produces something that we call “API spec”. It might look like this: Great job everyone! Although, we forgot a few “minor” things: Error messages and…
Bootstrapping Akka Cluster with Consul
As you probably know, I’m a big fan of Akka . Akka Cluster is a great low-level mechanism for building reliable distributed systems. It’s shipped with powerful abstractions like Sharding and Singleton . If you want to start using it you should solve cluster bootstrapping first. Almost every tutorial on the internet (including the official one ) tells you to use seed nodes. It looks something like…
Last week at Bench
This week is my last week at Bench . I still clearly remember how I was interviewed by Ian and then joined the project, more than 5 years ago. The whole company was just 6 (!) people, including Pavel (our CTO), myself and one more software engineer. Our little development team was all in Russia, working remotely. One of my first tasks was building this UI Somehow we managed to grow into 230+…
Vancouver AWS User Group Talk
This week I gave a talk at Vancouver Amazon Web Services User Group meetup about AWS Elastic Beanstalk: Introduction to the service Some implementation details Tooling we’ve built at Bench And lots of comments about my experience of using it My slides :
Software Engineering Blogs I Read
RSS is not dead. At least not for me :) I use NewsBlur to track and read 50+ various feeds - tech, travel, personal blogs. Today I want to share a list of software engineering blogs that I find very helpful and keep in my list for a long time. http://techblog.netflix.com - Netflix has one of the best blogs. They always post very interesting articles explaining their architectural decisions in…
Syrup - Scrum CLI tool
At Bench we’re convinced that GitHub should be a central place for everything project-related: codebase, documentation, tasks. That’s why we don’t use any specific Project Management software, but instead we try to use GitHub. Obviously GitHub has a very limited set of features: issues, labels, milestones. But, surprisingly, it’s good enough foundation that can be used by other services, for…
Postgres JSONB indexing is tricky
I’m not a Postgres expert, but I’ve been using it for about 6 months as an Event Store database. At Bench we’ve built our own eventing system on top of ActiveMQ, Camel and Akka and we use Postgres to persist every single Domain Event. Our event schema is very flexible and currently represented in JSON. We chose Postgres for persistence, because of the great JSON support. As you probably know,…
Polyglotconf 2016 Talk
Today I participated in Polyglotconf 3rd year in a row. This time I was prepared though, I wanted to talk about microservices and different advantages and disadvantages you can have if you use messaging as your main internal communication protocol. Also, it makes perfect sense for me to use Actors to build the actual microservices and I tried to explain why. It seems like this topic is quite…