Shahinism
Recent content on Shahinism
Latest posts
Legacy Migration Starts with Understanding, not Inventory
The Default Playbook Legacy migration has an almost universal playbook: Step 1 - Asset Discovery: Make an export of all assets in the environment to see what is there to migrate. The output is usually a massive spreadsheet or dashboard. The leadership sees the number and it becomes the anchor: We have X thousand resources to migrate. Step 2 - Categorize: Tag the assets based on their domain, owner…
Career Growth: Who is going to do it?
I found this essay by Nicola Ballotta quite interesting. A retrospective of his 25 years of professional life, and how he could improve it, if he was set to start it today. There are many similar articles out there, though the main appealing aspect of this one to me, is that I can relate to it effortlessly. Apart from the main goal of the article, I started to think about career growth in general.…
Useful output in CLI applications
I favored Nushell . I was using it for a few months, until I decided to switch back to good old ZSH. Why I did that is a separate story. However, I couldn’t get the idea of having quarriable outputs from CLI applications, off my mind. Look at this: This is natural. However, it’s let’s say… difficult to have it in any other shell, with all the CLI applications out there. Taking Nushell’s approach,…
System Integration Test for Data Products
Introduction One of the challenges of having a distributed solution for data platforms is testing the system end to end. Of course, each piece of the system can be tested in isolation, but it’s difficult to conclude the end to end integrity of the system without a testing solution, which covers the system as a whole. In a recent project at DataChef we found ourselves occupied with an…
Custom Dashboard for Great Expectation
Introduction Nowadays, Great Expectations is a very viable option for most of the organizations to introduce data quality solution for their data platform. Simplicity, being easily customizable and relying on the toolbox that is mostly known to modern data engineers, are the common factors which help with lowering the adaption barrier. However, there is one part of Great Expectation, which I hoped…
Software Engineering for Data Engineers: Introduction
Transitioning from a web development background to data engineering, I’ve encountered a significant cultural shift. One of the most striking differences is the apparent lack of established software engineering practices within many data engineering teams at various companies. This discrepancy, while noticeable, is not without reason. A majority of online resources offering guidance on…
Mysterious Spark Checkpoints
It all started from a change in the checkpoint path of our Spark applications. We use Spark Structured streaming and AWS S3 buckets to maintain checkpoints. Let’s say we were using s3://bucket/spark/topic/ as the checkpoint path, and we changed it to s3://bucket/spark/topic/v1 . Given that we weren’t touching anything from the content of this directory, and we didn’t want to double ingest the data…
Composing Functions for PySpark's Structured Streaming
In PySpark’s structured streaming, striving for modular, reusable code while ensuring adaptability for each unique use case is a common challenge. I recently had the task of leveraging the broad capabilities of the writeStream method without losing the granular control that the foreachBatch function provides. For those unfamiliar, foreachBatch Spark’s streaming API lets you perform…
Iceberg on Glue: The table name confusion!
Lately, our team has embarked on a journey to enhance our data storage infrastructure. We invested in Iceberg as our storage layer for data product implementation with transactional capabilities. The intent was clear: to run our applications on AWS Glue, and naturally, we wanted to leverage Glue Catalog for table registration and representation, especially in an unmanaged mode. In this endeavor,…
Introducing devenv Templates: Simplifying Development Environments with Nix
Over the past year, I’ve dived deep into the world of NixOS , discovering its robustness, flexibility, and power. I’ve crafted various configurations and setups tailored to my development needs during this journey. Today, I’m excited to introduce the devenv templates, a culmination of my learnings and experiences with NixOS. %[https://github.com/shahinism/devenv-templates/] Why…
The Fire of Purpose: From Personal Growth to Team Success
Deep within each of us, there’s a burning desire—a fundamental drive that shapes our actions and decisions. For me, it boils down to one thing: purpose . How can one truly live without goals? To me, that’s a path leading nowhere. Without a clear objective, our energies scatter, rendering us incapable of leveraging our true potential. And if we don’t tap into that potential, how…
TIL: Kafka CLI and group offset
Today, I discovered that when utilizing kafka-console-consumer and including the --group flag without specifying --from-beginning , the offset is set to the latest message. Subsequently, adding --from-beginning to later commands will be disregarded. Meaning it’ll only consume new events. This information is beneficial during debugging and maintenance operations. The offset must be reset to…
AI is here; what's your plan?
Over a decade ago, big data became a buzzword, and everyone envisioned a future where data-driven decision-making would be the norm. However, fast forward to today, and we still find ourselves selling Excel sheets as pricing models to companies or repackaging Redshift with a seven-figure price tag to address all data-related issues. This was the case with a predictable technological concept that…
Fixing the Git GPG sign issue
When submitting a commit command, I was facing: error: gpg failed to sign the data fatal: failed to write commit object After some searching, I managed to enable Git’s debugging log as: GIT_TRACE = 1 git commit And the output changed to: 08:55:55.059810 git.c:460 trace: built-in: git commit -m 'add readme file' 08:55:55.060693 run-command.c:655 trace: run_command: gpg --status-fd=2 -bsau…
Run Spark Applications on AWS Fargate
Overview In this post, we are going to review how to run a Spark application, as a single node Fargate task. If you are familiar with Spark and its potential workload, you might wonder: “why would anyone need to run Spark applications over AWS Fargate, instead of a proper cluster”. Well, to be honest, that’s a valid question, however, there is a profusion of deployment cases where the application…
How to Handle Null in Spark
Overview In this article, we will talk about the second-ugliest exception in the history of programming and attempt to handle it in our Spark apps. If you’ve ever worked with Spark in its native language, you’ve probably faced this bizarre, hard to debug exception, famously called as NullPointerException ! Also, you might have done your homework hoping to find a one fit all solution. But reading…
AWS VPC With Public and Private Subnets
Overview Virtual Private Clouds (or VPC) as you all probably know, is one of those services, which would be a lifesaver when you know how to use them. The isolation and service integrations provided by VPCs, suppose to reduce the common cloud management hassles and help you when you plan to scale your application. However, covering all that heavy lifting, it’s not that easy to abstract everything…
AWS Vault with Yubikey
Preface It’s been a while I’m using aws-vault to manage my AWS credentials for accessing the different client accounts. It has been quite comfortable, regardless of the setup the clients choose for their users. However, there are 3 places which I’m trying to improve my own development experience: Requirement of 2FA authentication SSO Login through a browser (to get access over…
Plumbum for shell scripting in Python
I never was a fan of scripting, and unless it was a real edge case, I was always ending up refactoring my needs based on existing tools instead of scripting. Why? Simply said, because as soon as I finish writing a Bash script, I quickly forget what the mess was about, and if I needed a refactor sometime later, I always end up with too many WTF/m. Yes, I realize how cool it is, but in 2022 I prefer…
Simple shell pop-up in Emacs
TLDR: To practice my Emacs-Lisp-fu and revise my terminal workflow, I implemented a Guake-like pop-up terminal inside Emacs. Emacs Rocks! I’m trying to integrate my shell activity with my Emacs workflow. For a long while, I was using Guake-like terminals as my main scratchpad while operating my machine. After switching to i3wm , however, I gave up on the idea of being able to open a terminal in…
Using data to improve software engineering
Whenever working on a new source code, I find it quite cumbersome to grow to enough understanding around it to be productive with it both in terms of security and speed of delivery. This problem can especially get challenging given different factors like: Lack of documentation around the source code or architecture. Unknown external dependencies relying on the current project to function as a…
Polyglot Dojo #4: Permutation
Challenge In this post I’m going to talk about the permutation challenge . What we require to accomplish here is to compare two input strings and check if they have the same length with the same set of characters case sensitively. Our main constraints are defined as: Can we assume the string is ASCII? Yes Note : Unicode strings could require special handling depending on your language Is…
Polyglot Dojo #3: Hashmap
Challenge The challenge I tend to review in this post is called hash map . To be honest, I couldn’t understand the requirement of the challenge from the description and had almost no idea what a hash map is. Even the test suite didn’t help much in visualizing the data structure I needed. So I first need to learn what a hash map is. The best resource I could find on the internet which…
Polyglot Dojo #2: FizzBuzz [2*]
Updates [2019-12-23 Mon] : Added Rust Solution. [2019-12-21 Sat] : Added Clojure Solution. Challenge This challenge was disappointingly dull, however, I saw some exciting things through the way. We are going to solve FizzBuzz , with the following constraints: What is fizz buzz? Return the string representation of numbers from 1 to n Multiples of 3 -> ‘Fizz’ Multiples of 5 ->…
Polyglot Dojo #1: Compress String
Challenge The first challenge we pick to solve is Compress Challange . The description of the challenge says: Compress a string such that ‘AAABCCDDDD’ becomes ‘A3BC2D4’. Only compress the string if it saves space To solve this challenge, we must satisfy the following constraints: We can assume the string is ASCII The compression should be case sensitive We can use…
Polyglot Dojo #0: Introduction
Overview In this series of blog posts, I plan to solve programming challenges I find around the internet, in books or throughout my daily job in different languages. Though, before we start, I want to set some ground rules to keep myself motivated and also share the point of this series. Language Choice I’m a Python developer; not a top, highly skilled developer of course, but I’m…
How to not use Google search
How to not use Google search Prelude It’s almost about 6 months since the last time I used Google search as my main search engine. This idea of using an alternative search engine was bugging me for a long time. Almost seven years to be exact. I remember the first time I heard about some reasons to reconsider my main search engine was from this Ted video: However, I couldn’t leave Google for a long…
Run Python test suites without leaving Emacs
Run Python test suites without leaving Emacs Recently, I’m trying to overcome my laziness and incorporate more automated testing into my software development procedure. It’s not that I wasn’t developing test suites before, but the problem was that the overall process of defining good tests, running and maintaining them was so complicated to me which most of the time was pushing…
My Journey to Privacy
My Journey to Privacy A decade ago, it was easier for ordinary people to start using the internet without having any concerns about being spied on. Accessibility of the internet hadn’t been grown like what we have today, so it wasn’t such a tasty platform for most of the people of the world. However, that was a long long time ago and the accessibility and usability of the internet platform have…
First Post
First Post A couple of years ago I was considering myself a blogger, with a decent archive of the blog posts (all in Persian ) on topics that were interesting to me at the time. However, in the last few years, I couldn’t manage my time good enough to keep the habit of writing. It’s a while since I have become to conclusion that the activity of writing, is crucial to my self-improvement process.…
For God’s sake, secure your Mongo/Redis/etc!
I was watching a lecture on computer security recently and came up to this website called shodan.io . Well apparently, I’m too late to the party, it is out there since 2009. Before finding Shodan, I always assumed that gathering data about system security, needs some basic knowledge about what you are looking for and at least access to some scanners like Nmap , but now, it’s just a click away! So…