My First Engineering Job and the Magic of ETL

Winter was coming. It was October 2019, and the air on campus was thick with the anxiety of placement season. “Day 0” and “Day 1” tech companies were flooding the college, and everyone was in full swing.

I’ll be honest, I was nervous. I was never the fastest at tricky aptitude tests or competitive programming puzzles. While many of my peers spent their days grinding on LeetCode and Codeforces, I had spent my time building and reading. I loved getting my hands dirty with real-world projects and digesting foundational software engineering books. I knew how to build, but I wasn’t sure if that would translate in a traditional 45-minute whiteboard gauntlet.

After jumping through multiple screening hoops, I landed an interview with a company that needed no introduction Sears.

They were the absolute OG of the American retail market. While the dot-com boom and the rise of modern e-commerce had brought them some rough patches, none of that mattered to me. This was a massive enterprise with decades of history. I just wanted to contribute, write code that impacted a real business, and learn from the veteran developers on their team.

The interview was scheduled for 9:00 AM. For 45 minutes, I didn’t get asked abstract brain teasers. Instead, the interviewer dug into real engineering. I wrote complex SQL queries (nested queries, various JOINs), architected a High-Level Design (HLD) on the whiteboard, and explained the architecture of the projects I had built. My practical experience paid off. There were a few small array and string manipulation questions, but they were grounded in reality.

I spent the rest of the day pacing. Finally, at 7:00 PM, the call came and I was selected.

I was interviewed on October 7th, and my call letter told me to report to the office on October 21st, 2019. I was on cloud nine.

When I walked into the Sears India office, I realised I was the junior guy in a room full of titans. The data engineering team was packed with industry veterans—brilliant minds with 20+ years of experience from product development giants like Adobe, IBM, and HCL. I was thrilled.

The first two weeks were a whirlwind of Knowledge Transfer (KT) sessions, reading endless documentation, and getting access to Google Cloud Platform (GCP). Every week, we had assessments to ensure we were absorbing the architecture.

Once the training wheels came off, I was introduced to the monster problem the team was tackling.

Sears owned a staggering amount of legacy infrastructure. We are talking about petabytes of consumer and transactional data sitting on aging IBM Power Systems, Hadoop clusters, and Teradata.

The business goal was modernisation: this massive ocean of data needed to be migrated to Google Cloud specifically into BigQuery and Spanner—so the business could run modern data analytics.

Every week, the onsite team in the US would provide us with the DDL (Data Definition Language) and DML (Data Manipulation Language) for the incoming downstream data. Our job in India was to write the pipelines to catch that data, transform it, and load it into the new cloud tables.

Our Tech Stack: Python, Shell Scripting, PySpark, Apache Airflow, Control-M, and massive amounts of BigQuery/Spanner SQL scripting.

If you are new to data engineering, you might hear the term ETL thrown around a lot. It stands for Extract, Transform, and Load. It is the backbone of almost every big tech company that relies on data.

Imagine you are running a retail giant that has been in business for decades. You process millions of transactions a day across thousands of stores. To survive, you need an analytics engine that tells you: What is our warehouse stock looking like? Which city is growing the fastest? Which winter jacket is failing in the market?

To answer these questions, you need all your data in one place, perfectly formatted.

• Extract: We pull the raw data out of the old legacy systems (like the IBM Power Systems).

• Transform: The raw data is often messy. We clean it, fix data type mismatches, remove redundancies, and shape it to match the new system’s rules.

• Load: We push this clean, organised data into a modern database where it can be analyzed.

Here is a visual breakdown of how this pipeline works

The reason Sears needed to migrate the data was fundamental to how databases work. The legacy IBM Power Systems were incredibly secure and fantastic at writing data as transactions happened. But they were terrible at reading massive chunks of data for analytics.

This introduces the two main categories of database processing:

Think of OLTP as the cashier at a supermarket. It needs to be incredibly fast at handling thousands of small, short requests at the same time (inserting a new sale, updating inventory, deleting a canceled order).

• Focus: Fast, real-time transaction processing.

• Structure: Highly normalised (organised to avoid duplicate data).

• Users: Thousands of point-of-sale systems or end-users.

• Examples: Banking apps, Amazon checkout, airline ticket bookings.

Think of OLAP as the CEO sitting in a boardroom analyzing a 10-year trend report. It doesn’t handle real-time sales; it looks at historical data from different perspectives.

• Focus: Analysing massive volumes of data with complex queries.

• Structure: Denormalised (data is grouped together to make reading much faster, even if it takes up more space).

• Users: A small number of data scientists or business analysts.

• Examples: Tableau, Data Warehouses, customer trend analysis.

When loading data into Google Cloud, we didn’t just throw it all into one bucket. We had to route it based on what the business needed to do with it. This is where the difference between BigQuery and Spanner becomes crucial.

The world of software and data engineering is endlessly fascinating. If there is one thing I learned during those early days at Sears, it is that there is no “one size fits all” silver bullet in technology.

You can never understand or solve all problems with a single tool. Every database, language, and framework has its own specific magic. Our job as engineers isn’t to blindly use the newest tech; our job is to understand the problem deeply enough to figure out exactly which tool to use, and exactly when to use it.

And if the right tool doesn’t exist? Well, then we get to build a new one.