Hey there,
I need to tell you about the biggest mistake I almost made in my career.
It was 2021. I was scrolling through LinkedIn (procrastinating, as usual) when I saw it:
“Data Engineer – Series B Startup – $180K base”
My heart raced.
This was it. The role I’d been working toward.
Then I opened the job description.
The list felt intimidating:
5+ years of data engineering experience
(I had 3 years in automation + 2 years as a data analyst)Spark expert
(I’d used it in a few projects, but expert?)Data engineering background required
(I came from automation engineering)CS degree preferred
(I had one, plus a Master’s in Engineering Management and still doubted myself)
I stared at the screen for 20 minutes.
Then I closed the tab.
“I’m not qualified.”
For three weeks, that job posting haunted me.
Every morning, I’d see it in my saved jobs. Every evening, I’d tell myself:
“I need more Spark experience first”
“They want someone from a pure DE background”
“I should wait until I’m more ready”
“What if they figure out I’m not qualified?”
Here’s what I didn’t know at the time:
I was already qualified. I just couldn’t see it.
On a random Tuesday, I was venting to a former colleague about my job search frustration.
Me: “I keep finding roles that want 5+ years of pure data engineering experience. I have automation and analyst background, but that doesn’t count, right?”
Him: “Wait, you built those automation pipelines that processed millions of events per day, right?”
Me: “Yeah, but that’s not data engineering—”
Him: “That IS data engineering. You built data pipelines. You designed data models. You worked with stakeholders on data requirements. What do you think data engineering is?”
That conversation changed my perspective completely.
I went back and looked at my actual experience with fresh eyes:
Built pipelines processing 2M+ events daily
Designed monitoring and alerting systems
Debugged production issues under pressure
Understood reliability and system design
Mastered SQL (wrote hundreds of complex queries)
Understood stakeholder needs deeply
Built data models for business analytics
Created dashboards and reports
Master’s in Computer Science
Master’s in Engineering Management
Strong fundamentals in algorithms and systems
I wasn’t missing data engineering experience. I was just calling it different names.
Instead of apologizing for my “non-traditional” path, I rewrote my story:
“I’m trying to transition into data engineering. I don’t have direct DE experience, but I’m a fast learner.”
“I’ve built data pipelines processing millions of events in my automation work. I bring SQL expertise from my analyst background and system design thinking from my engineering management studies. Here are three projects that demonstrate my DE skills...”
The skills were always there. I just needed to own them.
I finally clicked “Apply” on that job posting (it had been reposted after 3 weeks - they were still looking).
The interview came two days later.
Interviewer: “Tell me about your data engineering experience.”
Old me would have said: “Well, I don’t have traditional DE experience, but...”
What I actually said: “I’ve built automation pipelines that process 2 million events daily, designed data models for business analytics, and worked extensively with SQL for data transformation. Let me walk you through my most complex project...”
I got the offer two weeks later.
$185K base + equity + remote flexibility.
More importantly? I loved the work. My “non-traditional” background turned out to be an advantage. I brought perspectives the team needed.
If you’re reading this and thinking “but my background is different too,” that’s exactly my point.
Your background is probably MORE relevant than you think.
Coming from Software Engineering? You understand production systems, testing, and code quality - things many DEs struggle with.
Coming from Data Analysis? You understand stakeholder needs and data modeling - the “why” behind the pipelines.
Coming from DevOps/SRE? You understand reliability, monitoring, and infrastructure - critical for production data systems.
Coming from QA/Testing? You understand data quality, edge cases, and validation - increasingly important in DE.
“Non-traditional” is your advantage, not your weakness.
Here’s what companies ACTUALLY need (vs. what they write in job descriptions):
5+ years data engineering experience
Expert in Spark, Airflow, dbt
CS degree required
Big data experience
Can you solve data problems?
Can you learn new tools quickly?
Can you communicate with stakeholders?
Can you ship working pipelines?
If you can do the work, the specific background doesn’t matter as much as you think.
Your Experience → DE Skill → How to Frame It
Built automation scripts → Pipeline development → “Built automated data pipelines processing X events/day”
SQL for reporting → Data transformation → “Designed and optimized SQL queries for data transformation”
API integrations → Data ingestion → “Integrated data from 10+ external APIs”
Monitoring systems → Data observability → “Implemented monitoring and alerting for data quality”
Reframe, don’t apologize.
Pick projects that fill your biggest gaps:
If you lack pipeline experience: Build an end-to-end ETL pipeline with Airflow + dbt
If you lack big data experience: Process a large public dataset with Spark
If you lack cloud experience: Deploy a pipeline on AWS/GCP free tier
Time investment: 2-4 weekends per project
ROI: “Here’s a project I built” beats “I’m trying to learn” every time
My rule: If you have 60% of the requirements, apply.
Why?
Job descriptions are wish lists, not requirements
They’re often written by recruiters who don’t know the role
Teams are frequently flexible on specific requirements
Worst case? They say no. You lose nothing.
Let THEM decide if you’re qualified. Stop deciding for them.
If you’re in a similar position, here’s what to do this week:
Monday: Audit Your Skills (30 minutes) List everything you’ve built or done that involves data, even tangentially.
Tuesday: Rewrite Your Resume (1 hour) Reframe your experience using data engineering language.
Wednesday: Build Your First Project (Start it) Pick one weekend project that addresses your biggest gap.
Thursday-Friday: Apply to 5 Jobs (1 hour) Apply to roles where you have 60%+ of requirements.
Weekend: Ship Your Project Get something working and on GitHub.
Next Monday: Apply to 5 More Jobs Iterate weekly.
Don’t wait until you’re “ready.” You’re already more ready than you think.
Three months after I got that job, they hired someone else to the team.
Want to know their background?
Marketing analytics.
She had even less “traditional” DE experience than me. But she was brilliant, motivated, and brought a perspective the team needed.
That’s when I realized: The “perfect” candidate doesn’t exist. Teams need diverse backgrounds.
I shared my story on LinkedIn yesterday, and within an hour, I got dozens of messages:
“This is exactly my situation. I have X experience but job wants Y.”
So let me ask you:
What’s actually stopping you from applying?
Is it missing skills? (You probably have transferable ones)
Is it the job description? (It’s a wish list, not a requirement)
Is it imposter syndrome? (Everyone has it, even with 2 Master’s degrees)
The only thing that will definitely stop you from getting a DE job is not applying.
Here’s what I want you to do after reading this:
Find 3 job postings you’ve been avoiding because you feel “not qualified”
Make a list of why you think you’re not qualified
Reframe each objection as a transferable skill or learnable gap
Apply to at least 1 of them this week
If you’re serious about making this transition, I’ve created some resources to help:
What this resource includes
A Data Engineering Career Readiness Assessment to help you honestly evaluate where you stand today.
A curated list of 40 hands-on data engineering projects, organized by difficulty, so you know exactly what to build next.
Clear guidance on what each project demonstrates, how long it takes, and how to talk about it in interviews.
Practical frameworks to translate non-traditional experience into data engineering language recruiters understand.
An interview-focused approach so you’re not just building projects, but learning how to explain decisions, tradeoffs, and impact.
The person who got that job I almost didn’t apply for?
Me.
But it almost wasn’t.
I almost let imposter syndrome make that decision for me.
Don’t let your doubt decide your future.
That voice saying “not yet” is a liar.
You’re ready. Or close enough.
Apply anyway.
See you next week,
Avantikka Penumarty
P.S. What job are you going to apply for this week? Hit reply and tell me. I want to know.
P.P.S. If this resonated with you, forward it to someone else who needs to hear it. We all need permission to believe in ourselves sometimes.
Follow my journey:
LinkedIn: @Avantikka_Penumarty
Twitter: @avantikka_penumarty
Instagram: @avantikka.penumarty
Not subscribed yet? Subscribe here for weekly data engineering career advice.
Want to chat? Reply to this email. I read every response.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.