Airflow Best Practices — What to Avoid in Projects & Interviews

Most people say,

“Yeah, I’ve used Airflow.”

But interviewers can tell within 60 seconds whether you actually understand it — or just ran someone else’s DAG.

This post is not a checklist of features.
It’s a breakdown of the top Airflow mistakes that break pipelines, burn teams, and ruin interviews — and how to avoid them.

Let’s make you sound like someone who’s deployed DAGs in production, not just built toy examples.

If your tasks are running huge pandas transformations, calling 15 APIs, and returning massive objects between tasks, you're doing it wrong.

Airflow is not Spark. Not dbt. Not a transformation engine. It’s the orchestrator — the conductor of the data workflow.

Do this instead:

• Keep tasks modular

• Push heavy lifting into external jobs (Spark, SQL, cloud ETL)

• Use Airflow to manage dependencies, not business logic

How to say this in interviews:

“I used Airflow to orchestrate pipeline steps, but offloaded heavy data processing to Snowflake and Spark jobs to keep the DAGs lean and observable.”

Most junior engineers write DAGs that work… when nothing goes wrong.

But in production:

• APIs time out

• S3 files get delayed

• Database connections drop

Best practices:

• Always configure retries, retry_delay, and on_failure_callback

• Log why each task failed (don’t just rely on the default log dump)

• Use idempotent task design — so retries don’t break downstream logic

Pro-level tip: Add exponential backoff and SLA alerts.

Upgrade to Annual