Tag: #airflow
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 14 posts
14 posts are tagged #airflow; the most recent was published on 2026-07-17.
Often tagged together: #dagster 7 #data-engineering 7 #dbt 6 #kafka 5 #2026-03 4
Most read in the last 90 days:
- Durable Execution & Workflow Engines 2026 — Temporal, Inngest, Trigger.dev, Hatchet, Restate, DBOS, Step Functions Deep Dive (Infrastructure for the Long-Running AI Agent Era)
- Workflow Engines in 2026 — Temporal, Inngest, Trigger.dev, Hatchet, Restate, DBOS, Airflow Deep Dive (Mapping the Durable Execution Era)
- Hadoop Ecosystem & Data Engineering 2026 Deep Dive - Hadoop, Spark, Flink, Trino, Iceberg, Delta Lake, Hudi, Airflow, dbt
After Airflow 2 EOL — The Real Work List for Going 2 to 3, and Where 3.x Stands at 3.3
Apache Airflow 2 passed EOL on April 22, 2026, and it no longer receives security patches. But the road from 2 to 3 is not a pip upgrade — it is an architecture change. Direct metadata-DB access from workers disappears,
2026-07-17 · 15 min read #data-engineering#airflow#workflow-engine#migrationDurable Execution & Workflow Engines 2026 — Temporal, Inngest, Trigger.dev, Hatchet, Restate, DBOS, Step Functions Deep Dive (Infrastructure for the Long-Running AI Agent Era) ♪ Listenable
In 2026, workflow engines sit on top of two massive waves — first, long-running AI agents (LLM jobs that run for minutes to days) have effectively mandated durable execution; second, Postgres is becoming the backbone of
2026-05-25 · 32 min read #durable-execution#workflow-engine#temporal#inngest#trigger-devHadoop Ecosystem & Data Engineering 2026 Deep Dive - Hadoop, Spark, Flink, Trino, Iceberg, Delta Lake, Hudi, Airflow, dbt ♪ Listenable
A complete look at the data engineering stack in May 2026. The truth behind the "Hadoop is dead" headline (HDFS is shrinking, but YARN and the lakehouse pattern survive), the conclusion of the Iceberg vs Delta Lake vs Hu
2026-05-16 · 25 min read #hadoop#spark#flink#trino#prestoCron & Scheduled Tasks in 2026 — systemd timers / Vercel Cron / AWS EventBridge Scheduler / k8s CronJobs / BullMQ / Sidekiq / Temporal Deep Dive ♪ Listenable
The 2026 landscape of scheduled tasks. Classic cron is still alive, and systemd timers became the modern Linux replacement. Cloud-side: Vercel Cron, AWS EventBridge Scheduler (replaced CloudWatch Events Rules in Nov 2022
2026-05-16 · 25 min read #cron#scheduled-tasks#systemd-timers#vercel-cron#aws-eventbridge-schedulerWorkflow Engines in 2026 — Temporal, Inngest, Trigger.dev, Hatchet, Restate, DBOS, Airflow Deep Dive (Mapping the Durable Execution Era) ♪ Listenable
The AI-agent era has dragged workflow engines back into tier-1 infrastructure for every backend team. Temporal Cloud — founded by the people who built Uber Cadence — is the de facto standard, Inngest and Trigger.dev v3 a
2026-05-15 · 24 min read #workflow#durable-execution#temporal#inngest#trigger-devData Engineering Complete Guide — Lakehouse, Streaming, dbt, Orchestration, Data Mesh (Season 2 Ep 8, 2025)
Data engineering is no longer "running ETL scripts." In 2025 a data engineer designs Lakehouse architectures (Iceberg, Delta, Hudi), combines Streaming (Flink, Kafka) with Batch (Spark), standardizes modeling with dbt, r
2026-04-15 · 10 min read #data-engineering#lakehouse#iceberg#delta-lake#hudiData Orchestration 2025 Complete Guide: dbt, SQLMesh, Dagster, Airflow, Prefect, Data Contracts, CI/CD (2025)
Season 5 Ep 4. If engines query, orchestrators govern the pipelines. The standardization of dbt and its limits, the revolt of SQLMesh, the asset-centric model of Dagster, the evolution of Airflow 2.x and 3.0, the rebirth
2026-04-15 · 13 min read #data-orchestration#dbt#sqlmesh#dagster#airflowData Engineering Fundamentals — ETL, Data Warehouses, Streaming, and Data Lakes
How do you build data pipelines? From ETL/ELT and data warehouses to streaming, batch processing, Spark, Kafka, and Airflow.
2026-04-13 · 17 min read #data-engineering-aws#data-engineering#etl#data-warehouse#sparkData Engineering Pipeline Complete Guide 2025: ETL/ELT, Spark, Airflow, Real-Time Streaming
Everything about data engineering! ETL vs ELT, Apache Spark (PySpark), Apache Airflow (DAG/Operator/Sensor), real-time streaming (Kafka+Flink), dbt (data transformation), data warehouses (BigQuery/Snowflake/Redshift), da
2026-03-25 · 18 min read #data-engineering#etl#elt#spark#airflowData Engineering Complete Guide 2025: Flink vs Spark, dbt, Iceberg, Airflow — Modern Data Stack Deep Dive ♪ Listenable
Flink vs Spark stream processing comparison, dbt crossing $100M ARR, Iceberg vs Delta Lake vs Hudi lakehouse showdown, ClickHouse vs StarRocks real-time analytics, Airflow 3.0 vs Dagster vs Prefect orchestration — the co
2026-03-22 · 30 min read #data-engineering#apache-flink#apache-spark#dbt#apache-icebergToss Bank ML Engineer (MLOps) Complete Guide: From MLFlow to LLM Platform — Tech Stack Deep Dive ♪ Listenable
Complete analysis of Toss Bank ML Platform Team MLOps Engineer JD. Deep dive into MLFlow, Airflow, JupyterHub, Kubeflow, Triton Inference Server, ScyllaDB Feature Store, and LLM platform — with 30 interview questions and
2026-03-21 · 38 min read #mlops#ml-platform#tossbank#kubernetes#mlflowData Engineering & AI Pipeline Guide: From Apache Spark to Kafka
A comprehensive guide to data engineering for AI. Covers Apache Spark, Kafka, Airflow, dbt, Delta Lake, and Feature Stores for designing and implementing large-scale data pipelines.
2026-03-17 · 12 min read #data-engineering#apache-spark#kafka#airflow#dbtOrchestrating ML Training Pipelines with Airflow ♪ Listenable
Analyzing methods for automating ML training pipelines using KubernetesPodOperator, Dynamic Task Mapping, and more, based on the official Apache Airflow documentation.
2026-03-01 · 21 min read #airflow#mlops#ml-pipeline#orchestration#kubernetesAirflow Basics
The Basics of Apache Airflow
2024-07-27 · 1 min read #airflow