Astronomer

🤖 Did you know OpenAI powers its ETL pipelines with Apache Airflow? Download this free eBook for 9 production-ready ETL examples, complete with a linked GitHub repo: Examples include: → ETL pattern DAG using Dynamic Task Mapping → ELT pattern DAG using explicit external storage → ETL DAG using XCom Link below:

Legacy systems are holding hedge funds back. Slow pipelines. Manual reporting. Stale data. AI-first trading and real-time analytics can’t run on outdated infrastructure. That’s why top firms use Astro to power real-time fraud detection, LLM apps, and real-time reporting. Get this whitepaper to learn: 💼 3 priorities reshaping hedge funds (AI, data, investors) 🦺 How Astro eliminates latency and compliance risk 🤑 Real-world results from leading funds Link below.

🤖 Sure, you can ship another chatbot, but what about an application your competitor can’t vibe code within a day? That takes data orchestration. Learn how to upgrade from AI experimentation to shipping scalable, production-grade data products with Apache Airflow. Download this eBook to deep dive into: → Event-driven inference execution → Multi-agent workflows → Latest Airflow 3 GenAI features Link below.

Implementing CI/CD in your Airflow workflows Still skipping Dag tests to “save time”? One production SQL bug can cause hours of debugging, missed revenue, and unhappy customers – not to mention burnout. Join Astronomer’s webinar, hosted by Marc Lamberti, on December 4 at 11am ET/4pm GMT to learn how implementing CI/CD in your Airflow workflows can help your team:  🔸Automate testing and deployment management 🔸Reduce maintenance costs 🔸Improve code quality and reliability Register now: https://lnkd.in/dF7-Zk6X This post is sponsored by Astronomer. Thank you for supporting our community!

🤔 With recent leaps in AI readiness in Apache Airflow 3.1, the time is finally here to achieve exactly that: creating multi-agent data pipelines with human-in-the-loop steps for critical and final reviews, all interconnected within your existing Airflow setup. This new blog post shares a deep dive of a data pipeline that orchestrates 10 specialized AI agents working together, combining tools like Airflow, Apache Kafka, Aryn, and OpenAI. Link (with full code repo) below.

How do you test autonomous vehicles without constantly hitting the road? In this episode, Christian Schilling of Bosch unpacks how they use Apache Airflow to run large-scale closed-loop and open-loop simulations. By automating and orchestrating thousands of hours of test data, they ensure software reliability, accelerate bug fixes and cut costs — all without needing physical road tests for every update. Click the link in the comments to learn more. #AI #Automation #Airflow #MachineLearning

Most teams only use Apache Airflow for simple scheduling… but Airflow becomes a different kind of powerhouse when you treat it as the foundation of your data platform rather than just an orchestrator. Over the years, I’ve seen Airflow evolve into a central control plane that can drive: ✨ Data Quality — automated checks, anomaly detection, contract validation, SLAs 📦 dbt Integration — tightly coupled transformations, environment-aware runs, lineage-aware triggers 🔥 Spark & Big Data Workloads — launching distributed jobs, monitoring executors, and simplifying complex pipelines 🧩 Custom Operators — purpose-built tasks for business logic, cross-cloud APIs, and internal systems 🔍 Compliance & Observability — logging, auditing, alerting, and event-based workflows Airflow becomes incredibly powerful when you extend it to match your team’s specific needs. Instead of adapting your processes to fit the tool, you shape Airflow to support your data ecosystem. If you’re still using Airflow only for cron-style jobs… you’re leaving a lot on the table. #ApacheAirflow #DataEngineering #DataPlatform #ModernDataStack #dbt #Spark #DataQuality #Orchestration #WorkflowAutomation #MLOps #ETL #DataOps

Airflow can be turned into a simulation toolbox for chemistry and scale. “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI” features Anja M. at Covestro, revealing how custom operators, UI plugins, and Kubernetes with PVCs enable manual, expert-driven workflows, share 100+ GB outputs and keep simulations reproducible and compliant. Click the link in the comments for the episode. #AI #Automation #Airflow

🏆 We're proud to be recognized as a 2025 #Fast500 winner by Deloitte as one of the fastest growing companies in North America! This award is a testament to the booming popularity of Apache Airflow and Astro amidst the rise of LLMs, as part of a major shift in how organizations think about data orchestration. We also recently wrapped the largest bookings quarter in Astronomer's history, and will continue to be a champion for the data engineering community. Congratulations to all of the winners, and view the full list at the link below.

Our Board of Directors has concluded its CEO search and I’m humbled and proud to have been picked to lead Astronomer as its next CEO. In many ways, Astronomer feels like a part of my identity – I grew up at this company, starting 8.5 years ago with an incredible team of people spending our days preaching the good word of Apache Airflow. It’s been the journey of a lifetime since, fueled by an incredible team of folks who have joined the fray and helped drive Astronomer to the next level. While much has changed, much has stayed the same – as AI fundamentally transforms how we all interact with and derive value from data, we remain committed to raising the ceiling for data engineers. I did some math – I’m now in my 34th fiscal quarter working on Astronomer. I must admit, this one was a bit different from the rest. We kicked things off navigating a moment of uncertainty for the company, but the team here remained unequivocally committed to focusing on our customers, our product, and each other. Thanks to incredible execution across every single department, this wound up becoming the biggest bookings quarter in Astronomer’s history. A few highlights that led us to such an outstanding result: - A wave of recent major product updates our product and engineering teams have been shipping. The work data engineers do is critical at every organization, and our team is building incredible things to help them work faster and smarter: - Astro IDE - Our code agent and editor to help data engineers ship 10x faster, moving from prompt, to pipeline, to sandboxes for testing and deployment in seconds. - Astro Private Cloud - Brings enterprise-grade security, unified governance, and proven scalability to on-prem environments. - Apache Airflow 3.1 - Improved support for AI workflows, a new plugins interface, human-in-the-loop capabilities, and synchronous DAG execution.   - World-class execution from our GTM team, prioritizing customer value and delivering results at every turn for organizations like Ramp, Autodesk, Northern Trust, and Texas Rangers Baseball Club. - A community that’s only continuing to pick up steam: With 43,000+ GitHub stars, Apache Airflow is one of the most popular Apache projects of all time. It was incredible hearing from teams at OpenAI, Anthropic, GitHub, and more at the recent Airflow Summit – orchestration is the backbone to data and AI, and we’re privileged to serve such an incredible community of data engineers. - A move into our new and improved NYC HQ! Vibes in the new space are great. Shoot me a note if you are ever in Flatiron and want to stop by or cowork! Much work to be done, but the future here at Astronomer is bright. And now, back to work…