In this talk we will go over which benefits, drawbacks and lessons Airbus has encountered/learned in the switch from an API to a Python based Stream Architecture for continuous flight traffic prediction. The Goal is to highlight Stream based architectures as a architectural alternative and allow the attendees to decide if it could be a alternative to their current API based Setup worthwhile to look into.

The talk addresses the inefficiencies and limitations of traditional API-based architectures for real-time data delivery. Specifically, it explores challenges such as high latency, network overhead, customer effort, and scalability issues when APIs rely on polling mechanisms. These issues became apparent at Airbus during a project as customer needs evolved, highlighting the shortcomings of APIs in handling real-time updates effectively. This story of the project will aid as an example on how to identify a limiting architectural decision and what pain points can potentially be avoided by taking a new route guiding us along the talk.

For developers and architects building modern, data-driven applications, choosing the right architecture is critical. Many face similar challenges when scaling APIs for real-time use cases, such as IoT, financial data, or notifications. This problem is relevant because adopting an unsuitable architecture can lead to poor performance, higher costs, and frustrated users.

The proposed solution is to transition from an API-based architecture to a streaming architecture for real-time data delivery. This involves leveraging stream processing systems that push updates proactively, handle high-throughput data efficiently, and offer features like backpressure, partitioning, and stateful processing. Recent developments in Python based tools such as Bytewax, Faust and Quix are highlighted for their scalability and fault-tolerance capabilities.

Key Takeaways

1. Challenges of APIs: Polling APIs is inefficient for real-time updates, leading to delays, resource wastage, and customer dissatisfaction.
2. Advantages of Streaming: Streaming architectures offer real-time data delivery, lower latency, reduced customer effort, better scalability, and improved fault tolerance.
3. Key Streaming Concepts: Understanding backpressure, partitioning, and stateful processing is essential for understanding streaming specific limitations and solutions.
4. Architectural Considerations: Streaming is ideal for use cases where data changes frequently and needs to be delivered in real time, while APIs may still be suitable for low-frequency, static, or manual queries.
5. Strategic Transition: Adopting a streaming approach requires a paradigm shift in thinking about how data is delivered and processed, with significant changes to the system architecture which needs to be cautiously managed.