Flyte is the backbone for large-scale Machine Learning and Data Processing (ETL) pipelines at Lyft. It is used across business critical applications ranging from ETA, Pricing, Mapping, Autonomous etc. At its core it is a Kubernetes native workflow engine that executes 40M+ containers per month as part of ~1M pipeline executions. Flyte also helps Lyft orchestrate more than 300k Spark workloads a month. We introduced Flyte at Kubecon 2019. This talk will build on the base talk and dive deeper into the architecture and the developer experience. Thus the talk will help existing users of Flyte and new users as well.
- Quick overview of Flyte concepts
- Architecture of Flyte
- How we overcame various challenges to scale Kubernetes for stateful workloads.
- Provide an indepth look at building a real time forecasting platform @Lyft on top of Flyte.
- We will try to conclude with a demo
