🚀 Flyte at Hacktoberfest 2024 As we shared during today's community sync, Flyte is participating in Hacktoberfest. ➡️ Head over to the list where we track all the labeled issues ⚙️ Make sure to follow the guidelines 🏆 We have prizes! 👓 Learn the details and check out the You all are welcome to participate!

This week in Flyte (Sep 30 - Oct 4) • Flyte 1.13.2 and flytekit 1.13.6 releases ◦ Feature-packed releases. Read the blog to learn what's new! • Flyte docs now use a new theme and much better structure. ◦ Note: if you're working on docs contributions, make to sure to have your fork in sync with main, as multiple sections moved to different locations. • #C07NTJH4WDVcurrent status: ◦ 24 assigned issues / 61 open (37 unassigned) ◦ 3 open PRs / 0 merged ◦ Learn about full guidelines and swag • October community presentation: --- Thank you!

This week in Flyte: Hacktoberfest summary • 35 assigned Issues / 19 unassigned • 4 merged PRs 13 days to go, keep up the great contributions! --- Upcoming Flyte community sync - November • Tue, Nov 5. 9:00 am PT • Attend on LinkedIn or Thanks!

📣 Monthly Contributors Spotlight Hello Flyte community! 👋 A huge thank you to our contributors for their incredible work this month: • @witty-action-93503 tackled a bunch of Jupyter notebook and interactive mode tasks, pushing through several pull requests to get everything wrapped up—awesome work! • @nutritious-telephone-58961 added support for extra copy commands in imagespec. • @clean-glass-36808 implemented pod template support for init containers. Vincent and Jason, you’ve both been super consistent in your contributions over the past few months—major kudos to you! We also want to give a big shoutout to @straight-stone-69718, @brainy-beach-71906, @creamy-shampoo-53278, @brief-window-55364, @mthemis-provenir, @fierce-oil-47448, Felix Mulder, and @gorgeous-waitress-5026 for your continued support and contributions. You all make Flyte better every day! 👏 Thanks for all you do!

Thanks everyone who participated in #C07NTJH4WDV We wrapped up with 24 merged PRs in

flyte

,

flytesnacks

, and

flytekit

+ 2 blog posts and 1 video! You all helped make the docs and examples better, which benefits the entire community 💜 We'll be reaching out to each contributor with shipping details for your well-earned swag 🎁 See you later today in the Community sync 🙂

Upcoming Flyte community sync - November

• Tue, Nov 5. 9:00 am PT

• Attend on LinkedIn or

YouTube▾

This week in Flyte (Oct 28 - Nov 8) • Xinzhou Liu, from Atomwise, gave a very insightful presentation at the community sync ◦ Recording • We wrapped up Hacktoberfest with 24 merged PRs by 15 unique contributors. Thanks to everyone who participated, including @gorgeous-carpenter-81547, @helpful-article-75328, @thousands-wall-59116 @adamant-agency-9910 @adorable-yacht-81565 and @big-ability-13493. We're in the process of shipping your well-earned swag 🙂 • Two proposals are open for community feedback: ◦ Flyteadmin RBAC + Project/domain isolation ▪︎ Focus: if you're using an IdP other than Okta, please comment how you're handling things like groups; managing interop between providers could be a challenge during the implementation of this proposal. ◦ Difficult to debug behavior for new users ▪︎ If you have faced the issue described there or can relate to its potential impact, please comment or upvote. Up next • Flyte talk at next week's K8s meetup in Munich ◦ Register • December community sync will feature @billowy-church-83438 introducing the Flyte K8s agent LinkedIn has developed for large-scale GNN workflow training! ◦ Attend on LinkedIn or YouTube

📣 Monthly Contributors Spotlight Hello Flyte community! 👋 This month has been an absolute whirlwind of contributions! We’re incredibly grateful for your efforts and deeply humbled by your unwavering support. Thank you for helping the Flyte ecosystem thrive! 🚀 Here’s a highlight of the work from our contributors this month: • @creamy-shampoo-53278 made multiple contributions, including support for reading structured datasets from local/remote uri, a flytefile uri bug fix, and a protobuf struct attribute fix. • @fierce-oil-47448 added support for a multi-file error aggregation strategy to distributed plugins. • @straight-stone-69718 handled flytectl panics for incorrect configurations and removed Python 3.8 support in flytekit as it has reached EOL. • @brainy-beach-71906 enhanced code coverage in the flyte repository with additional tests. • @clean-glass-36808 added basic SASL and TLS support for Kafka cloud events and decoupled Ray submitter, worker, and head resources. • @witty-action-93503 introduced an unsafe mode in flytekit that automatically pickles values without type annotations when set to true. • @brief-window-55364 added a new `AgentError` field to the agent protos. • @nutritious-telephone-58961 fixed union type bugs and addressed dataclass serialization issues. • @full-toddler-99513 added support for multipart blob download to copilot 🎉! These contributors have shown remarkable consistency—kudos to all of you! 👏 We also want to give a big shoutout to @ancient-lawyer-94886, Peter Lee, and Marcin Zabłocki for your continued support and contributions. You all make Flyte better every day! 🙌 Thanks for all you do!

📢 Flyte Community sync going live in 3'! https://www.linkedin.com/events/flytek8sagent-scalabledataservi7260425538542936066/theater/

Hey everyone: Union.ai (Flyte's biggest contributor) announced the Unified AI Platform at re:Invent this week, helping teams achieve up to 25x faster dev cycles that scale. Union’s Unified AI Platform powers your end-to-end AI development, accelerating the entire lifecycle of developing, managing, and deploying AI models. It unifies data, models, compute, and executions on a single platform, driving a new level of developer productivity. Union also announced some new features: 💰 Cost observability: a clear view into end-to-end costs at the container level as you build, train, and experiment with AI workflows. ♻️ Actors: reusable containers that dramatically reduce the cost and upstart time for tasks with similar environment requirements. ⏩ Serving: real-time serving to reduce infrastructure complexity, including scale-to-zero, autoscaling, token streaming, and more. Request a Demo here or react to this message with a ✋ to learn more

🎉 flytekit + flyte 1.14 are here The biggest releases of the year in terms of # features and new contributors. Highlights: • Dataclasses, dicts, and Pydantic BaseModels are finally supported on their native formats. • Consume Flyte from Notebooks natively, no plugins required. • `FlyteDirectory`is now supported as input to

ContainerTask

• No more hitting gRPC limits when moving large data, this is now offloaded automatically. • Much more! Dive into the flyte and flytekit release notes (which we are trying to make more descriptive than before 🙂 ). This truly feels like a team effort with the 30+ developers who merged their first contribution to Flyte. 💜Thank you!💜

📣 Monthly Contributors Spotlight Here’s a roundup of the contributions from our community members this month: • @microscopic-oyster-89823 updated the single binary YAML configuration. • @cool-lifeguard-49380 implemented fixes for distributed error aggregation opt-in and made appprotocols optional in the helm chart. • @clean-glass-36808 introduced refresh token support for device flow authentication. • @witty-action-93503 contributed a jupyter notebook example to flyte docs. • @echoing-account-76888 added raw container docs and examples using imagespec in container task. • @nutritious-telephone-58961 updated the JSON schema version priority and added more dataclass tests. • Vincent customized the demo cluster port. A huge shoutout to @creamy-shampoo-53278, @ancient-lawyer-94886, @dry-parrot-15675, and @full-toddler-99513 for your continued support and contributions—you make Flyte better every day! Thank you all for your incredible work! 🙌

🌏 Hello everyone! 🔎 Union is conducting research to understand the work experiences, challenges, and needs of ML/AI Managers to better support your workflows. Looking for: 🤝🏽 ML/AI Managers & analogous roles kubernetes Must be using Kubernetes 🤖 Must be fine-tuning LLMs Discussions are informal, 60 min or less, & no prep is needed! Your time and thoughts are valuable! 💰 Participants will receive a $60 Visa Gift Card If this sounds like you, grab a time slot here: https://calendly.com/chelsea-union/ml-r3 Questions? Contact chelsea@union.ai

Join the upcoming community sync to learn how the most recent major Flyte release enables higher flexibility and robustness for AI workflows: https://www.linkedin.com/events/fromnotebookstoproductionwithfl7283168164148875264/theater/ Tuesday, Jan. 21 9:00 am PT We hope to see you there 😀

📓 Support for Notebooks in Flyte and Union This week we blogged about this recent integration. It really felt like only scratching the surface on the possibilities 😎 BTW thanks again @witty-action-93503 for your invaluable contributions! ➡️ Read the blog ▶️ Join us next week for the community sync to watch a live demo and get your questions answered!

📢 In 90 minutes we'll go live with the Community Sync special episode You won't want to miss some of Flyte's core maintainers live demoing the newest features plus a preview of what's coming in 1.15 🔥 Bring your questions! Join us on LinkedIn or YouTube Today, 9:00 a.m. PT

📣 Monthly Contributors Spotlight Here’s a roundup of the contributions from our community members this month: • @creamy-shampoo-53278 enabled opening a `FlyteFile` from a remote path and added support for accessing `StructuredDataset` attributes in a dataclass. • @clean-glass-36808 cleaned up auto-refresh, added an option to disable synchronization on creation for the auto-refresh cache, and configured the Ray pod runtime class based on custom pod specs. • @dry-pizza-97077 worked on switching to domain-qualified finalizers. • @billowy-church-83438 added documentation for the Flyte K8s service agent and introduced a new Flyte agent to create, update, and delete K8s StatefulSets and services for GNN use cases. A huge shoutout to Chung Chuan Wu, @ripe-smartphone-56353, @cool-lifeguard-49380, Marcin Zabłocki, @witty-action-93503, Pranshu, and @nutritious-telephone-58961 for your continued support and contributions—you make Flyte better every day! Thank you all for your incredible work! 🙌

Hey everyone! Some recent updates: • 💪🏽The fantastic team at Adarga.ai, who gave a last year, open sourced a K8s operator that registers workflows using a CRD. Kudos! • 🎁 ICYMI: we had some of the Flyte maintainers "unboxing" the 1.14 release. • 🦜 Flyte School is so back ◦ Flyte auth deep dive + RBAC and compliance in Union ▪︎ Wed, Feb 26. 9:00 am PT ▪︎ Register ◦ Batch serving with the

flytekit-inference

plugin ▪︎ Wed, March 12. 9:00 am PT ▪︎ Register ◦ We have some topics in the queue so please vote: ▪︎ 1️⃣ Troubleshooting wf executions with Grafana/Union observability ▪︎ 2️⃣ Measuring and optimizing execution performance ▪︎ 3️⃣ Parallel computing series • Ray • Spark • MapTasks ▪︎ 4️⃣ Flyte Agents deep dive and how to build one from scratch ▪︎ Propose a topic 🙂 --- Thank you!

🦜 Flyte 1.15 is out! Here are the major highlights: 📊 Streaming Decks: Now you can see your task metrics, data, and logs in real-time while tasks are running - perfect for monitoring hyperparameter optimization and debugging! 🔐 Better Secrets Handling: Simplified secret configuration for both environment variables and files - way less verbose than before. ⚡️ Enhanced Task Caching: Unified caching system with new features like cache policies, custom salts, and the ability to ignore specific inputs from cache keys. 📦 Quality of Life Updates: - Progress bars for package creation/upload - Faster GitIgnore checks - Support for debugging @dynamic tasks - Flexible pod template overrides Check out full release notes: https://github.com/flyteorg/flyte/releases/tag/v1.15.0 💜 Thanks to 12 new contributors who made it possible!

Hello everyone: Circulating here a couple of new RFCs that are open for feedback. As a reminder, RFCs are a mechanism to discuss and design significant changes to the Flyte API, and your feedback is important; • Simplify the `Resources` API • Decentralize Flyte Agents config management Thank you!

Hi everyone: This Slack workspace has been renamed to *“Building AI Together by Union.ai*" Here’s what this means for you: Why the change? 1️⃣ Unifying Our Communities • Union.ai supports multiple open-source projects and tools—including Flyte, Pandera, and Union’s Serverless platform—and we want this workspace to be a hub for all these communities. • Whether you’re a Flyte user, a Pandera contributor, or exploring Serverless, this is your space to connect and grow. 2️⃣ Expanding Collaboration for MLOps Practitioners • While Flyte remains a core focus, we’re broadening the scope to encourage sharing best practices, lessons learned, and resources across the entire MLOps lifecycle. • Expect discussions on everything from workflow orchestration and data validation to scalable infrastructure and team workflows. Our Commitment to You ✨ What’s Not Changing: • Our dedication to helping ML and Data teams productionize AI applications remains unwavering. • Flyte-specific channels and community support will continue as always, alongside new opportunities to engage with other Union.ai projects. 🌱 What’s Growing: • More cross-community events, and collaborative problem-solving. • A stronger emphasis on real-world use cases and collective learning. Have questions or ideas? Drop them in this discussion or DM us anytime! Thank you!

📣 Monthly Contributors Spotlight Here's a roundup of this month's amazing contributors and their contributions: • @faint-pilot-24699 upgraded flyteadmin to use Mockery v2 and improved error handling by passing error messages to the failure node. • Hsuan-Chen Chen introduced support for overriding pod templates with `with_overrides`. • @clean-glass-36808 added support for merging ray head and worker node pod templates and enabled Ray pod runtime class configuration based on custom pod specs. • @echoing-account-76888 added pod template support for driver and executor pods and worked on multiple fixes in flytekit. A huge shoutout to @microscopic-truck-17789, @cool-lifeguard-49380, @big-memory-81882, @creamy-shampoo-53278, Mark Zhao, @silly-book-73230, redartera, @microscopic-oyster-89823, @average-twilight-23656, and @nutritious-telephone-58961 for your continued support and contributions—you make Flyte better every day! Thank you all for your incredible work! 🙌

Hi everyone: Union will be at the NVIDIA GTC conference next week and we'd like to connect with you: 1. Drop by the Union booth (#2022), get some swag and talk to @powerful-horse-58724 and others! 2. If you want a closer/wider discussion, you can book time with us using this link 3. Stop by the WandB booth for a live demo and see how Union and W&B empower you to iterate faster and productionize with confidence. 🗓️ Wed March 19 at 2PM For those attending, we hope to meet you there! For the rest of us I can only say: next week will be big 🚀

x-posting 😎: https://flyte-org.slack.com/archives/C06B389BX43/p1742313994282049

👋🏽 你好 Flyte community! Some recent updates: • ⚠️ To prevent a cognitive name collision with LLM Agents, Flyte Agents are being renamed to Connectors. This is a WIP and no user-level changes are expected. We'll keep you posted. • 🚀 flytekit 1.15.3 released, including: ◦ Webhook Connector: easily send workflow notifications using webhooks. No more email queues! #3058 ◦ Simplified interface for declaring Resources, including support for tuple/list and stricter validations. #3177 ◦ Customize the Spark Driver and/or Executor Pod specs using PodTemplates! #3016 ◦ Both the FileSensor and BaseSensor now support timeout, preventing the Connector from waiting indefinitely for a condition to be met. #2745 • 🔥 Great community content ◦ Datology.ai on how they achieve image-text data curation at massive scale with Flyte --> Read the blog ◦ Artera.ai on how they achieve data locality with globally distributed serving using Union! --> Read the blog • 🎤 Upcoming events ◦ ⚙️ Compound AI systems: unboxing a reference architecture ▪︎ April 3, 10:00 am PT ▪︎ Register ◦ 🌏 Build large-scale Xarray datasets for geospatial computing with Union ▪︎ April 8, 9:00am PT ▪︎ Register --- Thank you!

📣 Flyte Monthly Contributors Spotlight Here's a roundup of this month's amazing contributors and their contributions: • @creamy-shampoo-53278 contributed a long-awaited ✨ Slurm agent ✨! More details coming soon! 🎉 • @cool-lifeguard-49380 made multiple PRs updating the Kubeflow training operator and also contributed to the docs. • @early-bird-77878 added support for per-launch plan notification templates. • @bored-beard-89967 built a GeoPandas plugin and will soon share insights on scaling Xarray datasets with Flyte/Union. Stay tuned for his talk! • @echoing-account-76888 introduced support for configuring driver and executor pods in the Spark plugin. A huge shoutout to @microscopic-minister-38807, @future-football-28532, @microscopic-oyster-89823, @polite-london-84566, Daniel Brunkow, @steep-jackal-21573, @clean-glass-36808, and @average-twilight-23656 for your continued support and contributions—you make Flyte better every day! Thank you all for your incredible work! 🙌

Hello everyone: We just went live with the Flyte docs integrated into a single site. • You can access the Flyte version of the docs with the drop down selector at the top of the page • We are be rolling out redirects from the old docs to the new to reduce the number of broken links out in the wild • All old docs content should be present in the new docs. In some cases outdated or redundant content has been reorganized and rewritten but the information should still be there. • The new single-site will enable the docs to be improved and kept up to date much more effectively than was possible before • The community will still be able to contribute. We will be opening up the docs repo soon There will be some rough edges so please use this thread to report anything you may find. Thank you for your patience 🙂

Many of you folks probably do not know me - my name is Ketan Umare, and I am the original author of Flyte. Since open sourcing flyte, I have been part of this community and I am grateful to all the amazing folks who have help make Flyte an indespensible part of their Platforms. Today Flyte is used by some of the largest companies in the world to build and deliver their compound AI products. Recently, we decided to make a change, move Flyte docs to the same platform where we host union docs. Below is my rationale, Flyte is, and will always remain, an open-source project under the Apache 2.0 license and a Linux Foundation product. Our commitment to openness and independence remains unchanged. However, maintaining separate documentation sites has become an increasing burden on our small team at Union.ai. Today, Flyte’s documentation is primarily maintained by our team, rather than through broad community contributions. Managing the infrastructure for docs, in addition to maintaining Flyte itself, introduces unnecessary overhead. (Flyte docs are not simply python docs, there are golang, protobuf, python, java, terraform etc) To improve efficiency and focus on making Flyte better, we are moving our documentation to a unified platform at union.ai/docs. This decision is driven by practicality, not commercialization. Why the Move? • We are adopting Hugo, which offers *faster builds, versioning support, and local development capabilities*—major improvements over using pure Sphinx. • Hosting Flyte docs alongside Union docs reduces maintenance costs and simplifies infrastructure. • This transition allows us to improve documentation quality with more frequent updates and enhancements. We’re excited about the improvements this will bring, and you can expect to see major upgrades soon! 🚀 Let us know if you have any questions. Thanks for being part of the Flyte community!

📣 📚 Documentation Initiative Update As we wind down the week I want to take a minute and give an update on our documentation efforts. First, I want to extend a big thank you! for all the community members that helped us improve the navigation and discoverability of the content. Your reports helped us hone in on the most painful areas quickly and rapidly iterate to address them. We learned a lot this week and will put that back into practices to improve the content and UX going forward. We did improve the new site experience in various ways: 1. ✅ Content: Added, reviewed, and updated content for correctness to latest Flyte version 2. 🕵️‍♂️ Links: We poured over thousands of links and most (all?) saved bookmarks should work correctly. 3. 🕵️‍♂️ Search Engine: We scanned hundreds of Google search result pages and ensured they will navigate you to the most appropriate content. This is especially important for the users that start from a search engine 4. ☀️ UX: Improved theming 5. 📈 Flytekit Plugins: We improved the documentation of the Flyte Plugins, extracting data directly from the plugin a. As a consequence we found code/package/conflict issues with some plugins, which we will follow up with authors/maintainers to update b. We will investigate how to make it easier to catch those issues during development of plugins and pre-submits and ensure high-quality content from the get go 6. 🔍 Search: Search integration complete 7. ♻️ Escape Hatch: Temporary fallback to the legacy site. Whenever your search results return a bad link, you will be offered the opportunity to (temporarily) jump to the old documentation. a. If you never see this feature, it means the work above worked! b. If you do, please report so we can make sure the new content is up and up. Please don't hesitate to let us know if some content is not available, not easy to find, or if something is not behaving as it should. Thanks for being part of the Flyte community!

This week in Flyte 📚 Documentation migration updates: • Link Maintenance: Actively using a linkchecker to identify and fix broken internal cross-references and external links across the site. • API Docs Fixes: Addressing issues with API documentation autogeneration; aiming to re-generate API docs once resolved (targeting completion next week). • Redirect Audit: Planning to audit the old Flytekit API reference section and improve redirects to new API docs (current redirects lead to the top of the new section, requiring manual in-site search). • Please use this discussion to report any docs issue 🌍 New Flytekit plugin: automate xarray/zarr data movement between tasks with the new plugin. Now available as

flytekitplugins-xarray-zarr

! ⚙️ Calling all GKE users! The GKE and Akvelon teams developed a Flyte on GKE guide and are looking for feedback. Add your comments in the thread or try out the guide!

📣 Flyte Monthly Contributors Spotlight Here's a roundup of this month's amazing contributors and their contributions: • @witty-alligator-81055 removed the grpcio upper bound in the flytekit library. • @gorgeous-waitress-5026 enhanced the flyteadmin OIDC configuration by allowing separate issuer URLs for token validation. • @cool-lifeguard-49380 upgraded the kubeflow training operator. • @rich-garden-69988 fixed async s3fs to handle the latest changes to flytefile. • Hsuan-Chen Chen added a user-defined builder image to imagespec and support for avoiding duplicated handler registration. • @clean-glass-36808 has been diligently reviewing PRs! 🙏 • Joe Polin added a check to detect extraneous arguments in the task decorator. • @brainy-carpenter-31280 upgraded Spark to 3.5.5 and added support for IPv6 and Iceberg. • @bored-beard-89967 added a plugin to support xarray object types, which persists the data to a zarr store. • @aloof-painting-18735 fixed a bug in Spark fast registration that caused an infinite loop due to recursive archiving. Huge thanks to everyone for your outstanding efforts! 🙌