Sorry team I won’t be able to make it to the contributors sync today!

I will join the sync today (no time no see 😆 busy with internal stuff recently)

https://github.com/flyteorg/flytekit/pull/1846 Propagating

FlyteRecoverableException

from torch elastic worker processes up to the main process to fix retries. Can somebody please review? 🙏

Hey there all

I just want to request maintainers, if my can get merged faster

As I have taken part in a competition, which ends on 31st October. If I get the PR merged before that, it will help me a lot

https://github.com/flyteorg/flyte/pull/3964 What should we do with this PR? It fixes a wrong flyteadmin port setting in the helm chart in case flyteadmin uses a TLS certificate. I first thought I needed this for the GCP IAP integration effort but in the end didn’t. It’s still a bug though

was evaluating this PR: https://github.com/flyteorg/flyte/pull/3256 but when I run flytectl demo start .... I get a lot of errors:

E1031 19:35:27.628662      66 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create containerd task: cgroups: cgroup mountpoint does not exist: unknown" pod="flyte/flyte-sandbox-postgresql-0"
E1031 19:35:27.628687      66 kuberuntime_manager.go:1166] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to create containerd task: cgroups: cgroup mountpoint does not exist: unknown" pod="flyte/flyte-sandbox-postgresql-0"
E1031 19:35:27.628758      66 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"flyte-sandbox-postgresql-0_flyte(43c0a74e-0d0d-46b9-b4e5-50d76ca102d7)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"flyte-sandbox-postgresql-0_flyte(43c0a74e-0d0d-46b9-b4e5-50d76ca102d7)\\\": rpc error: code = Unknown desc = failed to create containerd task: cgroups: cgroup mountpoint does not exist: unknown\"" pod="flyte/flyte-sandbox-postgresql-0" podUID="43c0a74e-0d0d-46b9-b4e5-50d76ca102d7"
W1031 19:35:28.620888      66 manager.go:1159] Failed to process watch event {EventType:0 Name:/kubepods/besteffort/pod174bfdd2-289e-45f4-8d89-3420e1fe3835/668db6ca7463b2bc7452a10b05dd6e48d41dc989cfb6b0771b958996958c76f2 WatchSource:0}: container "668db6ca7463b2bc7452a10b05dd6e48d41dc989cfb6b0771b958996958c76f2" in namespace "<http://k8s.io|k8s.io>": not found

I also have this error at the top of my logs:

sed: couldn't flush stdout: Device or resource busy
time="2023-10-31T19:41:12Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
time="2023-10-31T19:41:12Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/ab2055bc72380bad965b219e8688ac02b2e1b665cad6bdde1f8f087637aa81df"
time="2023-10-31T19:41:15Z" level=info msg="Starting k3s v1.28.2+k3s1 (6330a5b4)"

Does anyone have any idea why https://github.com/flyteorg/flyte/blob/master/docker/sandbox-bundled/bin/k3d-entrypoint-cgroupv2.sh#L19 this line might be failing?

Good morning team! I’ve seen that the

conda

packages for

flytekit

(here; latest

1.9.1

)) as well as

flyteidl

(here; latest

1.5.17

)) are not up to date. At the moment it’s not possible to release

flytekit

as that depends on

flyteidl>=1.10.0

, for which the auto-updater does not work (status page here; search for “flyteidl” in the “Errored” field). I’ve gone ahead and fixed the auto-update (by removing the duplicate

tests

section) and bumped `flyteidl==1.10.0 (this PR). I’ve also taken the liberty to add myself as a maintainer (this PR). Once that is in, I will go ahead and update

flytekit

. cc @Eduardo Apolinario (eapolinario)

Hi community, As this PR https://github.com/flyteorg/flytekit/pull/1896 merged, the integration tests of flytekit remote has been restored. Since the current code coverage is not that high, any contribution to the tests would be awesome! If you run into any problem while contributing, feel free to ping me!

Featured issue: • https://github.com/flyteorg/flyte/issues/3671 (a.k.a. Support for auth providers other than native K8s token) ◦ Open question/call to action: ▪︎ We could setup a Working Group around this effort, but we'll need some feedback from the community to gauge interest. ▪︎ If this feature request is something you/your organization need, please add a 👍🏽 reaction to the Issue

@David Espejo (he/him) / @jeev can either of you take a look at this PR https://github.com/flyteorg/flyte/pull/4337/files

cc @Guy Arad

🔥 Hey community, i would like to recommend @Yi Chiu as a core contributor for the significant contribution he did especially for rust data layer and vscode plugin. In the vscode one, he works closely with Linkedin to make necessary changes to open source it. Nomination: https://github.com/flyteorg/community/issues/18 Here is the contribution he made: Plugin: • vscode plugin: https://github.com/flyteorg/flytekit/pull/1922 (done, in review) • Improve data persistence: ◦ Rust: https://github.com/flyteorg/flytekit/pull/1936 (in progress) ◦ Python: https://github.com/flyteorg/flytekit/pull/1869 (in progress) Flytekit: • Add type transformer for datetime.date: https://github.com/flyteorg/flytekit/pull/1786 merged • Add support for execution name prefixes: https://github.com/flyteorg/flytekit/pull/1803 merged • add default project domain in execute launch plan: https://github.com/flyteorg/flytekit/pull/1894 merged • Fix remote integration tests: https://github.com/flyteorg/flytekit/pull/1896 merged • Add support for execution of reference entities: https://github.com/flyteorg/flytekit/pull/1808 done, in review • local cache ignore portion: https://github.com/flyteorg/flytekit/pull/1914 done, in review Flyte: • Provide a way to ignore portions of the signature for cache key calculation: https://github.com/flyteorg/flyte/pull/4324 done, in review • fix the compatibility to Mac: https://github.com/flyteorg/flyte/pull/4260 merged Flytesnacks: • add type datetime.date: https://github.com/flyteorg/flytesnacks/pull/1084 merged • fix execution name: https://github.com/flyteorg/flytesnacks/pull/1095 merged • add execution name prefix examples: https://github.com/flyteorg/flytesnacks/pull/1098 merged Thanks everyone for the considering. cc @Ketan (kumare3) @David Espejo (he/him) @Yee @Kevin Su

PTAL https://github.com/flyteorg/flyte/pull/4375 to add aws gov vs commercial parameterization to the Athena log urls.

Awesome!

Working on this PR: (Docker plugin -- use

docker

to build a container from

ImageSpec

with a defined

Dockerfile

)) https://github.com/flyteorg/flytekit/pull/1926 wanting to get some opinions on potential changes to the

ImageSpec

interface. (none implemented yet, just put the link as a reference) Basically: If you want to define a container for a task that Flyte manages/builds you currently must define an

ImageSpec

. I would like to create a new 'builder' that builds images via a

Dockerfile

, but I'm finding the current interface pretty restricted. I find the current

ImageSpec

implementation to be mainly targeted at

envD

. This is strange as

ImageSpec

as-is is in flytekit core, but is essentially useless without

flytekit-envd

installed. The current implementation also requires

flytekit-envd

to be installed on the task-node as the

EnvdImageSpecBuilder

must be instantiated at task-runtime as .build gets run at some point. I was wondering if any devs had thoughts on ways I might decouple

ImageSpec

from

envD

? Or if any thought has been put into this before? I have a few ideas, but each has tradeoffs and I don't want to restart discussions that have already been had. I will say that I would like to be able to use

Dockerfiles

with the least amount of modification possible, so it's not really ideal to add a

pip install flytekit-docker

step to every D`ockerfile` I want to use, but maybe that is unavoidable.

@Dennis Keck and I observed and tried to debug a truly peculiar bug in propeller this week 🤔 https://github.com/flyteorg/flyte/issues/4384 We would appreciate if you could take a look at the issue and see if this this rings any bells. Thanks!

Reminder: Flyte contributor's meetup will take place TOMORROW (Thursday) at 11AM PT. Agenda/notes - Add to your calendar

Hi, community, here's 2 PRs need to be reviewed. It's about • sync plugin interface for flyteplugins • agent sync plugin implementation • API_TASK Engine for Sync Plugin Task (a DoTask agent in flytekit) • ChatGPT plugin I am going to share the use case about the sync plugin and chatgpt in the next Flyte community sync, so I hope that it can be merged before then. Kevin and Honnix have already given me tons of advice, so there won't be any basic errors. Here's an upcoming post in the Flyte blog written by me, it might be helpful if the agent framework is unfamiliar to you. https://hackmd.io/@flyte-community/ryaSdNtmT Note: I know those files are really much, so I try to list the changes in the description and draw some diagrams to help you understand. When I become more skilled, I will help the community review PRs, thanks really much. Thanks really much. PR Links: https://github.com/flyteorg/flyte/pull/4107 https://github.com/flyteorg/flytekit/pull/1822

Hi @David Espejo (he/him), @Devin Stein is my friend from Docu, he is working on an AI agent which can help mitigating the maintenance cost of GitHub projects. He would like to briefly talk about the idea of integrating FLyte + Docu in the next community Sync. David can you help scheduling for him? Thanks! Devin feel free to provide more details!

Is there any plan when flytekit 1.10.1 will be released? 🙂

Next Contributor's meetup lands on Nov. 23, US Thanksgiving holiday. Ok to skip that one and keep it async? We'd be back on December 7th.

@Fabio Grätz / @Bernhard Stadlbauer / and so many others, how about talking at kubecon EU conference and sharing some things in flyte?

We should create flyte GPT cc @David Espejo (he/him) 🚀

Reminder: Flyte contributor's meetup will take place TOMORROW (Thursday) at 11AM PT. Agenda/notes - Add to your calendar

🚀 Hey community, I just nominated @L godlike as a flyte committer. He has consistently demonstrated exceptional dedication in contributing to the Flyte project, making invaluable contributions to the community. Nomination: https://github.com/flyteorg/community/issues/19 Here is the contribution he made: • ChatGPT agent • Databricks agent • Sync plugin • FlyteChatGPTSummaryBot • Email notification in the sandbox • Improve test coverage ◦ flyteorg/flyte@033276d ◦ flyteorg/flyte@ac4ec7d • Added Flyte sensor example. • Added GPU enabled sandbox image • Tons of PRs to improve the Flyte agent • Tons of PRs to improve the Kubeflow plugin • Flyte blog, tutorial, video. •

I am curious, what is the difference between flyte committer and flyte core contributor? i don’t find “committer” in https://github.com/davidmirror-ops/community-fork/blob/e6bbaff7ec78d58c2978317b6df5e8c5140cc134/GOVERNANCE.md cc @David Espejo (he/him)

committer is much understandable and widely used!