https://flyte.org logo
Powered by
r

Rahul Mehta

07/29/2022, 1:00 AM
Is it possible to mix 
@flytekit.dynamic
& 
with_overrides
? Curious if we could pass a resource configuration object to a workflow at runtime and have the 
@dynamic
step specify overrides at runtime
k

Ketan (kumare3)

07/29/2022, 2:14 AM
Yes you should be able to
That was one of the intended usecases
r

Rahul Mehta

07/29/2022, 2:20 AM
Ah, I was trying to parameterize a workflow with `cpu`/`mem` so that we could dynamically specify them as arguments (in lieu of https://github.com/flyteorg/flyte/issues/475)
Looks like this successfully compiles w/ dynamic but if there's a sub-workflow (
@workflow
, not 
@dynamic
) then it leads to the following error
Copy code
AttributeError: 'Promise' object has no attribute 'cpu'
Although, when actually examining the pod in the cluster it looks like supplying args to 
with_overrides
for limits doesn't have any effect/I still see the following when running 
kubectl describe
on one of the pods (when I specified 
cpu="1"
& 
mem=100Mi
)
Copy code
Limits:
      cpu:     100m
      memory:  200Mi
    Requests:
      cpu:     100m
      memory:  200Mi
If I hard-code the values in it seems to work fine but it doesn't appear to be possible to specify 
with_overrides
w/ an argument to a workflow
Copy code
'VoidPromise' object has no attribute 'with_overrides'
If anyone on the team could provide an example of setting overrides in a dynamic step based on an argument that'd be very helpful..seems like it's possible per https://flyte-org.slack.com/archives/CP2HDHKE1/p1655135984234209, but I must be missing something…
s

Samhita Alla

07/29/2022, 5:11 AM
I guess mixing 
@flytekit.dynamic
and 
with_overrides
isn’t yet possible. @Smriti Satyan, I remember you telling that this didn’t work. Could you elaborate more on this, please?
k

Ketan (kumare3)

07/29/2022, 5:15 AM
s

Smriti Satyan

07/29/2022, 5:16 AM
Thats right @Samhita Alla. Facilitating 
dynamic
with 
with_overrides
is a work in progress. Right now, it can't be used. cc @Yee If we try to use 
with_overrides
with 
dynamic
workflow, it fails to execute.
k

Ketan (kumare3)

07/29/2022, 5:17 AM
@Rahul Mehta I am sorry in that case, let us look into this tomorrow. For some reason this feels like a bug
🙏 1
r

Rahul Mehta

07/29/2022, 6:37 PM
Filed https://github.com/flyteorg/flyte/issues/2730 w/ some more details of the issue
k

Ketan (kumare3)

07/29/2022, 9:47 PM
cc @Yee
It works for me on a production cluster
it does not work locally, but that is because of the way we interpret it and that should be a more complex fix to make it run locally
r

Rahul Mehta

07/29/2022, 10:30 PM
Ah this was on a production cluster with the sandbox helm chart deployed. To confirm I was running on remote. I'll upgrade all the deps and give this a shot next week
k

Ketan (kumare3)

07/29/2022, 10:30 PM
sure
if you hop on a call i can show you as well
r

Rahul Mehta

07/29/2022, 10:31 PM
Sure - thanks very much for digging into it so quickly @Ketan (kumare3) ! Will reach out next week
k

Ketan (kumare3)

07/29/2022, 10:31 PM
absolutely
r

Rahul Mehta

07/30/2022, 2:24 AM
Confirmed this is no longer an issue on the most recent version of the chart that I deployed. Thanks again @Ketan (kumare3)!
k

Ketan (kumare3)

07/30/2022, 2:26 AM
Interesting, I don't even know we fixed it
r

Rahul Mehta

07/30/2022, 2:26 AM
This is definitely going to be very powerful and is the final blocker we had to fully moving towards flyte 🙂
❤️ 1
k

Ketan (kumare3)

07/30/2022, 2:28 AM
Hopefully you will spread the word
👍 1
3 Views
Powered by