Hi community are there plans on supporting more complicated

Question

Hi community are there plans on supporting more complicated logic for conditions As noted on the Flyte documentation ```Conditions are limited to certain binary and logical operators and can only be performed on primitive values ```

Dan Rammer (hamersaw) · Answer

Hey Yubo What kind of conditions did you have in mind

Yubo Wang · Answer

I am thinking about non primitive types like we define some custom objects and expose a function that conditional can depend on for the evaluation

Yubo Wang · Answer

< Ankit Goyal> FYI

Ankit Goyal · Answer

< Yubo Wang> perhaps a specific example would be helpful to share

Dan Rammer (hamersaw) · Answer

Can you mock up a python example maybe

Yubo Wang · Answer

in a sec

Yubo Wang · Answer

``` dataclass json dataclass class TFTrainerConfig object dataset config DatasetLoaderConfig hyperparameters Hyperparameters column names List str model code str batch size int num training steps int num eval steps int num eval frequency int model output names map Optional dict workflow def tf2 trainer wf trainer config TFTrainerConfig res = conditonal batch size greater if trainer config batch size gt 10 then ```

Yubo Wang · Answer

something like this

Yubo Wang · Answer

and also something like this ``` workflow def tf2 trainer wf trainer config TFTrainerConfig res = conditonal batch size greater if trainer config model output names map not None then ```

Dan Rammer (hamersaw) · Answer

Yeah so this would be very difficult to do using conditionals Conditionals are evaluated using BranchNodes within propeller which are able to be processed interally given the simple boolean logic For this you would basically need to spin up a Flyte task to evaluate It really doesn t fit the current BranchNode setup I m wondering if this can be done with dynamic tasks though ``` task def process success foo dynamic def tf2 trainer wf trainer config TFTrainerConfig trainer config batch size gt 10 process success ``` of course this spins up a separate Pod to process this and compiles injects a new DAG So there is some level of overhead

Dan Rammer (hamersaw) · Answer

Or you could have a separate task to return the condition variable ``` task def process bool return trainer config model output names map == None workflow def tf2 trainer wf trainer config TFTrainerConfig condition = process trainer config res = conditonal batch size greater if condition then ``` which also has some level of overhead obviously

Yubo Wang · Answer

thanks for the suggestion but we were trying to avoid dynamic workflows due to the overhead

Dan Rammer (hamersaw) · Answer

sure makes plenty of sense

Yubo Wang · Answer

I am looking into the implementation of BranchNode

Ankit Goyal · Answer

so is the only thing that doesn t work here is that `trainer config` is a dataclass If it was just an int it would work right

Yubo Wang · Answer

yes that is what I am doing for now

Yubo Wang · Answer

inputting as flags like `use model bundle creator`

Yubo Wang · Answer

interesting < Dan Rammer hamersaw > Does condition support chaining When I put ``` workflow def tf2 trainer wf use model bundle creator bool trainer res = trainer task res = conditional use model bundle creator if use model bundle creator is true then model bundle creator trainer res gt gt res``` I got `AttributeError Condition object has no attribute ref `

Dan Rammer (hamersaw) · Answer

< Eduardo Apolinario eapolinario > how are `dataclasses` stored as If there was some way to encode the specific field of the dataclass in a `Literal` proto then in propeller when we evaluate in the BranchNode you could look up the field and return the primitive value

Dan Rammer (hamersaw) · Answer

re chaining another question for eduardo I handle mostly backend stuff and can t speak for flytekit syntax with any authority sweat smile cc < Niels Bantilan>

Yubo Wang · Answer

Thanks Dan

Dan Rammer (hamersaw) · Answer

No problem Let me know if you find anything in BranchNode implementation I just took a quick look I think with some changes to Literal if we can parse things our correctly from flytekit the backend may be able to retrieve the primitive value We would have to have a larger discussion on this but it could be very useful

Eduardo Apolinario (eapolinario) · Answer

< Dan Rammer hamersaw > sorry for the delay was afk for a bunch of hours Regarding how we store `dataclasses` as literals We on

Yubo Wang · Answer

hey < Eduardo Apolinario eapolinario > thanks for looking at this so this means that we can actually implement some parsing as long as the input is decorated with dataclass json

Eduardo Apolinario (eapolinario) · Answer

< Yubo Wang> yeah I think so We d need to change a few things in the implementation of conditionals in propeller but nothing dramatic AFAIU Dan to correct me

Yubo Wang · Answer

Another question regarding to conditionals seems to me that the conditionals do not support empty else clause for example ```conditional use model bundle creator if use model bundle creator is true then model bundle creator else ```

Samhita Alla · Answer

Yeah Empty `else` isn t supported I think it makes sense to mandatorily input what needs to be done in case `if` isn t traversed

Yubo Wang · Answer

what if I expected nothing to be done for the else clause

Samhita Alla · Answer

That isn t possible You either need to call a task or initialize `fail`

Niels Bantilan · Answer

the `conditional` construct is more like ```x = 0 if cond else 1``` as opposed to ```if cond x = 0 technically don t need an else here``` i e all conditions must be specified You could use a `noop` task in the else clause to workaround this

Yubo Wang · Answer

< Niels Bantilan> Any example that I can refer to I can t seem to find noop task on the official doc or github repos

Niels Bantilan · Answer

oh just like ``` task def noop ```

Yubo Wang · Answer

does that require spinning up a container to handle it I assume it would

Yubo Wang · Answer

Hi < Niels Bantilan> I have this question We have this use case that we would add in some tasks subflows based on the user inputs by leveraging Flyte s conditions As you suggested using a noop task which is at cost of spinning up a container task I am thinking about if we introduce an `end ` function similar to the use of that allows us to do nothing on a branch

Yubo Wang · Answer

Do you think the above solution is doable

Samhita Alla · Answer

Makes sense < Niels Bantilan> WDYT

Niels Bantilan · Answer

yep this use case makes sense < Yubo Wang> can you create a flyte core issue below point down to support the `end ` use case If I understand correctly this would be for cases where you don t want to do any thing in one of the conditional branches right

Slackbot · Answer

star Create a new Flyte Core Feature issue <https github com flyteorg flyte issues new assignees= amp labels=enhancement 2Cuntriaged amp template=feature request yaml amp title= 5BCore+feature 5D+>

Dan Rammer (hamersaw) · Answer

< Yubo Wang> is this something you might be willing to contribute It sounds like you already dove through the propeller code a bit but would be happy to help

Yubo Wang · Answer

Sounds good I will be working on this