https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Archit Rathore

10/05/2023, 1:10 PM
Curious if there are plans/interest to support callables are arguments? What I am really trying to do: I have a generic preprocessing task which is shared by many other teams, and based on the type of ML model we are training, it needs to invoke different preprocessing functions A simplified example:
Copy code
class ExperimentType(Enum):
    FRAUD: "fraud"
    SIMILARITY: "similarity"

@task
def preprocess(data_path: str, experiment_type: ExperimentType):
    # common preprocessing
    data_path = common_preprocessing(data_path)

    # experiment specific preprocessing
    if experiment_type == ExperimentType.FRAUD:
        preprocessed_path = fraud_preprocessing(data_path)
    elif experiment_type == ExperimentType.SIMILARITY:
        preprocessed_path = similarity_preprocessing(data_path)

    # another step of processing
    return final_preprocessing(preprocessed_path)
The hard thing right now is to maintain the enums and threading those through the entire pipeline + if-else ladders. If we can pass a function to the preprocess task, that would make the code quite a lot simpler.
k

Ketan (kumare3)

10/05/2023, 2:28 PM
We definitely are interested but not prioritized
Check experimental eager mode that might solve many problems
a

Archit Rathore

10/05/2023, 2:30 PM
We are excited about eager mode! But even in eager mode, flyte needs a way to serde the callable, right?
k

Ketan (kumare3)

10/05/2023, 2:31 PM
Ohh you want it to be user passable
a

Archit Rathore

10/05/2023, 2:34 PM
Yes, Flyte already knows how to find tasks using the task resolver, I was wondering if something similar could be used for passing vanilla python functions as task arguments • It would be useful even if the passable functions are restricted to not have references to globals or closures
k

Ketan (kumare3)

10/05/2023, 2:46 PM
You should be able to use pickle already
Not from Ui
a

Archit Rathore

10/05/2023, 7:11 PM
able to use pickle already
Say more? You mean use the PickleTransformer?
s

Samhita Alla

10/06/2023, 5:28 AM
Types that aren't Flyte-native are pickled by default.