Is there a way to know inside a type transformer if it is being called for a

task

vs

dynamic

or

workflow

What I am really trying to do: • I have a sprawling workflow, that has multiple nested dynamics • Passing large lists/dataclasses through multiple levels of the nesting seems to slow the UI down significantly • I already have a TypeTransformer that creates a compact representation of the large inputs (by having users apply a custom decorator to the type definitoin of their large inputs) ◦ But for better inspectibility, I would like to expand the compact representation back into the original flyte representation, but only on

task

Is that possible?

I don’t follow. Honeythe way to think is - types are constant I. Programming languages. You can alter viz of these types in Python layer for sure. This can be done by using different types that transform to the same underlying type in Flyte. For example pandas dataframe, polars dataframe are all structured dataset

Maybe the example of custom TypeTransformer could help:

def to_literal(
        self,
        ctx: FlyteContext,
        python_val: PickleDataClassMixin,
        python_type: Type[PickleDataClassMixin],
        expected: LiteralType,
    ) -> Literal:
        path = persist_python_value(python_val)
        json_repr = maybe_get_json_repr(python_val)

        blob_literal = Literal(
            scalar=Scalar(blob=Blob(uri=path, metadata=BlobMetadata(type=self._TYPE_INFO)))
        )

        if json_repr is None:
            literal = Literal(map=LiteralMap(literals={"pickle_path": blob_literal}))
        else:
            structure_literal = Literal(
                scalar=Scalar(generic=_json_format.Parse(json_repr, _struct.Struct())),
            )
            literal = Literal(
                map=LiteralMap(
                    literals={"pickle_path": blob_literal, "structure": structure_literal}
                )
            )
        return literal

In this example: • the transformer returns

dict(pickle_path=blob_literal, structure=structure_literal)

if the object is json serializable, else returns

dict(pickle_path=blob_literal)

• The

to_python_value

method only uses the

pickle_path

to convert back into python object This works quite well - objects with a json repr show up with both

pickle_path

and

structure

attribute in flyte console (see screenshot 1) What I’d like to do is to switch between

pickle_path + structure

and only

pickle_path

based on if the literal is being created to be passed into a

task

vs

dynamic

(hope that makes it more clear, I am not trying to mutate types, just their representations in the UI)

Cc @Yee can you opine

no i don’t think this is possible without changing core flytekit. flytekit does know that it’s operating for a dynamic task, but the translation of inputs/outputs happens at the outer layer, before the task specific stuff happens.

however, what’s the core issue here though? Is it just UI performance?

Yes, this is especially worse when trying to run an HPO where one could want to spawn 100s of sub-workflows to train models across different hyperparameters

@Archit Rathore can you jump on to union serverless and try this, (dummy example) and if you then can share we can jump and help better

can you jump on to union serverless and try this

Happy to! Should I join the waitlist from here: https://signup.union.ai/?