#1700 PR into #1679 - Support for generalized container types and clean-up of promise.py and workflow.py in local execution Pull request opened by wild-endeavor on <!date^1687289828^{date_short}|2023-06-20T19:37:08Z> This is a PR into #1679 (currently this PR is a bit hard to follow because it merged in master, hopefully that gets easier once it's merged) Blocked This PR is currently blocked - it's unclear if the current approach works. Currently the new dataclass transformer in the main PR interprets a dataclass as a literalmap type with a

map_value_type

set to an empty

LiteralType()

. This makes the compiler think that no bindings are needed. Since dataclasses are inherently multi-variate, we're at a bit of a impasse. And this PR depends on that. If this approach is untenable, most of this PR should be discarded. (The parts that should be retained are in the Other issues below.) One possible way around this is to make the new dataclass transformer follow the pattern of the existing dataclass json transformer - just expose a scalar, and only interpret the Background There's a few places in flytekit core today that blur the line between the type engine and not type engine. These are: • Extraction of literals from nested promises (

translate_inputs_to_literals

) for local execution • Extraction of literals from nested promises for local workflow execution (

ensure_literal

) • Construction of literals in pyflyte run (one example), there are many others in the file. This PR attempts to resolve the first two by • Merging the logic for the first two and moving to the type engine. • Generalizing what it means to be a container type (see the test here to see more clearly what that implies), • Remove

isinstance(..., list)

and

isinstance(..., dict)

• Adding three new functions to the type transformer: •

is_container_type

(returns bool) •

flyte_container_type

(returns LiteralCollection or LiteralMap) •

traverse

(responsible for appropriately walking the structure of the container and returning one of two generators. •

ListGen = typing.Generator[typing.Tuple[typing.Any, typing.Type[T], _type_models.LiteralType], None, None]

•

MapGen = typing.Generator[typing.Tuple[str, typing.Any, typing.Type[T], _type_models.LiteralType], None, None]

(Some code has also been left in place to allow

flyte_container_type

to return Scalar.) Other issues In addition to merging the two very related calls (

ensure_literal

and

extract_value

), some other bugs/issues uncovered: • the handling of unions in the

extract_value

subroutine was not correct but

ensure_literal

was correct. • we were not correctly walking bindings when the binding was a list or map. That is, if you do

x = t1()
    y = t2(a={"key1": x})

the

t1

node would not have been listed as an upstream node of

n1

flyteorg/flytekit GitHub Actions: build-plugins (3.11, flytekit-sqlalchemy) GitHub Actions: build-plugins (3.11, flytekit-polars) GitHub Actions: build-plugins (3.11, flytekit-papermill) GitHub Actions: build-plugins (3.11, flytekit-pandera) GitHub Actions: build-plugins (3.11, flytekit-hive) GitHub Actions: build-plugins (3.11, flytekit-dolt) GitHub Actions: build-plugins (3.11, flytekit-deck-standard) GitHub Actions: build-plugins (3.11, flytekit-data-fsspec) GitHub Actions: build-plugins (3.11, flytekit-dask) GitHub Actions: build-plugins (3.11, flytekit-bigquery) GitHub Actions: build-plugins (3.11, flytekit-aws-sagemaker) GitHub Actions: build-plugins (3.11, flytekit-aws-batch) GitHub Actions: build-plugins (3.11, flytekit-aws-athena) GitHub Actions: build-plugins (3.10, flytekit-vaex) GitHub Actions: build-plugins (3.10, flytekit-snowflake) GitHub Actions: build-plugins (3.10, flytekit-ray) GitHub Actions: build-plugins (3.10, flytekit-papermill) ✅ 13 other checks have passed 13/30 successful checks