Upgrading to flytekit 1.0 and i have a task taking a single schema which can't marshal it to a dataframe because this conditional is false

lv and lv.scalar and lv.scalar.schema

. looking at the inputs it seems that old versions of the task don't have the

format

field in the flyte literal, but now they do. also looks like this code returns empty dataframe rather than erroring. any context on these changes that might help point to the behavior we're seeing?

Cc @Yee / @Niels Bantilan

can you copy your type signatures for the failing task and also the tasks responsible for any inputs to the failing task?

i can, but unfortunately this is an error in JasperSchema (or proto-structureddataset that we need to deprecate) so it won't point to your code

yeah

signature is bascically

@task
def t1() -> pd.Dataframe
    ...
@task
def t2(s: JasperSchema) -> ???

@workflow
def w()
    t2(s=t1())

we didn’t add the field to schema, we changed the backend representation entirely of what a

pd.DataFrame

is. now it’s this

oh shoot so it's serialized as a structured dataset 😢

yeah 😞

ok i think we can fix this by changing t1 to output jasperschema

yes

yea, it's tech debt we need unwind but want to upgrade to 1.0 first

i guess… it’s hard to know though right?

oh, so flyteschemas can read structured datasets

hang on… am i forgetting things again