https://flyte.org logo
Title
d

Dylan Wilder

08/01/2022, 3:40 PM
Upgrading to flytekit 1.0 and i have a task taking a single schema which can't marshal it to a dataframe because this conditional is false
lv and lv.scalar and lv.scalar.schema
. looking at the inputs it seems that old versions of the task don't have the
format
field in the flyte literal, but now they do. also looks like this code returns empty dataframe rather than erroring. any context on these changes that might help point to the behavior we're seeing?
cc @Govind Raghu
also cc @Babis Kiosidis for visibility
k

Ketan (kumare3)

08/01/2022, 3:45 PM
Cc @Yee / @Niels Bantilan
@Kevin Su
y

Yee

08/01/2022, 3:59 PM
can you copy your type signatures for the failing task and also the tasks responsible for any inputs to the failing task?
is anything cached?
also can you send the full stack trace?
d

Dylan Wilder

08/01/2022, 4:01 PM
i can, but unfortunately this is an error in JasperSchema (or proto-structureddataset that we need to deprecate) so it won't point to your code
the main diff on flytebackend is the the addition of the
format
field in the protobuf repr
y

Yee

08/01/2022, 4:02 PM
yeah
but it’s a different proto message.
d

Dylan Wilder

08/01/2022, 4:03 PM
signature is bascically
@task
def t1() -> pd.Dataframe
    ...
@task
def t2(s: JasperSchema) -> ???

@workflow
def w()
    t2(s=t1())
y

Yee

08/01/2022, 4:04 PM
we didn’t add the field to schema, we changed the backend representation entirely of what a
pd.DataFrame
is. now it’s this
but similar to flyteschema, the type message itself is in the literal, which is why you’re seeing it
d

Dylan Wilder

08/01/2022, 4:06 PM
oh shoot so it's serialized as a structured dataset 😢
well that explains it
y

Yee

08/01/2022, 4:07 PM
yeah 😞
sorry we didn’t think of this use-case.
d

Dylan Wilder

08/01/2022, 4:08 PM
ok i think we can fix this by changing t1 to output jasperschema
y

Yee

08/01/2022, 4:08 PM
yes
should be able to at least.
jasper is the thing that writes to bq?
d

Dylan Wilder

08/01/2022, 4:44 PM
yea, it's tech debt we need unwind but want to upgrade to 1.0 first
should this not be caught at registration time btw?
y

Yee

08/01/2022, 4:45 PM
i guess… it’s hard to know though right?
if that had been FlyteSchema instead of JasperSchema, it should work
cuz we also added code going the other way
so the default flyteschema transformer should know how to handle cached structureddatasets
d

Dylan Wilder

08/01/2022, 4:46 PM
oh, so flyteschemas can read structured datasets
got it, ok that's good to know, it's an issue on our side
y

Yee

08/01/2022, 4:46 PM
hang on… am i forgetting things again