https://flyte.org logo
Join Slack
Powered by
Upgrading to flytekit 1.0 and i have a task taking...
# flytekit
f
Upgrading to flytekit 1.0 and i have a task taking a single schema which can't marshal it to a dataframe because this conditional is false 
lv and lv.scalar and lv.scalar.schema
. looking at the inputs it seems that old versions of the task don't have the 
format
field in the flyte literal, but now they do. also looks like this code returns empty dataframe rather than erroring. any context on these changes that might help point to the behavior we're seeing?
cc @mammoth-lizard-41416
also cc @swift-animal-75798 for visibility
f
Cc @thankful-minister-83577 / @broad-monitor-993
@glamorous-carpet-83516
t
can you copy your type signatures for the failing task and also the tasks responsible for any inputs to the failing task?
is anything cached?
also can you send the full stack trace?
f
i can, but unfortunately this is an error in JasperSchema (or proto-structureddataset that we need to deprecate) so it won't point to your code
the main diff on flytebackend is the the addition of the 
format
field in the protobuf repr
t
yeah
but it’s a different proto message.
f
signature is bascically
Copy code
@task
def t1() -> pd.Dataframe
    ...
@task
def t2(s: JasperSchema) -> ???

@workflow
def w()
    t2(s=t1())
t
we didn’t add the field to schema, we changed the backend representation entirely of what a 
pd.DataFrame
is. now it’s this
but similar to flyteschema, the type message itself is in the literal, which is why you’re seeing it
f
oh shoot so it's serialized as a structured dataset 😢
well that explains it
t
yeah 😞
sorry we didn’t think of this use-case.
f
ok i think we can fix this by changing t1 to output jasperschema
t
yes
should be able to at least.
jasper is the thing that writes to bq?
f
yea, it's tech debt we need unwind but want to upgrade to 1.0 first
should this not be caught at registration time btw?
t
i guess… it’s hard to know though right?
if that had been FlyteSchema instead of JasperSchema, it should work
cuz we also added code going the other way
so the default flyteschema transformer should know how to handle cached structureddatasets
f
oh, so flyteschemas can read structured datasets
got it, ok that's good to know, it's an issue on our side
t
hang on… am i forgetting things again
161 Views
Open in Slack
PreviousNext
Powered by