https://flyte.org
Join Slack
Hi team.. for task caching to work, do the tasks need to return something? I’m running a workflow wi...
# flyte-support
l
Hi team.. for task caching to work, do the tasks need to return something? I’m running a workflow with 3 tasks that don’t return anything, have caching enabled but I don’t see anything being written to or read from the cache and the tasks are rerunning even when inputs are the same
t
what are your inputs? what does the ui show? is there a check mark next to it?
next to the node in the execution on the console page
you are using the tasks purely for side-effects?
l
I’m using tasks because I need to run them in a specific sequence and they have dependencies
inputs for each task are a bunch of s3 urls
I don’t see the check mark
t
can you say a bit more? what do you mean by specific sequence and dependencies?
and you can confirm that the tasks have caching enabled?
like cache=True in the Task JSON tab?
l
workflow has 3 python tasks. They all have (cache=True, cache_version=“1.0”) I’ve provided the dependencies which enforces the ordering
each task takes say a couple of s3 urls as input.. but does not return anything
I don’t see cache in the task details
even though I have this 
@task(cache=True, cache_version="1.0")
t
are you sure you’re on the version you expect?
how are you registering?
l
flytectl register files
t
and how are you registering?
package?
package will produce a tgz file. do you think you can unzip it and look at one of the tasks?
l
yes.. package
I see the .pb files…
t
can you pick a task pb file
and then run this command
Copy code
flyte-cli parse-proto -f /path/to/task.pb -p flyteidl.admin.tasks_pb2.TaskSpec
that should dump out the task json
and if that is not working…
l
Trying to resolve this error 
ImportError: cannot import name '_message' from 'google.protobuf.pyext'
t
what version of python are you on?
l
3.8.9
t
that looks like the pip protobuf dependency is missing
l
after installing protobuf 3.14 I’m getting this error 
'NoneType' object has no attribute 'message_types_by_name'
t
this is what i have
Copy code
googleapis-common-protos==1.56.3
proto-plus==1.20.3
protobuf==3.20.1
l
let me install these
Now I’m back to 
ImportError: cannot import name '_message' from 'google.protobuf.pyext'
t
are you on an m1?
l
yes
f
@yee why not get the task Json from the ui
l
I looked at the task details in UI too
not sure where I can find the caching info?
I confirmed my theory with a very basic task.. if the task returns something it is cached, else it is not
f
Yes output is required
else what are we catching
l
I guess it was just “knowing” that the task has been run already with those inputs.. not a problem… I’m now returning a status string so we are good
👍 1
155 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by