https://flyte.org logo
Join Slack
Powered by
Hello, Output data from a task can be consumed dir...
# ask-the-community
f
Hello, Output data from a task can be consumed directly by a subsequent task in the same workflow as the input data. However, what about consuming data from a task by another task from a separate workflow? One way is to save the output data from the first task to S3 and the second task can read the data from S3. Is there a better way provided by Flyte?
k
just add the task as a dependency and enable caching
you will not have to worry about recomputing it, as long as the inputs are the same
currently we do not really support consuming data from other workflows without ‘dependency’ but this is something we are interested in building and some of us are thinking hard about this. cc @Yee / @Eduardo Apolinario (eapolinario) / @Niels Bantilan
f
@Ketan (kumare3), thank your for the advices. Do you have an example for ‘add the task as a dependency and enable caching’?
k
example - is create a new workflow - with the existing task
🙂
so as an example
Copy code
@task(cache=True, cache_version="1.0")
def t1() -> int:
  ...

@workflow
def wf1():
   t1()
Copy code
@task
def t2(i: int) -> float:
  ...

@workflow
def wf2():
   i = t1()
   t2(i=i)
this should do the trick?
f
Yes, I will try it out. Thanks @Ketan (kumare3)!
CC @Adedeji Ayinde
@Ketan (kumare3), could you elaborate on tasks ’without ‘dependency’? What does it mean?
Thanks
@Ketan (kumare3), RE: you will not have to worry about recomputing it, as long as the inputs are the same If t1(data: pd.DataFrame), will that work? How does t2() in another workflow know what exactly is in data: pd.DataFrame?
k
So dataframe can work if you generate and create a hash for the dataframe- check out custom hash
e
@Frank Shen, here are the docs (with an example) explaining this idea of creating custom hashes for dataframes (and other kinds of complex objects): https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task_cache.html#caching-of-non-flyte-offloaded-objects
f
Thank you @Eduardo Apolinario (eapolinario)!
152 Views
Open in Slack
PreviousNext
Powered by