Channels
#ask-the-community
Title
# ask-the-community
f
Frank Shen
01/30/2023, 11:58 PMHello,
Output data from a task can be consumed directly by a subsequent task in the same workflow as the input data.
However, what about consuming data from a task by another task from a separate workflow?
One way is to save the output data from the first task to S3 and the second task can read the data from S3.
Is there a better way provided by Flyte?
k
Ketan (kumare3)
01/31/2023, 12:30 AM
just add the task as a dependency and enable caching
you will not have to worry about recomputing it, as long as the inputs are the same
currently we do not really support consuming data from other workflows without ‘dependency’ but this is something we are interested in building and some of us are thinking hard about this.
cc @Yee / @Eduardo Apolinario (eapolinario) / @Niels Bantilan
f
Frank Shen
01/31/2023, 12:54 AM@Ketan (kumare3), thank your for the advices.
Do you have an example for ‘add the task as a dependency and enable caching’?
k
Ketan (kumare3)
01/31/2023, 12:55 AM
example - is create a new workflow - with the existing task
🙂
so as an example
Copy code
@task(cache=True, cache_version="1.0")
def t1() -> int:
...
@workflow
def wf1():
t1() Copy code
@task
def t2(i: int) -> float:
...
@workflow
def wf2():
i = t1()
t2(i=i)this should do the trick?
f
Frank Shen
01/31/2023, 12:59 AMYes, I will try it out.
Thanks @Ketan (kumare3)!
CC @Adedeji Ayinde
@Ketan (kumare3), could you elaborate on tasks ’without ‘dependency’? What does it mean?
Thanks
@Ketan (kumare3),
RE: you will not have to worry about recomputing it, as long as the inputs are the same
If
t1(data: pd.DataFrame),
will that work?
How does t2() in another workflow know what exactly is in data: pd.DataFrame?
k
Ketan (kumare3)
02/01/2023, 3:07 AM
So dataframe can work if you generate and create a hash for the dataframe- check out custom hash
e
Eduardo Apolinario (eapolinario)
02/01/2023, 6:23 PM@Frank Shen, here are the docs (with an example) explaining this idea of creating custom hashes for dataframes (and other kinds of complex objects): https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task_cache.html#caching-of-non-flyte-offloaded-objects
f
Frank Shen
02/01/2023, 7:20 PMThank you @Eduardo Apolinario (eapolinario)!
2 Views