when we pass df to and from flyte tasks - is there...
# flyte-supportf
faint-cat-74485
03/12/2024, 4:26 PMwhen we pass df to and from flyte tasks - is there any native way to ask Flyte to do sampling as its reading and writing the data?
eg i sometimes want to just randomly sample or write N rows to/from a task and wondering if Flyte could in some way do this natively?
faint-cat-74485
03/12/2024, 4:27 PMuse case here is say task A makes a big df
task B1 and B2 need that as input - B1 needs it all but B2 actually only needs a random sample of N from df
h
high-accountant-32689
03/13/2024, 1:35 AM@faint-cat-74485, we don't support this today. That said, how should the sampling be expressed? Let's take your example, how would sampling of the upstream df be defined?
f
faint-cat-74485
03/13/2024, 6:24 AMI'm not 100% sure tbh, maybe I could pass the sampling param to the task decorator. Maybe it's a horrible idea idk. Just that I find myself having some tasks where first thing I do is quickly downsample the data right away.
h
high-accountant-32689
03/13/2024, 8:10 PM@faint-cat-74485, can you capture that in github issue? [flyte-core]
u
user
03/13/2024, 8:10 PM⭐ Create a new Flyte Core Feature issue: https://github.com/flyteorg/flyte/issues/new?assignees=&labels=enhancement%2Cuntriaged&template=feature_request.yaml&title=%5BCore+feature%5D+
f
faint-cat-74485
03/13/2024, 8:35 PM