https://flyte.org logo
k

komal azram

03/17/2022, 6:19 AM
Hi, can anyone tell me what's the datatype for spark data frame in flyte?
y

Yee

03/17/2022, 8:12 AM
could you elaborate please @komal azram?
in a task you should just be able to use a
pyspark.sql.DataFrame
which should get turned into a flyte
StructuredDataset
type/literal on the backend
or a
FlyteSchema
if you’re on an older version of flytekit
k

komal azram

03/17/2022, 8:20 AM
I am passing spark df as an argument of function and for flyte task i have to specify the dtype of argument e.g def abc(arg:int)->int here if i pass a df in argument what will be it's dtype here?
y

Yee

03/17/2022, 8:29 AM
can you copy paste the full signature?
not quite sure what you mean by dtype
k

komal azram

03/17/2022, 8:31 AM
data type
if I am passing data frame as argument what will be it's data type just like I have used int in this example def abc(arg:int)->int
y

Yee

03/17/2022, 8:33 AM
what kind of dataframe? Spark?
what is
type(df)
k

komal azram

03/17/2022, 8:34 AM
yes spark data frame
y

Yee

03/17/2022, 8:35 AM
can you just do
Copy code
@task
def task_name(a: pyspark.sql.DataFrame): ...
did you figure this out btw?
k

komal azram

03/18/2022, 6:14 AM
working on it
6 Views