Channels
#announcements
Title
k
komal azram
03/17/2022, 6:19 AMHi, can anyone tell me what's the datatype for spark data frame in flyte?
y
Yee
03/17/2022, 8:12 AM
could you elaborate please @komal azram?
in a task you should just be able to use a
pyspark.sql.DataFrameStructuredDatasetor a
FlyteSchemak
komal azram
03/17/2022, 8:20 AMI am passing spark df as an argument of function and for flyte task i have to specify the dtype of argument e.g def abc(arg:int)->int here if i pass a df in argument what will be it's dtype here?
y
Yee
03/17/2022, 8:29 AM
can you copy paste the full signature?
not quite sure what you mean by dtype
k
komal azram
03/17/2022, 8:31 AMdata type
if I am passing data frame as argument what will be it's data type just like I have used int in this example def abc(arg:int)->int
y
Yee
03/17/2022, 8:33 AM
what kind of dataframe? Spark?
what is
type(df)k
komal azram
03/17/2022, 8:34 AMyes spark data frame
y
Yee
03/17/2022, 8:35 AM
can you just do
Copy code
@task
def task_name(a: pyspark.sql.DataFrame): ...did you figure this out btw?
k
komal azram
03/18/2022, 6:14 AMworking on it
6 Views