https://flyte.org
Join Slack
Hi, can anyone tell me what's the datatype for spark data frame in flyte?
# announcements
r
Hi, can anyone tell me what's the datatype for spark data frame in flyte?
t
could you elaborate please @refined-dawn-96621?
in a task you should just be able to use a 
pyspark.sql.DataFrame
which should get turned into a flyte 
StructuredDataset
type/literal on the backend
or a 
FlyteSchema
if you’re on an older version of flytekit
r
I am passing spark df as an argument of function and for flyte task i have to specify the dtype of argument e.g def abc(arg:int)->int here if i pass a df in argument what will be it's dtype here?
t
can you copy paste the full signature?
not quite sure what you mean by dtype
r
data type
if I am passing data frame as argument what will be it's data type just like I have used int in this example def abc(arg:int)->int
t
what kind of dataframe? Spark?
what is 
type(df)
r
yes spark data frame
t
can you just do
Copy code
@task
def task_name(a: pyspark.sql.DataFrame): ...
did you figure this out btw?
r
working on it
167 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by