#1675 Fix a bug in gx integration Pull request opened by XinEDprob TL;DR Fixed a bug in the integration of Flyte and Great Expectations. Without this fix, pyspark dataframe will always be converted to pandas dataframe in Great Expectations. As a result, there will have an error when people want to use

SparkDFExecutionEngine

. Type ☑︎ Bug Fix ☐ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☐ Smoke tested ☑︎ Unit tests added ☑︎ Code documentation added • [] Any pending items have an associated Issue Complete description I use

selected_datasource[0]["execution_engine"]["class_name"]

to recognize the

execution_engine

declared by the user for Great Expectations. If it is a

SparkDFExecutionEngine

, the data which is in the form FlyteSchema is converted to

pyspark.sql.dataframe.DataFrame

, else transform the data to the default

pandas.dataframe

. Tracking Issue https://github.com/flyteorg/flyte/issues/ Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks