Robert Ambrus
06/14/2023, 9:42 AMpyflyte
on the cluster and triggered the run like this:
pyflyte run -remote dbx_example_existing_cluster.py my_databricks_job
The Databricks job is started, but failing with this error:
TypeError: loader must define exec_module()
Kevin Su
06/14/2023, 9:50 AMKevin Su
06/14/2023, 9:50 AMKevin Su
06/14/2023, 9:51 AMRobert Ambrus
06/14/2023, 10:03 AMRobert Ambrus
06/14/2023, 10:04 AMRobert Ambrus
06/14/2023, 10:04 AMFor demonstration purposes, I saved time on setup by employing an existing cluster that already had the necessary Python dependencies installed. In a production environment, you can choose to configure a job cluster using a custom Docker image based on Databricks' base build, ensuring that all necessary dependencies are present.
Robert Ambrus
06/14/2023, 10:05 AMRobert Ambrus
06/14/2023, 10:06 AMpyflyte
package and copied the actual code
to /databricks/driver/databricks/
Robert Ambrus
06/14/2023, 10:07 AMEvan Sadler
06/14/2023, 1:01 PMRobert Ambrus
06/14/2023, 1:03 PMRobert Ambrus
06/14/2023, 1:05 PMRobert Ambrus
06/14/2023, 1:05 PMRobert Ambrus
06/14/2023, 1:05 PMRobert Ambrus
06/14/2023, 1:06 PMpyflyte
package as I did?Evan Sadler
06/14/2023, 1:11 PMRobert Ambrus
06/14/2023, 1:19 PMRobert Ambrus
06/14/2023, 2:07 PMflytekit
package, create a notebook with these import statements:
from flytekit.core.workflow import ImperativeWorkflow as Workflow
from flytekit.core.workflow import WorkflowFailurePolicy, reference_workflow, workflow
from flytekit.deck import Deck
from flytekit.image_spec import ImageSpec
from flytekit.loggers import logger
and run it, I got the same issue:Robert Ambrus
06/14/2023, 2:09 PMRobert Ambrus
06/14/2023, 2:09 PMRobert Ambrus
06/14/2023, 2:09 PMRobert Ambrus
06/14/2023, 2:10 PMRobert Ambrus
06/14/2023, 2:11 PMRobert Ambrus
06/14/2023, 2:11 PMDBR 11.3LTS
and flytekit==1.3.0b4
Robert Ambrus
06/14/2023, 2:18 PMModuleNotFoundError: No module named 'flytekit.image_spec'
Robert Ambrus
06/14/2023, 2:18 PMDBR 11.3LTS
and flytekit==1.7.0
Robert Ambrus
06/14/2023, 2:28 PMTypeError: loader must define exec_module()
again, so this issue might be specific to flytekit==1.7.0