Robert Ambrus
06/14/2023, 9:42 AMpyflyte
on the cluster and triggered the run like this:
pyflyte run -remote dbx_example_existing_cluster.py my_databricks_job
The Databricks job is started, but failing with this error:
TypeError: loader must define exec_module()
Kevin Su
06/14/2023, 9:50 AMRobert Ambrus
06/14/2023, 10:03 AMFor demonstration purposes, I saved time on setup by employing an existing cluster that already had the necessary Python dependencies installed. In a production environment, you can choose to configure a job cluster using a custom Docker image based on Databricks' base build, ensuring that all necessary dependencies are present.
pyflyte
package and copied the actual code
to /databricks/driver/databricks/
Evan Sadler
06/14/2023, 1:01 PMRobert Ambrus
06/14/2023, 1:03 PMpyflyte
package as I did?Evan Sadler
06/14/2023, 1:11 PMRobert Ambrus
06/14/2023, 1:19 PMflytekit
package, create a notebook with these import statements:
from flytekit.core.workflow import ImperativeWorkflow as Workflow
from flytekit.core.workflow import WorkflowFailurePolicy, reference_workflow, workflow
from flytekit.deck import Deck
from flytekit.image_spec import ImageSpec
from flytekit.loggers import logger
and run it, I got the same issue:DBR 11.3LTS
and flytekit==1.3.0b4
ModuleNotFoundError: No module named 'flytekit.image_spec'
DBR 11.3LTS
and flytekit==1.7.0
TypeError: loader must define exec_module()
again, so this issue might be specific to flytekit==1.7.0