#3855 [BUG] Flyte task keeps running forever when running a Databricks job Issue created by rambrus Describe the bug BACKGROUND: I'm trying to run a simplified Flyte task using Databricks plugin. PREREQUISITES: • Flyte 1.5.0 installed on EKS cluster • Flyte Databricks plugins is configured as described here: https://docs.flyte.org/en/latest/deployment/plugins/webapi/databricks.html • entrypoint.py is uploaded to DBFS • Databricks cluster is created with the following settings: DBR 11.3 LTS, installed libs: flytekit==1.5.0, flytekitplugins-spark==1.5.0 • A simplified Flyte workflow is created:

@task(
    task_config=Databricks(
        databricks_conf={
           "run_name": "dbx simplified example",
           "existing_cluster_id": "<my-existing-cluster-id>",
           "timeout_seconds": 3600,
           "max_retries": 1,
       }
    ),
    limits=Resources(mem="2000M"),
    cache_version="1",
)
def print_spark_config():
    spark = flytekit.current_context().spark_session
    print(spark.sparkContext.getConf().getAll())

@workflow
def my_databricks_job():
    print_spark_config()

STEPS: • Run workflow:

pyflyte --verbose run --remote --destination-dir . dbx_simplified_example.py my_databricks_job

ISSUE: Databricks job run triggered and successfully completed, but the Flyte job status is not updated, it is stuck in

RUNNING

state. Expected behavior Flyte job status keeps updated. Additional context to reproduce Please note that we tried to upgrade Flyte to v1.7.0: • when I also upgraded the Flyte packages on the Databricks cluster to 1.7.0 -> #3853 • when I used the 1.5.0 Flyte packages on the Databricks cluster -> same issue - Databricks job successfully completed, but Flyte job keeps running forever Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte