Hi Team, we are planning to move our data-bricks integration to use the agent service, we ran some test and found one issues while running the config is that its not able to find the

databricksInstance

we we run the workflow we get this issue in flyte console as well as agent logs

flyteagent failed to create spark task with error: 'databricksInstance'.

Here's the propeller config we have used

enabled_plugins.yaml: |
    tasks:
      task-plugins:
        default-for-task-types:
          container: container
          container_array: k8s-array
          sidecar: sidecar
          databricks: agent-service
          spark: spark
        enabled-plugins:
        - container
        - sidecar
        - k8s-array
        - agent-service
        - spark 
---
  agent_service.yaml: |
    plugins:
      agent-service:
        defaultAgent:
          endpoint: dns:///flyteagent.ml-flyte-int.svc.cluster.local:8000
          insecure: true
        supportedTaskTypes:
        - spark

To make it work we have to specify the

databricks_instance

manually in the

task_config

@task(
    task_config=Databricks(**DATABRICKS_CONFIG, databricks_instance="<http://dbx.example.com|dbx.example.com>"),
    container_image="<http://docker.io/example/image:0.0.1|docker.io/example/image:0.0.1>"
  
)

Just wondering if thats normal when running databricks tasks with agent, or are we missing anything in the configuration

Did you configure the databricks plugin? Otherwise, how would the agent-service know how to call databricks: https://docs.flyte.org/en/latest/deployment/plugins/webapi/databricks.html#specify-plugin-configuration

not this is not databricks plugin right, this is agent?

Yes it not plugin its agent, and yes I have already got the databricks config available thats what I was thinking that it should pick the instance from there but I am getting above error in the agent.

oops disregard

Cool, thanks for checking it I just wanted to confirm if we are not missing anything in the config.

@adventurous-ability-21671 updating the docs is way more durable so that'd be a great contribution

Coll, will raise PR and let you know.