@Eduardo Apolinario (eapolinario) @Kevin Su I can’t see your replies to https://flyte-org.slack.com/archives/CP2HDHKE1/p1696954554555229 (it’s stuck on loading …) , so I am opening a new thread. Do you guys need me to open a bug report?

You can use discuss.Flyte.org

kk, https://github.com/flyteorg/flyte/issues/4243

Cc @Kevin Su

agent ?

We are introducing a framework of writing all external service plugins in python - things like databricks, bigquery etc. they can be executed locally and in the remote run like a service with no pod overhead (like today) but simple to write https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/development_lifecycle/agent_service.html#extend-agent-service

@Georgi Ivanov You can use this first: https://flyte--4008.org.readthedocs.build/en/4008/deployment/agents/databricks.html#deployment-agent-setup-databricks

thanks

Doc: https://github.com/flyteorg/flyte/pull/4008

Agent is available in 1.10, but the api might change slightly

Do you mean the flyteidl?

@Ketan (kumare3) I’ve tried to switch to flyteagent from the built-in plugins in propeller. I was able to start the agent service and point propeller to it. I am using flyte 1.10.0 by the way. However when I submit a databricks job to propeller which gets sent to the agent, instead of the agent contacting databricks and crafting an API call to start the job on a cluster, it starts it in a pod in the project-domain namespace. The pod is using this image cr.flyte.org/flyteorg/flytekit:py3.9-1.10.0 and the job is failing as the image does not have the flytekitplugins-spark python package installed. Seems wrong to me that the job is started in k8s and not in databricks. I believe I’ve used the correct configuration for the agent so I don’t know what I am doing wrong. Is there a way to troubleshoot this or perhaps someone can give me some guidance?

@Georgi Ivanov Maybe you can try to build a image with dockerfile or use imagespec feature. Here are the examples: https://github.com/pulls?q=is%3Apr+author%3AFuture-Outlier+archived%3Afalse+is%3Aclosed

@Georgi Ivanov it seems it’s not using agent but using regular container task execution

@Georgi Ivanov Can you provide the sandbox-configmap? I want to make sure you setup like this.

@L godlike @Ketan (kumare3) I can definitely register the agent with the propeller and the propeller is able to submit jobs to it. The errors above where because I was not adding a proper task decorator while I was testing the agent. Anyway I reverted to the correct task decorator and the propeller is sending the job to the agent. several observations - in the agent service plugin, the databricks agent class expects the databricks_conf in a certain format, i.e it expects “new_cluster” stanza https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-spark/flytekitplugins/spark/agent.py#L36-L39. If “new_cluster” is not present it errors with “failed to create spark task with error ‘new_cluster’“. We can’t use “existing_cluster_id” for instance. - we need to pass “databricks_instance” as part of the task decorator. The databricks instance is read from the task context by propeller when it constucts the task template when it passes it to the agent. From what I’ve found out, there is no mention anywhere in the docs how to add the databricks instance to the agent config (in the propeller configmap). If we don’t do it the agent error out with “failed to create spark task with error ‘databricksInstance’” - passing a valid databricks cluster config (as taken from the databricks UI as json) to the “new_cluster” stanza in the “databricks_conf” block in the task decorator and submitting the job results in {“asctime”: “2023-11-01 180833,995”, “name”: “flytekit”, “levelname”: “ERROR”, “message”: “failed to create spark task with error Failed to create databricks job with error: Bad Request.“} . I’ve tried increasing the log level on the agent by adding this file to the agent /etc/flyte/config/config.yaml logger: level: 6 show-source: true and adding this env var to the agent - name: FLYTECTL_CONFIG value: /etc/flyte/config/config.yaml but I can’t see anything else apart from the above log entry. Can you give me some hints on: • how to test my job locally by submitting it to databricks from my laptop • how to increase the verbosity of the agent to see what’s happening • how to add databricks instance and the entrypoint py settings to the agent config (currently I don’t know how or where to specify the entrypoint py)

@Georgi Ivanov 1. For the config structure, you can found it here: https://docs.databricks.com/dev-tools/api/2.0/jobs.html 2. Does you think the file here should provide the URL? https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-spark/flytekitplugins/spark/agent.py#L36-L39 We had provided here, so I think it's not necessary to add 1 more URL, but maybe it is better to add 1 more reference here.

@L godlike / @Kevin Su would love to see what is the api for databricks agent and cane we make it more type checked

For the verbosity, we have a PR here, but the FlyteIDL hasn't realeased yet, so please wait for it. https://github.com/flyteorg/flytekit/pull/1834

@Ketan (kumare3) @L godlike this is what I’ve found out: - the databricks built-in plugin allows you run a databricks jobs using the jobs API and you can either use a new_cluster for the job or an existing cluster. - the databricks agent expends you to use a new_cluster, i.e. you can’t use an existing cluster. Both use the Databbricks Job API 2.0. This means we can’t create a new cluster as part of a job and attach a new policy to it (while creating). Our org has strict rules and each cluster must have a policy. Thus the databricks API call submitted by Flyte when creating a new_cluster fails with PERMISSION_DENIED. I am able to run databricks jobs with the built-in databricks plugin on existing_clusters though. Also if I use a service account with elevated permissions, I am not able to see the job as jobs 2.0 API does not provide for access control lists. There is nothing code-wise we can fix in Flyte to overcome this - if Databricks is setup to require cluster policies for job clusters this will fail as it is a limitation of the Jobs API 2.0. However Databricks Jobs API 2.1 is a solution for this. It allows specifying policy_id when creating a cluster. Furthermore, it allows specifying access_control_list when you submit the job. This means flyte no longer has to use databricks user PAT to submit jobs, but rather use a single databricks service principal PAT, and provide access_control_list stanza (which can be passed as part of the databricks_conf) via the task decorator to say who can do what to the job. The access_control_list stanza is pretty flexible - users, group, service principals and a plethora of access permissions. https://docs.databricks.com/api/workspace/jobs/submit https://docs.databricks.com/en/workflows/jobs/jobs-api-updates.html Migrating the current Agent to 2.1 job spec is pretty easy imho. However Job 2.1 provides multi-tasks configurations, which might or might not be useful to adopt in the agent (I don’t know much about flyte to say). Since flyte packages the whole workflow as a since task/job it might not be useful anyway.

Thanks a lot, will take a look and update here

do you want me to create a feature request ?

Yes, you are right, currently databricks agent don't support for local.

Ohh does it need container

We will support 2.1 api and databricks local execution in this month.

thank you

@Georgi Ivanov Just create a PR here to solve support Databricks API 2.1 version. Would you like to try it? https://github.com/flyteorg/flytekit/pull/1935

@Georgi Ivanov qq: the reason you switch to flyte agent is that you want to run the databricks task locally, right? IIRC, you were using databricks plugin in Propeller, did you run into any issue?

@Georgi Ivanov Here's a solution for FlytePropeller https://github.com/flyteorg/flyte/pull/4361

@Kevin Su the databricks plugin in propeller works. However the reason we want to move the to agent is twofold. First, this is where the direction of using webapi plugins is going and we want to adopt that. This will speed up local development as well as people can test jobs locally using their tokens before moving them to flyte. Second, since the agent is written in python we can extend it, customize it if we need to. Currently we have added extra checks on the databricks side - like mandatory policy id for all new clusters. Instead of asking end users to add one with all their tasks for instance, we can add some logic to the agent to add one. And off course the access policies.

@Georgi Ivanov This is exactly what we are building the agent system for. But it is not yet fully mature

@Kevin Su can I ask what is the preferred way to submit databricks jobs?

3

Hi @Kevin Su, That would be very nice to have especially when we can do this locally for spark tasks that runs on k8s spark operator by passing the AWS S3 creds locally like this. Is this how databricks plugin will work similarly locally?

task_config=Spark(
        spark_conf={
            ....
            # The following is needed only when running spark task in dev's local PC. Also need to do this locally: export SPARK_LOCAL_IP="127.0.0.1"
            "spark.hadoop.fs.s3a.access.key": "aaa",
            "spark.hadoop.fs.s3a.secret.key": "bbb",
            "spark.hadoop.fs.s3a.session.token": "ccc",
        },
    ),

yes, when you run locally, flytekit submits a databricks job

No. spark job on k8s is when it’s run remotely.

yes, that’s exactly what I did in that PR

Awesome, thank you @Kevin Su!

pyflyte run spark.py wf  # Run spark in local python process
pyflyte run --raw-output-data-prefix <s3://saprk/output> spark.py wf  # We serialize the input and upload to s3, and submit the databricks job.

is this command confused to you? or should we add new flag, like pyflyte run --hybrid, --agent, or something else

@Georgi Ivanov did you get a chance to try this^ what do you think?

Sorry I was away. I will check it today