Hi community I m trying to run a dummy python job on a datab Flyte #ask-the-community

Hi community! I'm trying to run a dummy python jo...

Robert Ambrus

06/19/2023, 2:06 PM

Hi community! I'm trying to run a dummy python job on a 🧱 job cluster (_new_cluster_) using

pyflyte

. For some reason, when I add

policy_id

to the cluster config, the Flyte job fails with an error like this:

Kevin Su

06/19/2023, 11:16 PM

the
Databricks Console
link looks like this:

is that ink correct? are you able to open it?

Robert Ambrus

06/20/2023, 11:32 AM

no, it's an invalid link, the run id is

nil

Robert Ambrus

06/20/2023, 11:33 AM

I guess the request is not even submitted to Databricks, let me check the audit logs

Robert Ambrus

06/20/2023, 12:28 PM

ok, I think I found the root cause... When we are submitting a valid job run, 🧱 API returns:

HTTP 200

Copy code

{
    "run_id": <valid_run_id>
}

Robert Ambrus

06/20/2023, 12:31 PM

When we add a cluster policy to the cluster config and the cluster config conflicts with the policy, 🧱 API returns:

HTTP 400

Copy code

{
  "error_code": "INVALID_PARAMETER_VALUE",
  "message": "Cluster validation error: The instance profile arn (arn:aws:iam::<account_id>:instance-profile/<instance_profile>) has been removed from Databricks, Please contact your administrator."
}

Robert Ambrus

06/20/2023, 12:32 PM

I think this error case is not handled properly in Flyte, probably it's still trying to parse the

run_id

from the response, that's why it returns an url with

nil

run_id.

Robert Ambrus

06/20/2023, 12:34 PM

I don't think this is a big problem, but it would be nice to see the above error message displayed in the

Flyte

UI.

Robert Ambrus

06/20/2023, 12:37 PM

Probably the error can be found in pod logs via

kubectl

, but that's available only for admins as far as I know.

170 Views

Open in Slack

Previous Next