astonishing-lizard-78628
06/04/2022, 3:36 AMflytectl create execution
is fine, but now scheduled launch plans ignore the registered k8s service account and run as default
. My scheduled launch plan looks ok:
FA21110289:avexampleworkflows alex.bain$ cat lp.json
{
"id": {
"resourceType": "LAUNCH_PLAN",
"project": "avexampleworkflows",
"domain": "dev",
"name": "schedule_fabrik_parent_workflow_101",
"version": "31599fdeaaf34476ba54155426ae0707709300bb"
},
"spec": {
"workflowId": {
"resourceType": "WORKFLOW",
"project": "avexampleworkflows",
"domain": "dev",
"name": "app.workflows.fabrik.fabrik_parent_workflow.fabrik_parent_workflow",
"version": "31599fdeaaf34476ba54155426ae0707709300bb"
},
"entityMetadata": {
"schedule": {
"cronExpression": "*/15 * * * ? *"
}
},
"defaultInputs": {},
"fixedInputs": {},
"labels": {},
"annotations": {},
"authRole": {
"kubernetesServiceAccount": "avexampleworkflows"
},
"rawOutputDataConfig": {
"outputLocationPrefix": "<s3://lyft-av-prod-pdx-flyte/raw_data>"
}
},
"closure": {
"state": "ACTIVE",
"expectedInputs": {},
"expectedOutputs": {},
"createdAt": "2022-06-04T03:01:52.526120Z",
"updatedAt": "2022-06-04T03:01:52.526120Z"
}
}
I kind of thought there would be a securityContext
entry in the scheduled launch plan, but I don't see one. Anyways, I created this ^^^^^^ with:
schedule_fabrik_parent_workflow_101 = flytekit.LaunchPlan.get_or_create(
name="schedule_fabrik_parent_workflow_101",
schedule=flytekit.CronSchedule(cron_expression="*/15 * * * ? *"),
workflow=fabrik_parent_workflow,
)
All of this was registered with flytectl register files --k8sServiceAccount avexampleworkflows
. Then I did flytectl update launchplan --admin.endpoint <http://avflyteadmin.scratch-alexbain.dev|avflyteadmin.scratch-alexbain.dev>.l5.woven-planet.tech:443 -p avexampleworkflows -d dev schedule_fabrik_parent_workflow_101 --version 31599fdeaaf34476ba54155426ae0707709300bb --activate
to activate this launch plan (so that it would start executing every 15 minutes.
Ah, here is the execution. You can see that the securityContext
is broken in the execution. This is definitely a bug then as it ignored the declared kubernetesServiceAccount
on the actual launch plan. I ran `flytectl get execution --admin.endpoint avflyteadmin.scratch-alexbain.dev.l5.woven-planet.tech:443 -p avexampleworkflows -d dev at89nsj2879cfl6l2cbf -o json`:
{
"id": {
"project": "avexampleworkflows",
"domain": "dev",
"name": "at89nsj2879cfl6l2cbf"
},
"spec": {
"launchPlan": {
"resourceType": "LAUNCH_PLAN",
"project": "avexampleworkflows",
"domain": "dev",
"name": "schedule_fabrik_parent_workflow_101",
"version": "31599fdeaaf34476ba54155426ae0707709300bb"
},
"metadata": {
"mode": "SCHEDULED",
"scheduledAt": "2022-06-04T03:15:00Z",
"systemMetadata": {}
},
"securityContext": {
"runAs": {
"k8sServiceAccount": "default"
}
}
},
"closure": {
"error": {
"code": "RetriesExhausted|UnknownError",
"message": "[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.\r\n[at89nsj2879cfl6l2cbf-n0-0-n0-0] terminated with exit code (2). Reason [Error]. Message: \nation: '\n+ echo 'L5 data center: pdx'\n+ echo 'L5 domain: dev.l5.woven-planet.tech'\n+ echo 'L5 cluster name: scratch-alexbain'\nIn the avcontainers flyte-spark-entrypoint script\nL5 application: \nL5 data center: pdx\nL5 domain: dev.l5.woven-planet.tech\nL5 cluster name: scratch-alexbain\nL5 namespace: dev\n[INFO] Vault address: <https://vault.pdx.dev.l5.woven-planet.tech>\n[INFO] Flyte internal domain: dev\n[INFO] Flyte internal project: avexampleworkflows\n[INFO] Flyte internal execution project: avexampleworkflows\n+ echo 'L5 namespace: dev'\n+ export VAULT_ADDR=<https://vault.pdx.dev.l5.woven-planet.tech>\n+ VAULT_ADDR=<https://vault.pdx.dev.l5.woven-planet.tech>\n+ echo '[INFO] Vault address: <https://vault.pdx.dev.l5.woven-planet.tech>'\n+ echo '[INFO] Flyte internal domain: dev'\n+ echo '[INFO] Flyte internal project: avexampleworkflows'\n+ echo '[INFO] Flyte internal execution project: avexampleworkflows'\n+ role=avexampleworkflows-dev\n+ echo '[INFO] Role: avexampleworkflows-dev'\n+ SERVICE_ACCOUNT_TOKEN_PATH=/var/run/secrets/kubernetes.io/serviceaccount/token\n+ [[ ehxBE =~ x ]]\n+ debug=1\n+ set +x\n[INFO] Role: avexampleworkflows-dev\n+ AWS_SECRET_PREFIX=level5\n+ PROD_ENV_REGEX='[a-zA-Z]*prod[a-zA-Z]*'\n+ SCRATCH_ENV_REGEX='scratch[a-zA-Z]*'\n+ SECRET_PREFIX=level5\n+ [[ scratch-alexbain =~ scratch[a-zA-Z]* ]]\n+ AWS_SECRET_PREFIX=level5/scratch-alexbain\n+ SECRET_PREFIX=level5/scratch-alexbain\n+ [[ avexampleworkflows-dev =~ [a-zA-Z]*prod[a-zA-Z]* ]]\n+ AWS_SECRET_PREFIX=level5/scratch-alexbain/dev/aws/sts\n+ SECRET_PREFIX=level5/scratch-alexbain/dev/flyte/sts\n+ echo '[INFO] AWS secret prefix: level5/scratch-alexbain/dev/aws/sts'\n+ echo '[INFO] Secret prefix: level5/scratch-alexbain/dev/flyte/sts'\n+ set +x\n[INFO] AWS secret prefix: level5/scratch-alexbain/dev/aws/sts\n[INFO] Secret prefix: level5/scratch-alexbain/dev/flyte/sts\nError writing data to auth/scratch-alexbain/login: Error making API request.\n\nURL: PUT <https://vault.pdx.dev.l5.woven-planet.tech/v1/auth/scratch-alexbain/login>\nCode: 500. Errors:\n\n* service account name not authorized\n.",
"kind": "USER"
},
"phase": "FAILED",
"startedAt": "2022-06-04T03:15:13.228779416Z",
"duration": "49.957632481s",
"createdAt": "2022-06-04T03:15:12.947090574Z",
"updatedAt": "2022-06-04T03:16:03.186411481Z",
"workflowId": {
"resourceType": "WORKFLOW",
"project": "avexampleworkflows",
"domain": "dev",
"name": "app.workflows.fabrik.fabrik_parent_workflow.fabrik_parent_workflow",
"version": "31599fdeaaf34476ba54155426ae0707709300bb"
},
"stateChangeDetails": {
"occurredAt": "2022-06-04T03:15:12.947090574Z"
}
}
}
freezing-airport-6809
freezing-airport-6809
astonishing-lizard-78628
06/04/2022, 3:43 AMflytectl
fix... that's actually the one easy update we can make org wide!freezing-airport-6809
freezing-airport-6809
astonishing-lizard-78628
06/04/2022, 3:44 AMastonishing-lizard-78628
06/04/2022, 3:45 AMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
icy-agent-73298
06/04/2022, 5:17 AMicy-agent-73298
06/04/2022, 5:19 AMicy-agent-73298
06/04/2022, 4:47 PMfreezing-airport-6809
freezing-airport-6809
icy-agent-73298
06/04/2022, 4:50 PMicy-agent-73298
06/06/2022, 10:17 AM