https://flyte.org logo
Powered by
s

seunggs

09/23/2022, 10:44 PM
Copy code
{
  project: "shelly-robotics-bidepal-robot",
  domain: "development",
  name: "my-project.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1",
  spec: {
    launch_plan: {
      project: "shelly-robotics-bidepal-robot",
      domain: "development",
      name: "my-project.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1",
      version: "0.0.1",
    },
    inputs: {
      x: 1,
    },
  },
};
y

Yee

09/23/2022, 10:56 PM
do you have any more errors?
any stack traces/line numbers etc. would be helpful
s

seunggs

09/23/2022, 10:58 PM
This is all I got (it’s the response I get from the browser when I hit the /executions endpoint):
Copy code
{
  "error": "size of name exceeded length 20 : my_project.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1",
  "code": 3,
  "message": "size of name exceeded length 20 : my_project.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1"
}
Exact same workflow seems to work fine when I execute it from the dashboard ui so it looks like flyte’s rest api validation?
y

Yee

09/23/2022, 11:12 PM
can you check admin logs to see if there’s any more information?
s

seunggs

09/23/2022, 11:12 PM
So flyteadmin pod logs?
y

Yee

09/23/2022, 11:13 PM
yeah
s

seunggs

09/23/2022, 11:15 PM
I think these are it:
Copy code
2022/09/23 23:13:39 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/execution_repo.go:103
[1.064ms] [rows:9] SELECT "executions"."id","executions"."created_at","executions"."updated_at","executions"."deleted_at","executions"."execution_project","executions"."execution_domain","executions"."execution_name","executions"."launch_plan_id","executions"."workflow_id","executions"."task_id","executions"."phase","executions"."closure","executions"."spec","executions"."started_at","executions"."execution_created_at","executions"."execution_updated_at","executions"."duration","executions"."abort_cause","executions"."mode","executions"."source_execution_id","executions"."parent_node_execution_id","executions"."cluster","executions"."inputs_uri","executions"."user_inputs_uri","executions"."error_kind","executions"."error_code","executions"."user","executions"."state" FROM "executions" INNER JOIN workflows ON executions.workflow_id = workflows.id WHERE executions.execution_project = 'shelly-robotics-bipedal-robot' AND executions.execution_domain = 'development' AND (workflows.name = 'sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1') AND executions.state = 'EXECUTION_ACTIVE' ORDER BY created_at desc LIMIT 100

2022/09/23 23:13:39 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/workflow_repo.go:44 record not found
[1.267ms] [rows:0] SELECT * FROM "workflows" WHERE "workflows"."project" = 'shelly-robotics-bipedal-robot' AND "workflows"."domain" = 'development' AND "workflows"."name" = 'sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1' AND "workflows"."version" = 'draft' LIMIT 1
{"json":{"src":"workflow_manager.go:222","wf":"sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1"},"level":"info","msg":"Failed to get workflow with id [resource_type:WORKFLOW project:\"shelly-robotics-bipedal-robot\" domain:\"development\" name:\"sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1\" version:\"draft\" ] with err missing entity of type WORKFLOW with identifier project:\"shelly-robotics-bipedal-robot\" domain:\"development\" name:\"sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1\" version:\"draft\" ","ts":"2022-09-23T23:13:39Z"}
{"json":{"src":"workflow.go:43"},"level":"info","msg":"Adding resource type for unspecified value in request: [id:\u003cproject:\"shelly-robotics-bipedal-robot\" domain:\"development\" name:\"sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1\" version:\"0.0.18\" \u003e ]","ts":"2022-09-23T23:13:40Z"}

2022/09/23 23:13:40 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/workflow_repo.go:44
[0.571ms] [rows:1] SELECT * FROM "workflows" WHERE "workflows"."project" = 'shelly-robotics-bipedal-robot' AND "workflows"."domain" = 'development' AND "workflows"."name" = 'sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1' AND "workflows"."version" = '0.0.18' LIMIT 1
{"json":{"src":"execution_manager.go:818"},"level":"debug","msg":"Failed to validate ExecutionCreateRequest {Project:shelly-robotics-bipedal-robot Domain:development Name:sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1 Spec:launch_plan:\u003cproject:\"shelly-robotics-bipedal-robot\" domain:\"development\" name:\"sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1\" version:\"0.0.18\" \u003e inputs:\u003c\u003e security_context:\u003crun_as:\u003ck8s_service_account:\"flyte-executor\" \u003e \u003e  Inputs: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} with err size of name exceeded length 20 : sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1","ts":"2022-09-23T23:13:44Z"}

2022/09/23 23:13:44 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/execution_repo.go:103
[1.265ms] [rows:9] SELECT "executions"."id","executions"."created_at","executions"."updated_at","executions"."deleted_at","executions"."execution_project","executions"."execution_domain","executions"."execution_name","executions"."launch_plan_id","executions"."workflow_id","executions"."task_id","executions"."phase","executions"."closure","executions"."spec","executions"."started_at","executions"."execution_created_at","executions"."execution_updated_at","executions"."duration","executions"."abort_cause","executions"."mode","executions"."source_execution_id","executions"."parent_node_execution_id","executions"."cluster","executions"."inputs_uri","executions"."user_inputs_uri","executions"."error_kind","executions"."error_code","executions"."user","executions"."state" FROM "executions" INNER JOIN workflows ON executions.workflow_id = workflows.id WHERE executions.execution_project = 'shelly-robotics-bipedal-robot' AND executions.execution_domain = 'development' AND (workflows.name = 'sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1') AND executions.state = 'EXECUTION_ACTIVE' ORDER BY created_at desc LIMIT 100
y

Yee

09/23/2022, 11:16 PM
that’s it?
s

seunggs

09/23/2022, 11:16 PM
Based on the timestamp these are generated when I hit the /executions endpoint
There’s a lot more but I think that’s from before
y

Yee

09/23/2022, 11:17 PM
Copy code
kubectl -n flyte logs <pod> | egrep -B 2 -A 2 "size of name"
does that give anything?
s

seunggs

09/23/2022, 11:18 PM
Yes:
Copy code
2022/09/23 23:13:40 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/workflow_repo.go:44
[0.571ms] [rows:1] SELECT * FROM "workflows" WHERE "workflows"."project" = 'shelly-robotics-bipedal-robot' AND "workflows"."domain" = 'development' AND "workflows"."name" = 'sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1' AND "workflows"."version" = '0.0.18' LIMIT 1
{"json":{"src":"execution_manager.go:818"},"level":"debug","msg":"Failed to validate ExecutionCreateRequest {Project:shelly-robotics-bipedal-robot Domain:development Name:sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1 Spec:launch_plan:\u003cproject:\"shelly-robotics-bipedal-robot\" domain:\"development\" name:\"sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1\" version:\"0.0.18\" \u003e inputs:\u003c\u003e security_context:\u003crun_as:\u003ck8s_service_account:\"flyte-executor\" \u003e \u003e  Inputs: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} with err size of name exceeded length 20 : sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1","ts":"2022-09-23T23:13:44Z"}

2022/09/23 23:13:44 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/execution_repo.go:103
So it looks like this is the error: 
Inputs: XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} with err size of name exceeded length 20 : sidetrek_generated.workflows.04b2ed0b-434b-4d7b-9220-90606fdc2afc.test_wf_1
Also looks like my inputs field format might be wrong?
k

Kevin Su

09/24/2022, 5:31 AM
s

seunggs

09/24/2022, 3:11 PM
Thanks @Kevin Su ! But I thought I were using the latest helm chart already (v1.2.0-b3)? Not sure which helm version is linked to this flyteadmin version?
k

Kevin Su

09/24/2022, 4:16 PM
s

seunggs

09/24/2022, 8:33 PM
OK thanks - but I do have automation setup so just to confirm, this is not released in the helm chart yet?
k

Kevin Su

09/24/2022, 9:19 PM
yes, but we’ll release v1.2.0 soon, and it will include that patch.
s

seunggs

09/25/2022, 3:25 AM
OK awesome - thank you!
@Kevin Su - sorry to keep bugging you, but how do I update the pod image? I was going to just 
kubectl edit deployment flyteadmin
and then update the spec.containers.image to 
<http://ghcr.io/flyteorg/flyteadmin:v1.1.43|ghcr.io/flyteorg/flyteadmin:v1.1.43>
and then delete the flyteadmin pod. Is this correct?
Also, I see multiple references to 
<http://cr.flyte.org/flyteorg/flyteadmin-release:v1.2.0-b3|cr.flyte.org/flyteorg/flyteadmin-release:v1.2.0-b3>
in the flyteadmin deployment yaml file (strangely missing “g” in front of “cr.flyte.org”), for example in initContainers. Is it necessary to update all references to the old version?
k

Kevin Su

09/25/2022, 6:53 AM
1. You don’t need to delete the pod manually. k8s will help you delete the older version of the pod. 2. In this case, you could only update first container image
Copy code
containers:
      - command:
        - flyteadmin
        - --config
        - /etc/flyte/config/*.yaml
        - serve
        image: flyteorg/flyteadmin-release:v1.2.0-b3 # <- only need to update this image
        imagePullPolicy: IfNotPresent
        name: flyteadmin
        ports:
s

seunggs

09/26/2022, 4:15 PM
@Kevin Su Thanks for this - it worked great. But I’m still getting an error: 
invalid name format: <http://sidetrek.wf.wf|sidetrek.wf.wf>_<http://1.my|1.my>_test_wf1
.
Since the name looks fine, I dug into the flyteadmin pod logs, and this is what I see:
Copy code
{
  "json": {
    "src": "execution_manager.go:818"
  },
  "level": "debug",
  "msg": "Failed to validate ExecutionCreateRequest {Project:shelly-robotics-bipedal-robot Domain:development Name:sidetrek.wf.wf_1.my_test_wf1 Spec:launch_plan:<project:\"shelly-robotics-bipedal-robot\" domain:\"development\" name:\"sidetrek.wf.wf_1.my_test_wf1\" version:\"0.0.3\" > security_context:<run_as:<k8s_service_account:\"flyte-executor\" > >  Inputs:literals:<key:\"scalar\" value:<> >  XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} with err invalid name format: sidetrek.wf.wf_1.my_test_wf1",
  "ts": "2022-09-26T16:14:34Z"
}
Looks like it’s related to inputs - maybe I’m sending it in the wrong format? I’m just trying to test a simple integer 
2
Here’s my payload for 
POST /executions
Copy code
{
  project: "shelly-robotics-bipedal-robot",
  domain: "development",
  name: "sidetrek.wf.wf_1.my_test_wf1",
  inputs: {
    literals: {
      scalar: {
        primitive: {
          integer: {
            x: 2,
          },
        },
      },
    },
  },
  spec: {
    launch_plan: {
      project: "shelly-robotics-bipedal-robot",
      domain: "development",
      name: "sidetrek.wf.wf_1.my_test_wf1",
      version: "0.0.3",
    },
    security_context: {
      run_as: {
        k8s_service_account: "flyte-executor",
      },
    },
  },
};
k

Kevin Su

09/26/2022, 4:26 PM
The name format is incorrect. could you change the name from 
<http://sidetrek.wf.wf|sidetrek.wf.wf>_<http://1.my|1.my>_test_wf1
to 
test-wf
. IIRC, we can’t use underscore in the name
s

seunggs

09/26/2022, 4:29 PM
Sorry - so you want me to change all underscores to hyphens? I was using this as wf name because I saw it in the console (
<http://sidetrek.wf.wf|sidetrek.wf.wf>_<http://1.my|1.my>_test_wf1
)
Also, if I change it to hyphen, I think it’s an invalid python package name?
My workflow is inside 
/sidetrek/wf/wf_1.py
with workflow function name 
my_test_wf1
Also, I see that the workflow name in the hello-world documentation is 
my_wf
so it looks like underscore is not the issue??
k

Kevin Su

09/26/2022, 4:34 PM
no, I mean execution name
Copy code
project: "shelly-robotics-bipedal-robot",
  domain: "development",
  name: "sidetrek.wf.wf_1.my_test_wf1", <- this one
  inputs: {
    literals: {
...
s

seunggs

09/26/2022, 4:35 PM
Ohhh
I see - so launch_plan name refers to the workflow name (since Flyte creates the same named launch plan by default) and the execution name is just anything I specify it to be?
k

Kevin Su

09/26/2022, 4:36 PM
yes, exactly
s

seunggs

09/26/2022, 4:36 PM
Ok got it - let me try that thank you!
OK looks like there’s a character limit on the execution name as well? uuidv4 doesn’t seem to work but 
test-wf
worked. But then there’s another error:
Copy code
Invalid reference entity resource type [UNSPECIFIED], only [map[TASK:<nil> LAUNCH_PLAN:<nil>]] allowed
Not sure what this means?
k

Kevin Su

09/26/2022, 4:55 PM
you have to specify resource type in the launchPlan spec.
Copy code
spec: {
    launch_plan: {
      project: "shelly-robotics-bipedal-robot",
      domain: "development",
      name: "sidetrek.wf.wf_1.my_test_wf1",
      version: "0.0.3",
      resource_type: 3 <- here, 3 is launch plan
    },
    security_context: {
      run_as: {
...
s

seunggs

09/26/2022, 4:56 PM
Oh ok - I missed that from the swagger docs
And I see that you can either put inputs outside the spec or inside spec - which one is correct?
k

Kevin Su

09/26/2022, 5:07 PM
I think inside is correct
s

seunggs

09/26/2022, 5:08 PM
OK got it - testing it out now
Sorry to keep bugging you, but new error: 
missing valid literal in inputs scalar
with this input:
Copy code
inputs: {
  literals: {
    scalar: {
      primitive: {
        integer: {
          x: "2",
        },
      },
    },
  },
},
k

Kevin Su

09/26/2022, 5:17 PM
Hmm, perhaps I’m wrong .maybe outside?
s

seunggs

09/26/2022, 5:17 PM
OK let me try that
Same error
Is my inputs format wrong?
In the swagger docs,
Copy code
"integer": {
          "type": "string",
          "format": "int64"
        },
k

Kevin Su

09/26/2022, 5:18 PM
one sec, let me check
s

seunggs

09/26/2022, 5:18 PM
OK thank you!!
k

Kevin Su

09/26/2022, 5:21 PM
I think it should be
Copy code
inputs: {
 literals {
   x: {
      scalar {
      primitive: {
        integer: {
          "2"
   ...
s

seunggs

09/26/2022, 5:22 PM
Hmm that doesn’t look like a correct js syntax
You mean this?
Copy code
inputs: {
  x: {
    scalar: {
      primitive: {
        integer: '2',
      },
    },
  },
},
k

Kevin Su

09/26/2022, 5:24 PM
sorry, updated it. yeah, something like that
s

seunggs

09/26/2022, 5:25 PM
OK running this gives me this error: 
expected_inputs x missing
k

Kevin Su

09/26/2022, 5:31 PM
mind sharing your payload again. let me test it on my server
s

seunggs

09/26/2022, 5:32 PM
This is the original payload:
Copy code
const x = {
  project: "shelly-robotics-bipedal-robot",
  domain: "development",
  name: "test-wf",
  spec: {
    inputs: {
      literals: {
        scalar: {
          primitive: {
            integer: {
              x: '2',
            },
          },
        },
      },
    },
    launch_plan: {
      project: "shelly-robotics-bipedal-robot",
      domain: "development",
      name: "sidetrek.wf.wf_1.my_test_wf1",
      version: "0.0.3",
      resource_type: 3,
    },
    security_context: {
      run_as: {
        k8s_service_account: "flyte-executor",
      },
    },
  },
};
You can disregard 
security_context
of course
Hey @Kevin Su I got it to work! It’s really easy to misread the swagger docs I guess:
Copy code
inputs: {
            literals: {
              x: {
                scalar: {
                  primitive: {
                    integer: '2',
                  },
                },
              },
            }
          },
This works
k

Kevin Su

09/26/2022, 5:46 PM
Oh, nice
s

seunggs

09/26/2022, 5:46 PM
Thank you for all your help!!
Really appreciate it
4 Views
Powered by