Babis Kiosidis
03/17/2022, 7:44 PM"error": {
"code": "USER:NotReady"
Ketan (kumare3)
Fabian Baier
03/22/2022, 8:47 PMlabel_values(flyte:propeller:all:collector:flyteworkflow, wf)
When I look at my propeller prom metrics however the wf
label is empty. Why?
# HELP flyte:propeller:all:collector:flyteworkflow Current FlyteWorkflow levels per instance of propeller
# TYPE flyte:propeller:all:collector:flyteworkflow gauge
flyte:propeller:all:collector:flyteworkflow{domain="development",project="testproject",task="",wf=""} 1
Fabian Baier
03/22/2022, 8:51 PMflyte:propeller:all:workflow:accepted
is not empty. So if I want a dropdown of my workflows is this metric actually the right on to query for the workflow
variable?Fabian Baier
03/22/2022, 8:52 PMdomain
as that is another query variable to select as a step befor the workflow selection (otherwise you would see all the wf rather then the one from your domain)Hampus Rosvall
03/28/2022, 1:22 PMpsql -U flyteadmin --host=<RDS_WRITER_ENDPOINT> --dbname=flyteadmin --port=5432
psql: FATAL: password authentication failed for user "flyteadmin"
FATAL: password authentication failed for user "flyteadmin"
Hampus Rosvall
03/30/2022, 6:55 AMFlyteAdmin
Pod in order to perform registration and request executions. I tried to port-forward to the FlyteConsole
Service as well as Pod to access the UI but I am getting the following error, shouldnāt this be possible as well?Brian Tang
04/06/2022, 10:09 PMAnna Cunningham
04/13/2022, 7:41 PMkatrina
Brian Tang
04/28/2022, 9:50 PMcache_version
, task signature, or inputs change. This aligns with the example provided:
In the above example, calling square(n=2) twice (even if itās across different executions or different workflows) will only execute the multiplication operation once.ā¢ However, in my testing ā caching a task and then making a change to a different task in the workflow causes the ācachedā task to be recomputed. The code shows the hashed key contains a
core.Identifier
that has the task version in it. Logs are also showing the same, which would mean each iteration on the workflow would recompute the ācachedā task:
"Successfully cached results to catalog - Task [resource_type:TASK project:\"fraud-intelligence\" domain:\"development\" name:\"src.python.flyte.fraud_intelligence.training_set.main.filter_and_label\" version:\"e8758db3b7a1e6fffbb0bb73c742310acd7e774f\" ]"
Are the docs just outdated? If the implementation is correct, how are cached tasks expected to be reused when iterating on a workflow? Is the only way to reuse a cached task through a reference task?Hampus Rosvall
05/09/2022, 5:46 PMRoberto Ruiz
05/18/2022, 8:27 PMflytectl register examples -d development -p flytesnacks
I'm getting:
Error: example 0xc0006ae770 failed to register rpc error: code = Internal desc = failed to write marshaled workflow [resource_type:WORKFLOW project:"flytesnacks" domain:"development" name:"blast.blastx_example.blast_wf" version:"v0.3.81" ] to storage <s3://flyte-cluster-bucket/metadata/admin/flytesnacks/development/blast.blastx_example.blast_wf/v0.3.81> with err Failed to write data [3170b] to path [metadata/admin/flytesnacks/development/blast.blastx_example.blast_wf/v0.3.81].: PutObject, putting object: WebIdentityErr: failed to retrieve credentials
caused by: ValidationError: Request ARN is invalid
But I can't seem to find which ARN this error is referring toMatthew Krueger
05/24/2022, 8:21 PMpyflyte
I am getting the following error:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
I have been double checking all the SAs and everything seems to be in order but obviously I am missing something. Any pointers would be appreciated!Babis Kiosidis
05/25/2022, 7:05 PMdocker run --rm --privileged -p 30081:30081 -p 30082:30082 -p 30084:30084 <http://cr.flyte.org/flyteorg/flyte-sandbox:dind|cr.flyte.org/flyteorg/flyte-sandbox:dind>
It fails with an error:
"flyteorg" has been added to your repositories
Deploying Flyte-deps...
Release "flyte-deps" does not exist. Installing it now.
Error: failed to download "flyteorg/flyte-deps" at version "v1.0.2-b1" (hint: running `helm repo update` may help)
Ketan (kumare3)
Emirhan KaragĆ¼l
05/30/2022, 4:21 PMflyte-core
deployments from v0.18.2
to v1.0.1
. So far I didn't have any issues with regular tasks. But when I'm trying to execute spark tasks, I see that the default service account is assigned to the spark tasks. Is there something I need to change when upgrading to 1.+ versions? For context, I am using flytekit
0.26.1.
Thanks in advance!Shihgian Lee
06/02/2022, 7:20 PMHaytham Abuelfutuh
Brian Tang
06/06/2022, 5:36 PM$ kubectl get configmaps -n flyte flyte-propeller-config -oyaml
apiVersion: v1
data:
...
task_logs.yaml: |
plugins:
logs:
templates:
- displayName: Splunk
templateUris:
- "<https://splunk.corp.stripe.com/en-US/app/search/search?q=search%20source=>"/pay/log/pods/{{.namespace }}_{{.podName }}*"%20%7C%20reverse&display.events.type=raw&display.prefs.events.count=50&earliest=-7d&latest=now"
messageFormat: "json"
kubernetes-enabled: true
kubernetes-url: "<http://localhost:8001>"
Any pointers on what I might be missing?Edgar Trujillo
06/21/2022, 3:32 PMaccountNumber
as a float in all annotation related lines in values-eks.yaml
<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/iam-role-flyte
I changed the lines to explicitly define the arn and can now schedule executions to run remotely
But the console shows no actual executions and the role assigned is the default
rather than 1 of the 2 roles created (iam-role-flyte
, flyte-user-role
)
Doing a describe on the flytepropeller shows
Environment:
POD_NAME: flytepropeller-7d67df4c85-7sf4v (v1:metadata.name)
POD_NAMESPACE: flyte (v1:metadata.namespace)
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: us-east-1
AWS_REGION: us-east-1
AWS_ROLE_ARN: arn:aws:iam::1.AWS_ACCTe+11:role/iam-role-flyte
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Any idea where the AWS_ROLE_ARN
is being mounted from?Shahwar Saleem
06/23/2022, 5:50 PMcore.yaml
. I am particularly interested in this line: https://github.com/flyteorg/flyte/blob/ca704e24daad628535a4414e0a7cb1164621105d/charts/flyte-core/values.yaml#L599
I believe it should be namespace: {{ namespace }}
rather than hard coded flyte
. Please correct me if I am wrong.Shahwar Saleem
06/23/2022, 9:43 PMrror: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Ingress.spec.tls[0].hosts): unknown object type "nil" in Ingress.spec.tls[0].hosts[0]
When I render the template for helm I see:
tls:
- secretName: my-secret-name
hosts:
- null
Why is it generating this hosts as null? How can I override/correct this hosts as there is no option to do that in the values-eks.yaml.Vinicius EsperanƧa
06/27/2022, 8:51 AMGET <https://flyte.sidetrek.com/me> 501 (Not Implemented)
. This shows on browser console as well. If I try to curl
, I receive the following:
HTTP/1.1 501 Not Implemented
content-type: application/json
date: Mon, 27 Jun 2022 08:50:07 GMT
content-length: 131
x-envoy-upstream-service-time: 0
server: envoy
{"error":"unknown service flyteidl.service.IdentityService","code":12,"message":"unknown service flyteidl.service.IdentityService"}%
the final one is: if I go to <https://flyte.sidetrek.com/login>
, it returns Not Found
on the page.
any help on this?Edgar Trujillo
06/27/2022, 9:56 PMlog
plugin is defined as well
task_logs.yaml: |
plugins:
logs:
cloudwatch-enabled: true
cloudwatch-log-group: 'bv-ml-pipelines'
cloudwatch-region: 'us-east-1'
kubernetes-enabled: false
krishna Yerramsetty
06/28/2022, 9:30 PM@task(requests=Resources(cpu="8"))
But when I try registering the files I get the following error:
Requested CPU default [8] is greater than current limit set in the platform configuration [2]
How and where do I change the default platform configuration? I am running on an AWS EKS cluster with a minimum compute group node size of 2, desired size of 2, and maximum of 24. Each node is a 16 cpu EC2 instanceMatheus Moreno
06/30/2022, 7:04 PMShahwar Saleem
07/06/2022, 5:44 PMFailed to pull image "<http://cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2|cr.flyte.org/flyteorg/flyteadmin-release:v1.0.2>": rpc error: code = Unknown desc = Error response from daemon: manifest unknown
Katrina P
07/06/2022, 6:02 PMflyte-pod-webook
won't come up due to: Failed to start webhook. Error: open /etc/webhook/certs/tls.crt: no such file or directory.
Anyone have some insight into this?Shahwar Saleem
07/06/2022, 7:04 PM