• Pradithya Aria Pura

    Pradithya Aria Pura

    9 months ago
    We are experiencing memory leak in flyteadmin pod, the instance is not yet used but the memory consumption is continuously building up. It doesn’t happen with other components (propeller, scheduler, or datacatalog). Anyone has same issue?
    Pradithya Aria Pura
    p
    2 replies
    Copy to Clipboard
  • Emirhan Karagül

    Emirhan Karagül

    9 months ago
    Hey everybody, I'm back with yet another question. This week we realized that deploying flyte to gcp using the manual deployment guide, caching does not work. We had some cached tasks working as expected in the sandbox deployment, but it doesn't retrieve the values from cache anymore. We also realized that the UI doesn't show the inputs&outputs of the tasks anymore: "No data is available." for all the tasks and workflow. Are these issues somehow related? Thank you!
    Emirhan Karagül
    y
    +3
    12 replies
    Copy to Clipboard
  • Pradithya Aria Pura

    Pradithya Aria Pura

    8 months ago
    Is there a way to invalidate auth token? We have 2 separate environment and use google auth. Currently, if users switch environment, they will have authentication issue
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-01-05T11:44:53+08:00"}
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [root] updated. No update handler registered.","ts":"2022-01-05T11:44:53+08:00"}
    {"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-01-05T11:44:53+08:00"}
    {"json":{"src":"auth_flow_orchestrator.go:37"},"level":"debug","msg":"got a response from the refresh grant for old expiry 2022-01-05 11:54:53.262787 +0800 +08 with new expiry 2022-01-05 11:54:53.262787 +0800 +08","ts":"2022-01-05T11:44:54+08:00"}
    {"json":{"src":"client.go:54"},"level":"info","msg":"Initialized Admin client","ts":"2022-01-05T11:44:54+08:00"}
    Launch plan plan_scorer_data_pipeline.workflows.launchplan.plan_scorer_pipeline_workflow_schedule failed to get updated due to rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken
    Error: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken
    {"json":{"src":"main.go:13"},"level":"error","msg":"rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2022-01-05T11:44:54+08:00"}
    Any workaround for this?
    Pradithya Aria Pura
    Yee
    +1
    9 replies
    Copy to Clipboard
  • JP Kosymna

    JP Kosymna

    8 months ago
    I'm having some trouble configuring a new installation of flyte, but I think I'm close. I installed flyte with opta using the documentation here https://docs.flyte.org/en/latest/deployment/aws/opta.html#deployment-aws-opta I'm using an external-ssl-cert and am able to access the flyte console. My two problems: 1. I don't think I am using the correct endpoint for flytectl. I thought it should be the subdomain I access the console through but that didn't work. After trying a few things I was able to get it working by pointing flytectl directly to the flyteadmin service's load balancer on port 81. I used 
    kubectl -n flyte get services flyteadmin
     to find it. Is this the correct way to do it? 2. I am having trouble configuring authentication with google cloud. Using https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html#deployment-cluster-config-auth-setup I did the following+ Setup my google cloud OAuth2 Client Credential + Ran 
    kubectl edit secret -n flyte flyte-admin-secrets
     and added the client secret+ Ran 
    kubectl edit configmap -n flyte flyte-admin-config
     updated the config according to the docs+ Restarted flyteadmin with 
    kubectl rollout restart deployment/flyteadmin -n flyte
    I didn't get everything wrong because when I visited the flyte console it redirected me to google to login before going to the dashboard. However when I tried to run a workflow the new execution just hung with status unknown. I also was unable to connect with flytectl no matter what I tried. I'm not sure what I'm doing wrong here. Any help is much appreciated.
    JP Kosymna
    Ketan (kumare3)
    +4
    91 replies
    Copy to Clipboard
  • Ketan (kumare3)

    Ketan (kumare3)

    8 months ago
    @Nitin Aggarwal / @JD Palomino how difficult would it be to target azure using opta for Flyte - cc @Emirhan Karagül
    Ketan (kumare3)
    Emirhan Karagül
    +1
    5 replies
    Copy to Clipboard
  • Nicholas LoFaso

    Nicholas LoFaso

    8 months ago
    Hi is there a preferred approach to deploy Flyte to GKE (Helm or Kustomize)? Currently we’re using an older version of the Flyte helm chart. I tried to upgrade to the latest, but I run into two issues. Before I dig too deeply into fixing it I thought I might switch to kustomize if that is the preferred method
    Nicholas LoFaso
    Ketan (kumare3)
    +1
    5 replies
    Copy to Clipboard
  • Arief Rahmansyah

    Arief Rahmansyah

    8 months ago
    Hi, in Flyte 0.18.2 and 0.19.0, the pytorch-operator is got deleted from the chart. Is it expected? I cannot find it in the changelog and Slack.
    Arief Rahmansyah
    Ketan (kumare3)
    +3
    13 replies
    Copy to Clipboard
  • n

    ncouronne

    8 months ago
    Hi, i'm trying to configure a secret to be able to pull our docker image from a private registry when starting a workflow. Ie i'd like to configure the
    imagePullSecrets
    field when starting pods. I've look at the secrets docs but i'm not sure this is what i'm looking for. As i understand, this will provide secret inside the tasks so, during the execution of the tasks not during the schedule of the pods, am i right ? 1/ does the secret shall be global or one by namespace (ie flyte's project ?) 2/ can i configure this secret once for all workflows ? Thanks !
    n
    Ketan (kumare3)
    +1
    4 replies
    Copy to Clipboard
  • j

    jeev

    7 months ago
    are folks seeing memory allocations for tasks being squashed when working on the sandbox?
    j
    5 replies
    Copy to Clipboard
  • JP Kosymna

    JP Kosymna

    7 months ago
    I installed flyte v0.19.1 on aws using opta, but I can't seem to get logging to work. I created a logGroup in aws named
    flyte-prod-tasks-logs
    and updated the
    values-eks.yaml
    file accordingly. In
    opta/aws/flyte.yaml
    I added the following.
    task_logs:
        plugins:
            logs:
                cloudwatch-enabled: true
                cloudwatch-log-group: flyte-prod-tasks-logs
                cloudwatch-region: "{vars.region}"
    My workflow runs successfully but I don't see anything in the log group. My ShellTask prints to standard out and I have a PythonTask that uses logging. Is there anything special I need to do? Or some configuration I missed?
    JP Kosymna
    Ketan (kumare3)
    +4
    43 replies
    Copy to Clipboard