• Brian Tang

    Brian Tang

    4 months ago
    👋 I’m seeing a discrepancy in Flyte datacatalog between the docs and my testing: • In the docs — it says the cache is only invalidated if the
    cache_version
    , task signature, or inputs change. This aligns with the example provided:
    In the above example, calling square(n=2) twice (even if it’s across different executions or different workflows) will only execute the multiplication operation once.
    • However, in my testing — caching a task and then making a change to a different task in the workflow causes the “cached” task to be recomputed. The code shows the hashed key contains a
    core.Identifier
    that has the task version in it. Logs are also showing the same, which would mean each iteration on the workflow would recompute the “cached” task:
    "Successfully cached results to catalog - Task [resource_type:TASK project:\"fraud-intelligence\" domain:\"development\" name:\"src.python.flyte.fraud_intelligence.training_set.main.filter_and_label\" version:\"e8758db3b7a1e6fffbb0bb73c742310acd7e774f\" ]"
    Are the docs just outdated? If the implementation is correct, how are cached tasks expected to be reused when iterating on a workflow? Is the only way to reuse a cached task through a reference task?
    Brian Tang
    Yee
    +2
    24 replies
    Copy to Clipboard
  • Hampus Rosvall

    Hampus Rosvall

    4 months ago
    Hey, as an extra layer of security I want to authenticate users in the ALB by adding oidc Ingress annotations. How would I configure flytectlto still be able to authenticate users when the OIDC flow happens in the ALB rather than in the Flyte Console? On another note, is it possible to deploy Flyte internally and still be able to authenticate users/deploy workflows in CI/CD? I guess the topics are quite different, but it would be interesting to see what choice you prefer and how that is implemented in practice. Thanks 🙂
    Hampus Rosvall
    Ketan (kumare3)
    +1
    20 replies
    Copy to Clipboard
  • r

    Roberto Ruiz

    4 months ago
    Hi, I have a flyte cluster on AWS with the manual setup. I'm having trouble registering workflows, for example trying to register the examples with
    flytectl register examples -d development -p flytesnacks
    I'm getting:
    Error: example 0xc0006ae770 failed to register rpc error: code = Internal desc = failed to write marshaled workflow [resource_type:WORKFLOW project:"flytesnacks" domain:"development" name:"blast.blastx_example.blast_wf" version:"v0.3.81" ] to storage <s3://flyte-cluster-bucket/metadata/admin/flytesnacks/development/blast.blastx_example.blast_wf/v0.3.81> with err Failed to write data [3170b] to path [metadata/admin/flytesnacks/development/blast.blastx_example.blast_wf/v0.3.81].: PutObject, putting object: WebIdentityErr: failed to retrieve credentials
    caused by: ValidationError: Request ARN is invalid
    But I can't seem to find which ARN this error is referring to
    r
    k
    17 replies
    Copy to Clipboard
  • Matthew Krueger

    Matthew Krueger

    4 months ago
    Hello! I have a deployment up and running (mostly?) in AWS right now. However, when I am trying to run something with
    pyflyte
    I am getting the following error:
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
            status = StatusCode.UNKNOWN
            details = "failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials
    caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
    I have been double checking all the SAs and everything seems to be in order but obviously I am missing something. Any pointers would be appreciated!
    Matthew Krueger
    r
    +1
    6 replies
    Copy to Clipboard
  • b

    Babis Kiosidis

    4 months ago
    Hey friends! When I try to run the sandbox with:
    docker run --rm --privileged -p 30081:30081 -p 30082:30082 -p 30084:30084 <http://cr.flyte.org/flyteorg/flyte-sandbox:dind|cr.flyte.org/flyteorg/flyte-sandbox:dind>
    It fails with an error:
    "flyteorg" has been added to your repositories
    Deploying Flyte-deps...
    Release "flyte-deps" does not exist. Installing it now.
    Error: failed to download "flyteorg/flyte-deps" at version "v1.0.2-b1" (hint: running `helm repo update` may help)
    b
    k
    4 replies
    Copy to Clipboard
  • Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    Attila Nagy [12:27 PM] BTW, is there an easy way to set up an authentication without any external providers (like fixed secrets)?
    Ketan (kumare3)
    3 replies
    Copy to Clipboard
  • Emirhan Karagül

    Emirhan Karagül

    3 months ago
    Hi all, I am back with yet another question about spark. I am trying to upgrade our
    flyte-core
    deployments from
    v0.18.2
    to
    v1.0.1
    . So far I didn't have any issues with regular tasks. But when I'm trying to execute spark tasks, I see that the default service account is assigned to the spark tasks. Is there something I need to change when upgrading to 1.+ versions? For context, I am using
    flytekit
    0.26.1. Thanks in advance!
    Emirhan Karagül
    p
    +1
    18 replies
    Copy to Clipboard
  • Shihgian Lee

    Shihgian Lee

    3 months ago
    Hi All, we are deploying flyte to our kubernetes cluster. We are at the SSL Certificate step. We don’t want to expose flyte to outside of our organization. But, it looks like the SSL Certificate for GKE is only for external HTTPS. Are there other options we can use so that we don’t expose flyte to outside world? Appreciate any helps from the community.
    Shihgian Lee
    Ketan (kumare3)
    +2
    11 replies
    Copy to Clipboard
  • Haytham Abuelfutuh

    Haytham Abuelfutuh

    3 months ago
    @austin
  • Brian Tang

    Brian Tang

    3 months ago
    Hi — I’m trying to configure log links for task executions, but after adding the config section, I’m still not able to see the templatized and or kubernetes log links:
    $ kubectl get configmaps -n flyte flyte-propeller-config -oyaml
    apiVersion: v1
    data:
    ...
      task_logs.yaml: |
        plugins:
          logs:
            templates:
            - displayName: Splunk
              templateUris:
                - "<https://splunk.corp.stripe.com/en-US/app/search/search?q=search%20source=>"/pay/log/pods/{{.namespace }}_{{.podName }}*"%20%7C%20reverse&display.events.type=raw&display.prefs.events.count=50&earliest=-7d&latest=now"
              messageFormat: "json"
            kubernetes-enabled: true
            kubernetes-url: "<http://localhost:8001>"
    Any pointers on what I might be missing?