I am trying to run a wandb.sweep as shown in this quickstart. The idea is to run the hyperparameter sweep, on a conditional, based on a boolean flag passed to a workflow and save the best model as an artifact. The main workflow is static, which calls a dynamic sweep_setup function. The wandb.agent calls a callback function hyperparameter_sweep which is also dynamic and which calls the train_model dynamic function. Is there an example of this somewhere or any best way to do this? Because, I am facing too many errors to get this to work. The code is a bit long to share, so first wanted to check if someone has done this successfully.

Hi, I’m trying to import wget in the sandbox via ‘import wget’, since it is not preinstalled, there is an error. How can I install wget?

Hi 🙂

(Caching) Hey all, I rewrote one of my company’s critical workflows in Flyte and it works well. However it frequently runs with multiple repetitive sets of inputs. For example, it may run 50 times a week with (mostly the) same 2-3 sets of inputs. From my understanding, cache_version only supports one version, so both can’t be cached at the same time. If I were to run a workflow with the first set of inputs, then the second, then the third, and repeated it 10 times, caching would effectively be disabled, because the cache keys keep changing, right? Is there any workaround or other solution to this problem? I’d also like to configure cache staleness so cache can expire after a while, I tried to achieve this using lifecycle rules on my cloud storage but in case the cloud deletes the cached outputs, Flyte still tries to fetch them and errors on it. Let me know how you dealt with this issue in your cases, thanks

Hello, I am trying to develop an 'interactive' workflow that waits for an external input. Flyte allows me to build this pipeline with the

wait_for_input

method. My question is, can I enable the users to upload a let say a json file through the

wait_for_input

method?

Hello, I am trying to configure auth across azure AD, I added the following as configuration:

auth:
    enabled: true
    oidc:
      baseUrl: "<https://login.microsoftonline.com/TENANT_ID/v2.0>"
      clientId: CLIENT_ID
      clientSecret: CLIENT_SECRET
      scopes:
        - openid
        - email
        - profile
    internal:
      clientSecret: CLIENT_SECRET
      clientSecretHash: CLIENT_SECRET_HASHED
    flyteClient:
      clientId: CLIENT_ID
      redirectUri: "<http://localhost:53593/callback>"
      scopes:
        - all
    authorizedUris:
    - <https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0>
    - <https://my.domain.com>

When trying to login I can walk through azure login step, but then receive a 403 from my.domain.com/callback?code=XXXX After an increase of the log level, I can see the following :

{"json":{"src":"handlers.go:238"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2023-06-27T14:31:08Z"}
{"json":{"src":"token.go:83"},"level":"debug","msg":"Could not retrieve bearer token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2023-06-27T14:31:08Z"}
{"json":{"src":"handlers.go:248"},"level":"info","msg":"Failed to parse Access Token from context. Will attempt to find IDToken. Error: [JWT_VERIFICATION_FAILED] Could not retrieve bearer token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer","ts":"2023-06-27T14:31:08Z"}
{"json":{"src":"token.go:103"},"level":"debug","msg":"Could not retrieve id token from metadata rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken","ts":"2023-06-27T14:31:08Z"}
{"json":{"src":"cookie.go:79"},"level":"info","msg":"Could not detect existing cookie [flyte_idt]. Error: http: named cookie not present","ts":"2023-06-27T14:31:08Z"}
{"json":{"src":"handlers.go:65"},"level":"error","msg":"Failed to retrieve tokens from request, redirecting to login handler. Error: [EMPTY_OAUTH_TOKEN] Failure to retrieve cookie [flyte_idt], caused by: http: named cookie not present","ts":"2023-06-27T14:31:08Z"}
{"json":{"src":"handlers.go:121"},"level":"debug","msg":"Setting CSRF state cookie to tb9f2xhb2y and state to 2419390fb3ddca455183ba94811c3c6a3a9d988b99536691fde913716e22cd65\n","ts":"2023-06-27T14:31:08Z"}
{"json":{"src":"composite_workqueue.go:88"},"level":"debug","msg":"Subqueue handler batch round","ts":"2023-06-27T14:31:08Z"}
{"json":{"src":"composite_workqueue.go:98"},"level":"debug","msg":"Dynamically configured batch size [-1]","ts":"2023-06-27T14:31:08Z"}
{"json":{"src":"composite_workqueue.go:129"},"level":"debug","msg":"Exiting SubQueue handler batch round","ts":"2023-06-27T14:31:08Z"}
{"json":{"src":"handlers.go:141"},"level":"debug","msg":"Running callback handler... for RequestURI /callback?code=XXXXXX

I am behind an azure/application-gateway as ingress, and as far as I know appgw does not support gRPC, can it be the issue? Or should I look somewhere else?

Reminder: 🔴 Community meeting is starting now - Info: https://www.addevent.com/event/EA7823958

Are folks using Flyte to do hyperparameter optimization to the scale of 1.4k tasks in a single workflow? We are hitting error from K8S request entity too large, where Fyte propeller OSS did make a patch explicitly for this.

Hello everyone. I am trying to use Flyte to launch a dynamic workflow of Pytorch Elastic tasks and keep on running into the following error when using more than a single Node

Workflow[...] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[dn0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [pytorch]: number of worker should be more then 0

Is this something people have seen before? Is there a more appropriate place to ask about this?

Hello, Could someone please point me to the latest docker file sample for building a custom flyte image with added custom python packages? I will install the latest flytekit release in it, plus my custom packages. The one I am currently referencing is old https://github.com/flyteorg/flytesnacks/tree/master/cookbook/integrations/kubernetes/k8s_spark. And I heard I don’t have to include spark related commands with the latest flytekit anymore.

Hi Flyte community, Is 500MiB roughly expected for the hello_world.py workflow in the flyte cookbook? Running locally without Flyte (but with Flyte annotations) seems to use 140MiB (but when I execute remotely through Flyte with

./bin/pyflyte run --remote core/flyte_basics/hello_world.py my_wf

an allocation of 300MiB results in OOM, 500MiB allowed workflow to complete successfully). Running hello_world.py without importing flyte modules seems to use 0.55MiB. Are there potential settings I could look into to optimize memory usage? Looking into using Flyte for long running operations, so reducing memory overhead as much as possible would help reduce costs for these long running concurrent workflows.

Does Flyte support the caching of M2M authorization tokens? In the context of Auth0 and external auth servers - or is this something we'll have to build out with middleware + Redis, etc?

Hi, I am trying to mount a path from the host node to the task container using pod templates. For that I created a pod template on kubernetes using the following yaml:

apiVersion: v1
kind: PodTemplate
metadata:
  name: my-pod-template
  namespace: flyte
template:
  metadata:
    labels:
      pod-template-name: my-pod-template
      pod-template-namespace: flyte
  spec:
    containers:
      - name: default
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
        volumeMounts:
          - name: mnt-volume
            mountPath: /mnt
    volumes:
      - name: mnt-volume
        hostPath:
          path: /mnt

And trying to reference this pod template on the client side with:

@task(pod_template_name="my-pod-template")
def print_files() -> None:
    print(os.listdir("/mnt"))

But the task immediately errors out with PodTemplate does not exist My flyte deployment is on version 1.3 and flytekit on version 1.2.11. Is there anything I am missing or do I need different flyte versions? I was trying the same using PodSpec (flytekitplugins-pod) which worked fine.

Hi everyone! I've recently started trying Flyte in order to deploy some ML pipelines. Since i use some privately developed libraries in my code i figured out that i must dockerize and push it to the registry in order to execute the flows w/ pyflyte . However, i haven't use Docker extensively so i have trouble pushing and using. Is there any sort of guide on how to extend the demo image or a base dockerfile to begin with? Thank you very much!

Hi Team, We have our flyte cluster url enabled with IAP (Identity-Aware Proxy). While the UI access to flyte cluster works fine using the google account login but accessing flyte cluster using flytectl fails at authentication and produce below error

Error: authentication error! Original Error: rpc error: code = Unauthenticated desc = Invalid IAP credentials: empty token, Auth Error: failed to initialized token source provider. Err: failed to fetch auth metadata. Error: rpc error: code = Unauthenticated desc = Invalid IAP credentials: empty token

Anyone knows how to deal with this? Is there a way to provide the identity token which can be used by flytectl, pyflyte, and flytekit_sdk for authentication

Hey all I have a workflow that runs for 2-3 hours, use many nodes, and works well. I tried running 10 in parallel and this happened on some of them:

Workflow[flytesnacks:development:algopipelines.flytedatapipeline.main_pipeline.smplx_proccess_pipeline] failed. RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: 0: 0: 0: 0: 0: 0: [system] unable to read futures file, maybe corrupted, caused by: [system] Failed to read futures protobuf file., caused by: path:gs://<my-gcs-bucket>/metadata/propeller/flytesnacks-development-algopipelinesflytedatapipeline-da0bfb4f/n10/data/0/dn19-dn0/0/futures.pb: not found
1: 0: [system] unable to read futures file, maybe corrupted, caused by: [system] Failed to read futures protobuf file., caused by: path:gs://<my-gcs-bucket>/metadata/propeller/flytesnacks-development-algopipelinesflytedatapipeline-da0bfb4f/n10/data/0/dn19-dn0/0/futures.pb: not found

Any ideas what may be causing this?

Hello everyone, I have a question. Suppose I build a workflow that follows this logic: A >> B >> C >> D and C takes in external input. What i want to do, is to enable users to put in different inputs on C and it will re-trigger D. How do I go about doing that using flyte? I tried to re-run only C but it does not trigger D. Do i need to use condition or something to do it? FYI the output from A & B DO NOT change.

@Tommy Nam thank you for being an excellent community member and helping folks 🎉

hey all! quick storage question, does flyte provide any way to automatically delete a file from blob storage once it’s been successfully read in from a new task? I pretty much have a few tasks that transform data in a file, and my first preference would be to avoid saving down the intermediate data altogether, as it’s quite sensitive. my understanding is that it needs to be saved down somewhere outside the task pod, so the next best thing would be to delete it after it’s read in by the subsequent task

Hey everyone, I need some help with debugging why accessing flyteadmin through my nginx ingress doesn’t work. Details in 🧵 (Background: We run Flyte without auth behind a GKE Ingress with a GCP identity aware proxy using istio. Flyteadmin is available within the VPC from where we register. I want to explore whether I can make Flyte work in authenticated mode with GKE ingress and with GCP identity aware proxy. This would be my dream setup for GCP. For that I first want to have a working flyte deployment with authentication and nginx ingress.)

Hello there, how does defaults and limits work on flyte ? When does limits get executed over defaults?

task_resource_defaults:
    task_resources:
      defaults:
        cpu: 4
        memory: 2Gi
        storage: 500Mi
      limits:
        cpu: 6
        memory: 4Gi
        storage: 750Mi

hey gang, new user for the python SDK - are there examples of good patterns for mocking remote tasks/workflows, a la what

task_mock

can do for local python tasks? While I can get local patching working sometimes, it runs into issues e.g. with explicit task dependencies set like

t1() >> t2()

where the return value (mocked as a python type) is missing some flyte node plumbing under the hood.

Hi, I was attempting to use the ImageRenderer class from the flytedeckplugins library, but ran into an error:

module 'PIL' has no attribute 'Image'

. I dug into the weeds, and found out that PIL was not being imported at all, so I added it to the top of the file. Should I file an issue on Github for this? I notice that ImageRenderer is not on the Flyte docs at all as well.

Hello, In ImageSpec https://docs.flyte.org/projects/cookbook/en/latest/auto/core/image_spec/image_spec.html Can the base_image be a AWS ECR repository url, e.g. 613630599xxx.dkr.ecr.us-east-1.amazonaws.com/ml-flyte:latest ?

Hi flyte folks, i think i encountered another Optional type related issue. This seem to be happening only during local execution though. Issue: For dynamic tasks with Optional return it crashes if we explicitly return None. Example code:

@dynamic
def dyn_optional_str(optional_str: Optional[str]) -> Optional[str]:
    return None

@workflow
def workflow_with_optional_input(optional_str: Optional[str]) -> Optional[str]:
    hello_output = dyn_optional_str(optional_str=optional_str)
    return hello_output

and test was done like this

def test_example() -> None:
    result = workflow_with_optional_input(optional_str=None)
    assert result is None

and it crashes with this error

except Exception as e:
                    logger.error(f"Failed to convert return value for var {k} with error {type(e)}: {e}")
>                   raise TypeError(
                        f"Failed to convert return value for var {k} for function {self.name} with error {type(e)}: {e}"
                    ) from e
E                   TypeError: Failed to convert return value for var o0 for function sandbox.dynamic_optional_type.dyn_optional_str with error <class 'AssertionError'>: Task sandbox.dynamic_optional_type.dyn_optional_str returns nothing, NoneType return cannot be used

/fn/lib/venv/lib/python3.10/site-packages/flytekit/core/base_task.py:572: TypeError

i think it might be some oversight on flytekit side 🤔

Has anyone encountered ImageSpec removing packages? I ran

python -m pip list

inside my base image container and new image container, and noticed that the new image is missing several packages. In trying to debug this, I didn't specify any new packages to install in the image spec, so the images should be identical.

hello, (not sure if this is the right forum). I was wondering if flyte can handle realtime (while user is in session) data munging use cases. I see this was brought up in late 2021 in this thread https://github.com/flyteorg/flyte/discussions/1481

Is there some documentation for setting up Flyte with Azure blob storage as storage backend ? There seems to be no coverage in the docs.

Hey all! We are missing a flyte-native mechanism for executing our scheduled workflows based on delayed external upstream dependencies. This mechanism should: On a schedule kick-in: • check if the dependencies (S3 file or S3 prefix) are in place or not; • if the dependencies are ready and the workflow is not running yet - run the workflow; • if the dependencies are ready and there is a workflow running already - do nothing; • if the dependencies are not ready - retry N times with an exponential delay and otherwise log message and do graceful exit (terminate the workflow);

Hi, Im looking into setting up a dev environment, and running into some issues with a hanging java container (on running

make generate

in flyteidl repo which runs a container executing

/opt/swagger-codegen-cli/swagger-codegen-cli.jar generate ...

). For quick experimentation adding functionality I dont need the java bindings, but to get a PR merged I would. I see issues like this one that mention you’re moving towards a monorepo setup: https://github.com/flyteorg/flyteidl/pull/338, and I was wondering if there is a timeline on this? (and basically, should I wait for this, or try to fix my dev environment with the current setup). Cheers 👍