Hello, I keep getting the following error on a custom image:

Original exception: module 'aiobotocore' has no attribute 'AioSession'

(error log in thread below) could it be related to the fsspec version issue by any chance?

Is there a way to add a custom toleration to the flag for spot nodes? We need both the affinity for spot, as well as a spot toleration to enable scheduling - however I can’t find where this would be done via the

flyte-binary

helm chart? I’ve seen this but I’m unsure how that is surfaced via the helm chart?

Hey there! We are noting some relatively high latency for simple task execution. We're using the flyte-binary deployment in GKE. Tasks start immediately and the pods show complete but take a couple minutes before formally completing in flyte (these are simple hello world tasks). Are there any knobs to turn to lower this execution latency. Some sort of heartbeat interval we can tighten? We've tried lowering both workflow-reeval-duration and downstream-eval-duration, but regardless a task, even running on its own, takes ~2 minutes. @Dan Rammer (hamersaw)

Hi, I'm trying to understand concurrency using Flyte. I've implemented this example, • https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/subworkflows.html#example-2 Under which or what circumstances does concurrency occur?

Hello! Back with more Azure adventures 🙂 I've managed to trigger a basic workflow from

flytesnacks

now, but in the UI I see the following error for my execution:

FlyteAssertion: Failed to get data from 
<abfs://flyte/flytesnacks/development/A46TBP35TISQ2EVJVNOGLJTARM======/script_mod>
e.tar.gz to /root/ (recursive=False).

Original exception: unable to connect to account for Must provide either a 
connection_string or account_name with credentials!!

Here's my storage config in the helm chart:

storage:
  type: custom
  bucketName: "flyte"
  custom:
    container: "flyte"
    enable-multicontainer: true
    connection: {}
    type: stow
    stow:
      kind: azure
      config:
        account: "<storage_account_name>"
        key: "<storage_account_key>"
  limits:
    maxDownloadMBs: 10

Does anyone know what would cause this? My first intuition would be that this URI is wrong:

<abfs://flyte/flytesnacks/development/A46TBP35TISQ2EVJVNOGLJTARM======/script_mode.tar.gz>

because in Azure the path should look like this:

abfs://<CONTAINER>@<STORAGE_ACCOUNT>.<http://dfs.core.windows.net/flytesnacks/development/A46TBP35TISQ2EVJVNOGLJTARM======/script_mode.tar.gz|dfs.core.windows.net/flytesnacks/development/A46TBP35TISQ2EVJVNOGLJTARM======/script_mode.tar.gz>

Hello. I am able to register my python workflows through

pyflyte register

and able to see it in the UI, however I am importing some packages like boto3 etc and it throws and error while running the workflow.

Hi All, I am following the flyte deployment from Github and I am facing an issue setting up. Below is the error from docker:

Timed out while waiting for the datacatalog rollout to be created

Terminated

Timed out while waiting for dockerd to start

Terminated

found several pods were in ContainerCreating status, when i logged in one of them - found some "MountVolume" error as shown below: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 21m default-scheduler Successfully assigned flyte/flytepropeller-6f5c985cd4-kflzv to a7b7d18b2fc5 Warning FailedMount 21m (x2 over 21m) kubelet MountVolume.SetUp failed for volume "auth" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 21m (x2 over 21m) kubelet MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 21m kubelet MountVolume.SetUp failed for volume "kube-api-access-jcdxv" : failed to sync configmap cache: timed out waiting for the condition Normal Pulling 21m kubelet Pulling image "cr.flyte.org/flyteorg/flytepropeller-release:v0.19.2" Warning FailedMount 5m19s kubelet MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 5m12s (x4 over 5m19s) kubelet MountVolume.SetUp failed for volume "auth" : failed to sync secret cache: timed out waiting for the condition Normal Pulling 5m6s kubelet Pulling image "cr.flyte.org/flyteorg/flytepropeller-release:v0.19.2" Any pointer to fix such issues? I've tried removing image and redeploying without any luck.

We are regularly hitting the issue mentioned in this resolved ticket: https://github.com/flyteorg/flyte/issues/1234 In our case we have a

@workflow

that uses map `@task`s and spins up 10-100s of pods. Our EKS setup is very elastic and sometimes will bin-pack these pods onto the same node. When it does that there's a high chance that one of the pods in the map

@task

will fail with an error similar to the issue linked above. Our kubelet is currently setup to do serial image pulls. So, in theory, once one of the `@task`s pulls the image it should then be available for all of the other pods. But it seems that's not the case. Initially i thought the fact that flyte is setting

imagePullPolicy: Always

was a problem, but reading the docs more closely it seems that's not the case.

Always

every time the kubelet launches a container, the kubelet queries the container image registry to resolve the name to an image digest. *If the kubelet has a container image with that exact digest cached locally, the kubelet uses its cached image*; otherwise, the kubelet pulls the image with the resolved digest, and uses that image to launch the container.

(emphasis mine) Has anyone observed this issue? Any recommendations?

Running into issues with flytepropeller and flytescheduler when trying to setup okta authentication using OpenId https://docs.flyte.org/en/latest/deployment/configuration/auth_setup.html#openid-connect flytepropeller complaining

{"json":{"exec_id":"avjrcs5qwjzgnwz7cxfs","ns":"flytetester-development","res_ver":"76272166","routine":"worker-1","wf":"flytetester:development:<http://turbine_flyte.example.wf|turbine_flyte.example.wf>"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]]","ts":"2023-05-02T18:07:36Z"}

saw this thread https://flyte-org.slack.com/archives/CP2HDHKE1/p1676910082719799 but the PR is closed

Hello community, how to use image from AWS ECR while running pyflyte register -i command .. by default it's pointing to the public docker hub .. is there any configuration given for the same.. i have tried imagePullSecret by attaching to the default service account still it's not working so...