My company has configured Flyter Admin server to store flyte data in another S3 bucket/folder. How can we change the flyte config to replace s3://my-s3-bucket/data with the the other bucket/folder that has the necessary permissions?

Hey yall! I'm running into the same stack trace outlined on this issue, and seemed to be addressed. I'm on a recent version of flyte (propeller, data catalog, admin 1.1.78), but trying to use map task for 30k inputs with concurrency=10. Am I stressing the k8s-array beyond it's limitation?

Running into a weird issue where when I submit a job using FlyteRemote that has optional args (in this example,

Optional[str]

, when I try to rerun it using the UI Rerun Button, the field doesn't get populated. But if I launch the job using the UI, it does get populated in the rerun panel.

Hi, when namespacing is {project}-{domain} is there any way to set the resourcequotas based on {project}-{domain} combinations?

Hi, is it possible to change the bucket that flyte uses? If so, are there migration instructions? We tried, and on the console, see 500 errors reading data from the old bucket for old workflows

Could it be that something has change within propeller when getting default platform resources? We’re trying to update from Flyte 1.3.0 to 1.6.0 at the moment and scheduler and worker pods of the

dask

plugin are unschedulable, as limits are not set properly. We’re thinking

executionMetadata.GetPlatformResources()

returns

nil

in the dask plugin, leading to limits not being set which is invalid with `ResourceQuota`’s in place. We do have a workaround of explicitly setting those limits for now but wanted to flag this.

Is there a way to completely mute logs from

flytectl

? We’ve tried

--logger.mute

but that doesn’t seem to do the trick:

root@316456918fe4:/# flytectl --logger.mute version      
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
{
  "App": "flytectl",
  "Build": "bd6b856",
  "Version": "0.6.36",
  "BuildTime": "2023-05-17 07:34:12.851237172 +0000 UTC m=+0.290831418"
}

Dear all, I am having some troubles configuring the project resource quotas in my deployment. I am using the flyte-binary Helm chart on Kubernetes (on-premise) and keep getting the error message below when trying to register my workflow. I tried several things none of which changed the error I am getting. I also confirmed that the Helm deployment is actually updated and I can see changes in the resources created (helm get ...) 1. Add the following to helm values.yaml

configuration:

inline:

task_resource_defaults:

task_resources:

defaults:

cpu: 500m

memory: 500Mi

gpu: 0

limits:

cpu: 8

memory: 32Gi

gpu: 1

2. Add the following to helm values.yaml

configuration:

inline:

cluster_resources:

customData:

- production:

- projectQuotaCpu:

value: "5"

- projectQuotaMemory:

value: "4000Mi"

- staging:

- projectQuotaCpu:

value: "2"

- projectQuotaMemory:

value: "3000Mi"

- development:

- projectQuotaCpu:

value: "48"

- projectQuotaMemory:

value: "100Gi"

3. Update task-resource-attributes through flytectl as described here and here. Any ideas on this? Would be greatly appreciated.

How can I use openai package in Flyte? I get some errors caused by the runtime env not have openai.

Hi Flyte team, When can we expect a new release of flyte-binary helm chart with the below feature? Thanks https://github.com/flyteorg/flyte/pull/3631

Hi Flyte folks, i have a question or maybe a bug So I have a workflow that itself calls other subworkflows. i set the

overwrite cache

in the main parent workflow however that value is not propagating to the subworkflows so i have this weird execution that partially ran. All task and dynamic tasks have been rerun but the subworkflows used cached result is that an intended behavoir or an oversight in propagating the value

Hi, are there any known problems with updating

cluster-resource-attribute

? Following this section to

flytectl update cluster-resource-attribute --attrFile cra.yaml

and when trying to restart flyteadmin it fails in the

sync-cluster-resources

init container (without any error logs).

Can flyte use azure blob as underly storage? I got an error “UPGRADE FAILED: execution error at (flyte-binary/templates/deployment.yaml3735): Invalid value for storage provider. Expected one of (s3, gcs), but got: azure” when I want use blob.

Hello all Which variable should I set on my

values.yaml

to change the default service account? I have tried

k8sServiceAccount

but it is not working...

Dear all, I ran into an inconsistency between flytectl and FlyteRemote which has been reported before. When trying to create a FlyteRemote with this config

admin:

endpoint: dns:///A.B.C.D:PPPPP

insecure: false

authType: Pkce

insecureSkipVerify: true

flytectl works just fine. But when I try to fetch an execution on a FlyteRemote with this code

project = "myproject"

domain = "development"

execution = "a5cphsfgc57nt6nxbknt"

flyte_config_file = "flyte.config.yaml"

remote = FlyteRemote(config=Config.auto(config_file=flyte_config_file))

flyte_workflow_execution = remote.fetch_execution(project=project, domain=domain, name=execution)

I get the following error

Traceback (most recent call last):
  File "debug_remote.py", line 12, in <module>
    flyte_workflow_execution = remote.fetch_execution(project=project, domain=domain, name=execution)
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/flytekit/remote/remote.py", line 353, in fetch_execution
    self.client.get_execution(
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/flytekit/clients/friendly.py", line 582, in get_execution
    super(SynchronousFlyteClient, self).get_execution(
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/flytekit/clients/raw.py", line 43, in handler
    return fn(*args, **kwargs)
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/flytekit/clients/raw.py", line 651, in get_execution
    return self._stub.GetExecution(get_object_request, metadata=self._metadata)
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/micromamba/envs/OHLI/lib/python3.8/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:A.B.C.D:PPPPP: Peer name A.B.C.D is not in peer certificate"
        debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:A.B.C.D:PPPPP: Peer name A.B.C.D is not in peer certificate {created_time:"2023-05-18T14:40:11.018710903+01:00", grpc_status:14}"
>

The cluster is running the flyte-binary Helm chart in version 1.3.0 and I tried flytekit 1.2.11, 1.3.0, and 1.6.1, all resulting in the same error message.

How can I get the status of a list of tasks running in a dynamic task using

flytekit.remote

? I have the execution information and can sync it for the workflow, however I want to be able to give a user in a CLI a command they can run to check the status of a large job (amount of tasks in RUNNING, SUCCEEDED, FAILED, etc..)

Can anyone in the community provide some rough metrics on the Blob Storage usage by Flyte? For example: total # of workflows, executions, tasks is raw user data stored in the same blob storage? and the existing amount of storage used. Also will appreciate it a lot if someone can share experiences where the blob storage becomes the bottleneck/where failure happens.

hello, i was wondering if it’s possible to run a shell task using a specific image? Is there any example of how to do this?

Hey friends, I got tired of pushing and pulling my chonky image from remote so I wrote a little script to push locally to the sandbox, hoping it might help someone..

#!/bin/zsh

REPO=$1
IMAGE=$2
TAG=$3

printf "Building Docker image: %s/%s:%s\n" "$REPO" "$IMAGE" "$TAG"
docker build . -t "$REPO/$IMAGE:$TAG"

printf "Saving Docker image: %s/%s:%s\n" "$REPO" "$IMAGE" "$TAG"
docker save "$REPO/$IMAGE:$TAG" -o ~/Downloads/"$REPO-$IMAGE-$TAG.tar"

printf "Copying Docker image to flyte-sandbox:/tmp\n"
docker cp ~/Downloads/"$REPO-$IMAGE-$TAG.tar" flyte-sandbox:/tmp

printf "Importing Docker image on flyte-sandbox\n"
docker exec -t flyte-sandbox ctr image import /tmp/"$REPO-$IMAGE-$TAG.tar"

Theoretically you should be able to skip saving it on the host first and stream

docker save

directly to

docker cp

, but alas I could not get it working..

Does anybody else run into errors when trying to use map task workflows with Papermill/Jupyter NotebookTasks? Running the example from the docs (https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/map_task.html#map-a-task-with-multiple-inputs), and a plain @task decorated function works fine - but the moment I attempt to run a map task with NotebookTasks, I get this error:

ValueError: Error encountered while executing 'run_map_task_workflow':
  Map tasks can only compose of Python Functon Tasks currently

Is

flytekitplugins.papermill.NotebookTask

currently incompatible with map_tasks?

What’s the best way to attach URL to a Grafana dashboard from Flyte console? Currently, we are able to generate link for stackdriver logs using this. Wondering if there is something similar for monitoring dashboard.

Hello all I'm trying to return machine learning models to find the hyperparameters from a task, the functions in that task is build models, the models then should be passed to another task which will find the hyper parameters, the issue here is that the models are passed as Tuple, so the the task which will find the hyperparameter should expect a tuple of modules, yet it outputs an :Transformer for type <class 'tuple'> is restricted currently, The peace of code as follows

@task
def build_models() -> Tuple[LinearRegression, RandomForestRegressor, Any]:
    lr = LinearRegression()
    rf = RandomForestRegressor()
    ann = HyperModel1()
    return lr,rf,ann
@task
def pipline_(models: Tuple[LinearRegression, RandomForestRegressor, Any], (.....etc)) -> List[Dict]:
       def itirate(models):
           for i in models:
              # Do STUFF

def wf() -> pd.DataFrame:
   ....
....
    models = build_models()
    s_summary = HyperSearch(models=models, ....)
    return s_summary

We are regularly hitting the issue mentioned in this resolved ticket: https://github.com/flyteorg/flyte/issues/1234 In our case we have a

@workflow

that uses map `@task`s and spins up 10-100s of pods. Our EKS setup is very elastic and sometimes will bin-pack these pods onto the same node. When it does that there's a high chance that one of the pods in the map

@task

will fail with an error similar to the issue linked above. Our kubelet is currently setup to do serial image pulls. So, in theory, once one of the `@task`s pulls the image it should then be available for all of the other pods. But it seems that's not the case. Initially i thought the fact that flyte is setting

imagePullPolicy: Always

was a problem, but reading the docs more closely it seems that's not the case.

Always

every time the kubelet launches a container, the kubelet queries the container image registry to resolve the name to an image digest. *If the kubelet has a container image with that exact digest cached locally, the kubelet uses its cached image*; otherwise, the kubelet pulls the image with the resolved digest, and uses that image to launch the container.

(emphasis mine) Has anyone observed this issue? Any recommendations?

Does flyte support returning a

FlyteFile | None

? Or

Optional[FlyteFile | None]

Hi, I am very new to the Flyte and trying to learn how flyte compiles its workflow and there’s a cli: pyflyte serialize, but I would not get it work with an example project, created via pyflyte init. It will be great if anyone can share some insights. Here is my question in the SO: https://stackoverflow.com/questions/76298549/how-to-run-pyflyte-serialize-workflows

I am new to flyte. Looks like I can not install flytectl https://docs.flyte.org/projects/flytectl/en/latest/. on a Windows machine. Is that correct? Thanks !

Hi I was wondering if anyone has run into

ValueError: Empty module name

when running a

map_task()

with flytekit and trying to load the actual

task

object. More details below ⬇️ cc: @Heidi Hurst

Hi! I wondered how would it be possible to allow a flyte user to log a training to his Weights & Biases account. Send the logging via a service account could of course be solved e.g. with a Kubernetes secret. But we want that the logs end up in the console of the starting user. From W&Bs site you can control this by e.g. setting the env vars. But how would I do this with flyte? How can a I set env variables depending on who authenticated user?

[1/1] currentAttempt done. Last Error: USER::containers with unready status: [primary]|context deadline exceeded

I’ve seen a few times such error. What would be the root cause of it?