It looks like this in the UI, with many sub tasks named with xgbse_train. Is there a way to make the task names to show the different values of tenure, e.g. xgbse_train_1, xgbse_train_2, …xgbse_train_24?

Hi everyone… from the docs..

Flyte User Role
Finally create a role for Flyte users. This is the role that user pods will end up assuming when Flyte kicks them off.

Create a role flyte-user-role from the IAM console. Select "AWS service" again for the type, and EC2 for the use case. Also add the AmazonS3FullAccess policy for now.

The flyte user role is the one the docker image will use to access resources (like S3, elasticsearch, etc?)

Hi all, just getting my hands dirty and trying to wrap my head around FlyteFile/FlyteDirectory. From the ShellTask example I can see the output paths being generated dynamically based on current_context() etc. My question is: what is the best practice for referencing pre-existing data in the object store (minio in the local sandbox in this case) in tasks inputs? From the data management docs I see "primitive data types and references to large objects fall under Metadata - Meta input or Meta output, and the actual large object is known as Raw data", but I'm not entirely sure how to apply that. Any help is greatly appreciated!

Hey peeps, we've been wondering if anyone has experience in running a multi-arch cluster/workflow. Currently, we are running the flyte cluster on amd64 but have a task where we would like to e.g. build a docker image using kaniko on an arm64 node. What keeps us from doing so is that flyte-copilot seems to use the amd64 built image and not "know" that he is going to be executed on an arm64 node (which is ensured by using node selection). Is there a way, besides hosting the full setup on arm64, to make flyte-copilot more "flexible"?

Hi all, does anyone know why CPU limits if omitted are set to CPU requests? I need to run my pods with CPU requests but no limit.... https://github.com/flyteorg/flyte/issues/3574

Hey, is there a place where i can see all the docker images versions for flyteadmin, flytepropeller etc for all available cpu architectures?

Hey flyte team im trying to use flyteremote and am running into a SSL error. Peer name is not in peer certification. The documentation doesnt make it exactly clear how to pass in the cert unless I missed something. Is it as simple as passing in the path to the “certificate_chain” option?

Hey gang, curious if anyone has experience running dynamic shell tasks? Been banging my head against a "Failed to Bind variable" error for a couple hours now.. I realize that dynamic tasks are quite strict around what you can and can't do owing to the promises / run-time compilation but I just can't figure out what I'm doing wrong:

from typing import List
from flytekit import kwtypes, workflow, dynamic
from flytekit.extras.tasks.shell import OutputLocation, ShellTask
from flytekit.types.file import FlyteFile

s1 = ShellTask(
    name="shorten",
    debug=True,
    script="""
    set -ex
    head {inputs.infile} > {outputs.i}
    """,
    inputs=kwtypes(infile=FlyteFile),
    output_locs=[OutputLocation(var="i", var_type=FlyteFile, location="outfile.txt")],
)

@dynamic
def shorten_files(files_in: List[FlyteFile]) -> str:

    for i in range(len(files_in)):
        s1(infile=files_in[i])

    return "DONE"

@workflow
def wf(files_in: List[FlyteFile]) -> str:

    return shorten_files(files_in=files_in)


if __name__ == "__main__":
    print(f"Running wf() {wf()}")

.. invoked with:

pyflyte run --remote dynamic_shell.py wf --files_in '["xaa", "xab"]'

Any help would be greatly appreciated!

Hi all, the flyte setup I'm working with has

OIDC

userAuth

(and

Self

appAuth

if it's relevant), meaning any time I want to register a workflow or list a resource with flyte a browser window gets opened to complete the auth flow. I'm trying to set up a service that will be able to run workflows but won't have access to a browser. I was looking around the authentication page in the docs but I don't really understand which

clientId

/

clientSecret

does it refer to in my case. Has anyone set this up before?

A flyte task that takes 3-5 minutes locally (relatively simple pytorch project) has been running for over an hour with default resources (2 cpu, 0.2Gi) - is there a way to inspect what’s going on directly via kubectl? If it’s insufficient memory, it should’ve been killed with OOMKilled, but it just says it’s running and I’d like to find out why

If you can get the execution ID you should be able to pull the logs from the pod with the same ID directly

e.g.

k logs ff1da0f5f07414327b85-n0-0 -n flytesnacks-development

@seunggs

Hello, I am using dynamic to kick off parallel XGBSE training tasks. For very small dataset, it works fine. However, with large dataset, the xgbse_training() will be in [ContainersNotReady|ContainerCreating]: containers with unready status: [arlhbbz2jt56sq7fs2hj-n2-0-dn0-0]| for hours. If I am not using the dynamic, then the xgbse_training task will succeed for the same datasize. Do you knwo what might be going on? Thank you!

@dynamic(
    requests=Resources(cpu="4", mem="20Gi"),
)
def train_foreach_tenure_small(
    df: pd.DataFrame,
) -> None:
    
    for tenure in range(1, 3, 1):
        data = df[df['TENURE']==tenure]
        xgbse_training(df)


@task(requests=Resources(mem="5Gi"))
def xgbse_training()

I observed that both the dynamic and xgbse_training tasks are executed in different pods, but on the same EC2 instance and there are plenty of resources left on the EC2 instance.

I also observed that both the dynamic and training tasks are running on the same EC2 instance where the flyteadmin, flytepropeller, flytescheduler are running.

How can I change the dynamic’s configuration to move those tasks to a different EC2 machines where all other flyte tasks not managed by dynamics are normally executed?

Hi everyone, I am looking to take in a the output of SerializeToString from a proto as a param of a task. Flyte gives warnings when I use a byte as the type of the param. Does anyone know what the best way to do this would be?

For using arbitrary containers with a shell based tasks, do you guys typically go straight to ContainerTask or is there an easy way to specify a different image for ShellTask via the inherited task_config param?

Hi, I am trying to play with the sandbox, but I always get oom killed no matter what images or tasks I use. I use the following code:

curl -sL <https://ctl.flyte.org/install|https://ctl.flyte.org/install> | sudo bash -s -- -b /usr/local/bin 
flytectl demo start 
export FLYTECTL_CONFIG=/root/.flyte/config-sandbox.yaml 
# then, pyflyte run --remote ./test.py wf

It is unlikely due to insufficient memory since any tasks can cause the above problem. Any hint on this? thank you in advance!

Hi! I have a few questions about Flyte caching.. I read through the documentation here a few times but I’m still a bit confused about why the “Caching of Non-flyte Offloaded Objects” examples are hashing the task output instead of the input. My intuition for how a cache would work is that the input values for a task needs to be hashable so that they can be used as keys for the cache. In which case, I feel like it would make more sense to add the

Annotated HashMethod

to the input types of a task instead of the output. Then, the

HashMethod

can be run on the inputs when a cached task is invoked to determine the cache keys and then check if there's a hit. I tried this out and it kind of works for local caching? but not for remote.

@task(cache=True, cache_version="1.0")
def my_task(obj: ty.Annotated[pd.DataFrame, HashMethod(hash_df)]) -> pd.DataFrame:
    redis_client.incr(REDIS_COUNTER_KEY)

    return obj

@dynamic
def my_workflow():
    obj = pd.DataFrame(
        {
            "name": ["a", "b"],
            "val": ["test1", "test2"],
        }
    )

    obj = my_task(obj=obj)
    obj = my_task(obj=obj)
    my_task(obj=obj)

In the example above

my_task

will be be called three times the first time

my_workflow

is called. This still doesn't match my expectation since I thought it would be called once for the first call and for the cache to be hit on the second and third call since the input

obj

is the same. However! The second run of

my_workflow

has a cache hit for all three calls to

my_task

so it does work to some extent even though I don’t fully understand what it’s doing. For remote caching, this doesn’t seem to work at all and there are no cache hits no matter how many times I run

my_workflow

.

I might have missed this in the docs, but is it expected that the flyte backend is backwards-compatible with older

flytekit

versions? We have some other dependencies blocking upgrading

flytekit

presently, but would like to keep the backend services as up-to-date as possible

Hey Folks, i am currently trying to set up authentication on my flyte-binary (single-cluster) deployment. As soon as i turn "auth.enabled: true" my pod is failing over and over again. Could you give me a hint how i can get some logs to find out what part of the config could be the source of the problem. kubectl describe and kubectl logs didn`t really help yet and im quite new to kubernetes and helm.

Hi all, I’m trying to launch executions of a workflow via HTTP requests. Is there a GET method to start execution of a launch plan that’s already uploaded to the project/domain? Or are there only POST methods?

Is there a way to specify the k8 service account to use when executing a task/workflow with FtelyRemote.execute?

Hello, I have a workflow that executes training tasks parallel. If one of the parallel training tasks failed, I observed that all the other tasks are aborted by Flyte. How do I change the config so that other parallel tasks and their downstream tasks will continue to the end, even when one or two training tasks failed?

@workflow
def wf_train_parallel():
    for tenure in range(1, 25, 1):
        xgb_train(tenure=tenure)
        downstream_task()
...

@task
def xgb_train(tenure: int)
...
@task
def downstream_task()
...

hi

@channel

, the core Flyte team has heard reports of long import times when simply running a script locally that does

import flytekit

(the number I’ve heard of is ~8 seconds). I’ve been unable to reproduce this locally, but can anyone with this experience provide instructions on how to reproduce this behavior? We’d like to look into reducing this import time.

hey quick question -- I have some config maps that are domain/account specific I would like to have mounted onto my flyte pods as environment variables but I see no docs on how to do that I do see docs about mounting secrets but I would prefer not to have go through flyte secret manager and specify in each workflow the env variable when I use an environment manager already that handles it I feel like I am missing something here so if anyone could point me in the right direction that would be great

Hi all. A QQ. Can I enforce max task parallelism per workflow execution so it cannot be overwritten via

LaunchPlan.get_or_create(… max_parallelism=100)

?

yes

Hey everyone. is flyte single cluster deployment supported in AKS? I have an instance running in AKS after modifying the binary helm chart, but having difficulty running workflow using flytekit. I don't see much of docs for flyte in AKS. Just want some confirmation for my team if it is officially supported or not (and resources or tips if any). Thanks!