Hey team! Hope you’re doing okay 🙂 . I’ve got one question related to PodTemplate vs the config through the Plugin. I’m using the plugin config, and also the PodTemplate. Both are working as expected, so for example, if I set some labels in the plugin, and some labels in the PodTemplate, Tasks have the expected labels 👍 . However, when I define envs in the PodTemplate, they are not being reflected at all in the Flyte Task. If I define the default envs in the plugin config, they show up in the Task. Something I could be missing here?

How is the role setup meant to work in a multi cluster environment. Let’s consider the task that was scheduled on our data plane cluster (see screenshot below). 1. The only way we could make it work successfully is by allowing the data plane cluster to assume the user role of the control cluster (role that ends with MOUzPYyR), to download the metadata from that bucket. Is that intended? 2. We also couldn’t really overwrite this role. Even if we set other roles in the UI on the control cluster, the task always used this one 3. Finally, how restrictive are execution labels really? We set a project + domain to only run on cluster3. But when that cluster is not connected, Flyte seems to schedule the workflows somewhere (randomly?). Can this behaviour be prevented?

Hey everyone, I’m new to Flyte, and I’m not a k8s pro so I would like some help to clarify practical concerns. Let’s assume Flyte is running on an AWS/GCP, let’s say EKS/EC2. • How does autoscaling work with Flyte + k8s? • (How) can you control resource allocation, e.g. team or user quotas, for it be fair and efficient while controlling costs? • can you control queueing, priorities? Thank you!

I'm trying to run train a variety of models against a single dataset. The general process works fine but I can't find a clean way to dynamically rename the tasks for each of the model types. Am I missing something? I tried

.with_overrides(name="blah")

but it didn't seem to be picked up.

Hello, Could you point me to an example where I can customize a task plugin to configure it with

FlyteRemote

to wait for a task from a different workflow execution to complete? This is a very common use case supported in Airflow by its native ExternalTaskSensor. I think in Flyte I have to build a User Container Task Plugin to do that. Or if you can provide the FlyteRemote API call code to check if a task from a different workflow is succeeded, it will be very helpful as well. Thanks

Hello, Is Flyte capable of MANUALLY resuming the workflow from the failed task forward? By manually I mean either from the UI or CLI. I see a rerun button on the UI to the right of the tasks. Will clicking the rerun button resume the workflow or just rerun the failed task? What about manually marking a failed task as SUCCEEDED and resume the rest of the tasks forward?

Hey guyz, just wanted to know can we use tfx api with flyte? if yes, do we have a doc / reference link for the same?

Hello folks! Suppose we have Flyte workflow with some failed map_task. At the start, this task submits a lot(300-500) pods to Kubernetes, and one of them dies, so the whole workflow is failed. But the rest pods are still in Kubernetes and waiting to be executed. Why does Flyte not kill the rest pods if one of them is failed? To be able to rerun this workflow, all these pods should be manually deleted from the Kubernetes cluster.

Hey all! anyone has given it a try to convert Kedro Pipelines to Flyte Workflows. We are starting to use flyte and it would be nice to have some kedro pipelines running there

I registered my

ContainerTasks

with

pyflyte register -p myproj -d devel flyte/my_workflow.py

so far and it worked fine. Now I added one Python task, but that one fails with

Pod failed. No message received from kubernetes.

[a9lcdpfc5ldtrh9tlqf8-fl6pvdgq-0] terminated with exit code (1). Reason [Error]. Message: 
Traceback (most recent call last):
  File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 496, in fast_execute_task_cmd
    _download_distribution(additional_distribution, dest_dir)
  File "/usr/local/lib/python3.8/site-packages/flytekit/tools/fast_registration.py", line 107, in download_distribution
    raise ValueError("Destination path is required to download distribution and it should be a directory")
ValueError: Destination path is required to download distribution and it should be a directory
.

I don't get what is wrong here... I can see that the file is in the package..

I'm looking for some more detailed documentation about flyte checkpoints from the doc it appears

current_context().checkpoint

returns a file like object - but no further details are provided specifically I'd like to know if there are integrations with frameworks like keras (the native keras checkpointing mechanism expects a directory, not a single file) and if not, is there a mechanism to write a new file instead of appending to a single file

Hey all, I have a quick question about fast registration. The question is, if my project is formed as a Python package and installed in the docker image, will fast registration load the code into the image and “override” the installed package? I would have thought “Yes” but it hasn’t been working that way.

Hello all, I have a pretty deep repository with multiple tasks and workflows. When I am registering the entire repository with

pyflyte register

, it is taking up a lot of RAM ,sometimes crashing and is pretty slow. Is this expected behaviour. Also what is some best practices for registering such repository.

Hey, I'm looking to deploy Flyte to the GCP for data and ML pipelines following the tutorial here: https://docs.flyte.org/en/stable/deployment/gcp/manual.html#create-a-gke-cluster. I'm wondering if someone has experience in setting up a small to medium production cluster, what kind of instance types they used, and some other things to be careful about ...? Thanks!

Hey guys. Could someone help me with an issue. When I click login, it takes me to a ‘Not Found’ error page and I can’t progress further on this. I need help for this. 1. Request method: GET 2. Status code: 404

Quick question for anyone who has auth enabled (we don’t have it enable at the moment) - is the information about who started a workflow saved/shown somewhere? E.g. in console or as a label/annotation on the k8s resources?

How do you configure node selectors with plugins such as pyspark and ray? For example add non-spot node selector to spark driver and spot node selector to spark executors or non-spot for ray head node and spot for worker nodes?

Anyone having experience in running kaniko in a flyte workflow? Would that be done with e.g. a ShellTask?

I think I found another bug: I tried to recover a workflow that was already itself a recovery. In that case the recovery did not work. IMHO that should work just like triggering the recovery on the original workflow. Or is it intended to work like it is?

Hey folks, anyone know how to override task default and limit resources using the

flyte-binary

helm chart with

eks-starter.yaml

?

I'm trying to write a basic unit test for a task in our workflow and am running into an issue. I've tried using the example from https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.testing.task_mock.html and I've tried calling the python method directly. In both cases I'm not able to get the output from the function. Is there something very simple I'm missing?

Also it looks like flytekit will use the

AWS_ACCESS_KEY_ID

env var instead of the also existing

FLYTE_AWS_ACCESS_KEY_ID

env if the former exists in the container.... this made my task which needs to retrieve data from a different bucket with different creds (injected via secret) fail upon downloading/uploading flyte inputs/output. Will dig into it further next week...

Is there any declerative approach to flyte stuff like gitopsy ?

Hi Flyte Community, is JFlyte compatible with spring boot applications?

Has anyone successfully deployed Flyte in a FedRAMP High environment?

I’m fetching a workflow from Flyte remote and find that the order of the inputs does not match the ordering of the original workflow. Is there any way to preserve this order, or somehow find the original order (from remote only)? Second question, I know the inputs/outputs have docstrings, but is it possible to fetch the workflow docstring? Will this only be possible with the new docs changes?

How are you supposed to launch a workflow/task with a

StructuredDataset

or dataframe input? I get the following error when running `flytectl get task ...`:

Error: failed to convert to a known Literal. Input Type [structured_dataset_type:<columns:<name:"prediction" literal_type:<simple:FLOAT > > columns:<name:"target" literal_type:<simple:FLOAT > > > ] not supported

Is the only option here to wrap it in a workflow?

#ask-the-community @Ketan (kumare3) @Kevin Su My snowflake wf is getting queued. I have updated the JWT token as well.. Any specific reason??{"json":{"exec_id":"az4hkx5fdxq75t7mb5rr"},"level":"warning","msg":"Failed to fetch override values when assigning execution queue for [{ResourceType:WORKFLOW Project:cloudops-max-flyte-demo Domain:development Name:.flytegen.sql.snowflake.no_io Version:ls7kSOwwU6zhrGWUw-T81Q== XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}] with err: Resource [{Project:cloudops-max-flyte-demo Domain:development Workflow:.flytegen.sql.snowflake.no_io LaunchPlan: ResourceType:EXECUTION_QUEUE}] not found","ts":"2023-02-27T112156Z"}

#ask-the-community @Ketan (kumare3) @Kevin Su @Samhita Alla Snowflake task integration issue

I am trying to run basic BigQueryTask example provided in bq plugin doc, when it runs the

convert_bq_table_to_pandas_dataframe

task below error is thrown. Is this combination not supported in structure_dataset encoding decoding?

flytekit/types/structured/structured_dataset.py", line 351, in _finder
        raise ValueError(f"Failed to find a handler for {df_type}, protocol {protocol}, fmt {format}")
Message:
    Failed to find a handler for <class 'pandas.core.frame.DataFrame'>, protocol bq, fmt parquet

[RESOLVED]