Is it possible the the move to

StructuredDataset

broke support for schema urls in the flyteconsole launch form? Not sure if it ever worked, but https://github.com/flyteorg/flyte/issues/405 seems to indicate it did. Given this example

@task
def t1(df: pd.DataFrame) -> pd.DataFrame:
    return df

@workflow
def wf(df: pd.DataFrame) -> pd.DataFrame:
    return t1(df=df)

I get the following message in the launch form:

hello, Flyte newbie here: I'm trying to use

ContainerTask

where name of the image is a parameter into my workflow. Naive approach does not work as parameter is a promise that is unresolved at task declaration time:

@workflow
def run_test(version: str) -> str:
    run_container_task = ContainerTask(
        ...
        image="repository/test:{version???}",
        command=[
            "uname",
        ],
    )

    return run_container_task(version='v1')

Thanks!

and a quick follow-up: when passing

secret_requests

to

ContainerTask

- how this is made available within container? I don't see

/etc/flyte/secrets

mounted

I'm trying to do some tests in flyte sandbox, but

pyflyte --config ~/.flyte/config-sandbox.yaml register ...

gives me exceptions: failed to put data... Is this supposed to work out of the box? I can't seem to find where I configure credentials as that seems to be the problem.

Any way to run the ShellTask on a specific image right now (besides using another image for the whole workflow)? To my understanding, is neither the task_config option nor the pod_template argument yet possible, right? Can I use ShellTasks right now only with the flyte image that runs by default on tasks?

Hello everyone, look at me again here. I’m almost done installing flyte where I work I have this error:

Attempt failed due to rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken"

The flytescheduler pod did not start because of this error above Has anyone gone through something similar?

Running the greeting example... I am getting the following error on the

say_hello

task 🧵

#ask-the-community Can someone help me how to configure spark plugin in Flyte. I am stuck in cluster_manager_resource. Should i override the service account name ??

is there a way to completely delete execution/workflow? have accidentally leaked sensitive information in parameters,

flytectl delete execution

seems to actually abort, but not delete

Hello, is there a way to get the execution date of a task or workflow scheduled to run at intervals. The idea is to dynamically pass in the execution date as parameter in the task/workflow. Also, if there is need to rerun a task from past runs, the specific execution date for that run will be passed and not the current date.

Hi guys, I'm using Flyte on a project that predicts glass type using

KNeighborsClassifier

It seems Flyte doesn't accept

KNeighborsClassifier

as the type of my function. What would be the right type hint for this? I get the following error:

[4/4] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[sd4h34hqd15wyc3wy32t-n0-3] terminated with exit code (137). Reason [OOMKilled]. Message: 

{"asctime": "2023-03-05 20:24:43,219", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type <class 'sklearn.neighbors._classification.KNeighborsClassifier'> found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."}
2023/03/05 20:24:43 WARNING mlflow.utils.git_utils: Failed to import Git (the Git executable is probably not on your PATH), so Git SHA is not available. Error: Failed to initialize: Bad git executable.
The git executable must be specified in one of the following ways:
    - be included in your $PATH
    - be set via $GIT_PYTHON_GIT_EXECUTABLE
    - explicitly set via git.refresh()

All git commands will error until this is rectified.

This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
    - quiet|q|silence|s|none|n|0: for no warning or exception
    - warn|w|warning|1: for a printed warning
    - error|e|raise|r|2: for a raised exception

Example:
    export GIT_PYTHON_REFRESH=quiet

{"asctime": "2023-03-05 20:24:43,424", "name": "flytekit", "levelname": "WARNING", "message": "Unsupported Type <class 'sklearn.neighbors._classification.KNeighborsClassifier'> found, Flyte will default to use PickleFile as the transport. Pickle can only be used to send objects between the exact same version of Python, and we strongly recommend to use python type that flyte support."}
.

Can you please point me to the right direction?

Hello guys Can someone please show me an template of a working AWS S3 Config? And how to connect it to FlyteRemote? Flyte doesn't seem to notice my configuration.

Hi Everyone, I have a question about conditionals, I trying to implement a workflow with a task that is ran if an input flag is set to True, otherwise this task is skipped. I attempted to use conditional for this, but it seems like the workflow terminates after executing the conditional and does not carry on to next tasks. Code:

import logging
import time
import typing
import uuid
from typing import Tuple

from flytekit import task, workflow, conditional

logger = logging.getLogger()

@task
def create_dataset() -> uuid.uuid4:
    print("Creating dataset")
    time.sleep(1)
    dataset_id = uuid.uuid4()
    print(f"Dataset created with id {dataset_id}")
    return dataset_id

@task
def sort_dataset(dataset_id: uuid.uuid4) -> None:
    print(f"Sorting dataset with id {dataset_id}")
    time.sleep(1)
    print(f"Dataset is sorted")

@task
def train_model(dataset_id: str) -> uuid.uuid4:
    print(f"Training model using dataset with id {dataset_id}")
    model_id = uuid.uuid4()
    time.sleep(1)
    print(f"Trained model with id {model_id}")
    return model_id

@workflow
def train_workflow(sort_dataset_flag: bool) -> None:
    dataset_id = create_dataset()
    conditional("Sorting").if_(sort_dataset_flag.is_true()).then(sort_dataset(dataset_id=dataset_id))
    train_model(dataset_id=dataset_id)

def test_workflow():
    train_workflow(sort_dataset_flag=True)

Output:

Creating dataset
Dataset created with id bba83be5-8b15-417d-bd48-5cce55d1856f
Sorting dataset with id bba83be5-8b15-417d-bd48-5cce55d1856f
Dataset is sorted

Notice how last task is not executed. How can I make this work?

👋 Hello, team!

I have a few questions: 1. Can I have a dynamic workflow as a subworkflow of a static workflow? 2. The chain flyte entities documentation shows chaining of tasks with single or no returns, how to chain tasks/workflows with multiple returns? 3. Can the DAG of a static workflow be acquired through flytekit on creation of workflow? What method allows this? 4. Is the same (question 3) possible for a dynamic workflow?

Hi all - I'm writing an article about about using Flyte to run a docker container based workflow, much like this: https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/raw_container.html However when I try to run on my local sandbox the tasks all stall out

[ContainersNotReady|PodInitializing]: containers with unready status: [... flyte-copilot-sidecar]|

Is this because it can't access the image inside the sandbox, or something more complicated? Is there a way to get this type of ContainerTask example to work on the sandbox? As is, my article ends like this - 'So you can push the dag (

pyflyte run --remote ..

), see it in Flyte ui, but it won't run locally, so go find a prod server if you want to fullly test, ok gl with that bye'

is this already supported and I need to upgrade, if not - when?

File "/opt/venv/lib/python3.8/site-packages/flytekit/core/map_task.py", line 271, in map_task
        raise ValueError(

Message:

    Only Flyte python task types are supported in map tasks currently, received <class 'flytekit.core.container_task.ContainerTask'>

👋 Hi! pyflyte fast registration command takes at least 5+ minutes every time when packaging and registering via github action on to a standalone cluster? Thoughts?

Hi there, we've been having some challenges with the token cache for flytectl inside devcontainers. The use of gnome keyring inside docker containers is tricky business. Is there a possibility of moving to a more straightforward (file based say) token cache instead?

What's the best way to handle workflow registrations when the workflow code only works inside the docker containers that they are meant to run in. e.g. importing a library that you don't have installed but is in the docker.

Is there any examples of using ContainerTasks with FlyteFile and FlyteDirectory? I’m trying to use a ContainerTask that has a FlyteFile as an input, and the output is a directory but struggling to make anything work. Any pointers?

Hi Flyte community, Does using the pod template setting for task registration completely override the pod template that Flyte defaults to or does it add to it? I'd like to add the following setting so my raw container Flyte task pods can refer to the their corresponding settings (based on stage/app):

envFrom:
      - configMapRef:
          name: special-config
      - secretRef:
          name: special-config

https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/pods/pod-configmap-envFrom.yaml

Hi Flyte community, Does Flyte happen to support keeping raw containers alive across task requests? I'm experimenting with some java defined workflows that involve spring, but there's a cold start of roughly 30 seconds just from starting up the application in a custom defined pod (not including Flyte overhead).

Hello Flyte community, related to Flytefile usage - would it be feasible to handle S3 interactions via

boto3

instead of

aws s3

subprocesses? The reason being that

awscli

does not play nice with other dependencies I have, so I wanted to know if migrating that code to a boto client is an option to be able to drop

awscli

. Thanks in advance!

Are there any disadvantages to using dynamic rather than static workflow? I can only think of the disadvantage that we cannot access the DAG on registration. Any other reasons why static workflow might be better in some cases?

Hi! Is there a way to make caching using the

Annotated[..., HashMethod(...)]

work with

FlyteFile

? It doesn't work out-of-the-box, failing an `issubclass` check in flytekit/types/file/file.py. However, I have put together a two-line patch (can submit an MR) for that file that would allow the use of the annotated type alias with a custom hash function. Or am I chasing a red herring here and there is an easier way to get content-aware caching for `FlyteFile`s? Thanks! 🙏 Here's the example workflow (expected behavior is that

print_file

is executed only when either its inputs change, or the signature/`cache_version` is bumped -- if I omit the custom hash method, the cache is hit regardless if the inputs have changed in content):

def calc_hash(f: FlyteFile) -> str:
    ...


CachedFlyteFile = Annotated[FlyteFile, HashMethod(calc_hash)]


@task
def write_file() -> CachedFlyteFile:
    print("write_file")
    local_path = "data.parquet"

    df = pd.DataFrame(data={"a": [1, 2, 3], "b": [3, 4, 5]})
    df.to_parquet(local_path)

    return FlyteFile(local_path, remote_path="s3://...")


@task(cache=True, cache_version="1")
def print_file(file: FlyteFile) -> None:
    file.download()
    print(pd.read_parquet(file))


@workflow
def wf() -> None:
    f = write_file()
    print_file(file=f)


if __name__ == "__main__":
    wf()
    wf()  # don't expect to see output from `print_file`

I’m encountering a problem during workflow registration when task arguments (e.g. an

nn.Module

) need to be serialized into blob storage. Instead of a bucket uri, a

/tmp/flyte-xyz

path is shown in the console: I expect that my flytekit/remote config is not correct and would appreciate some help to figure out where exactly. Thanks 🙏 More in the 🧵

Hello @Yee and @Dan Rammer (hamersaw) please your attention is needed here. Thanks. https://flyte-org.slack.com/archives/CP2HDHKE1/p1678169278076989?thread_ts=1678076792.831259&cid=CP2HDHKE1

Hi Everyone… I’m trying to register a package but getting this.. couldn’t find information about it…

(mlp2) ematus@eduardos-mbp dockerization % flytectl -c ~/.flyte/config.yaml register files flyte-package.tgz -p flytesnacks -d development --version 6 --archive
Error: 
strict mode is on but received keys [map[task_resources:{}]] to decode with no config assigned to receive them: failed strict mode check
ERRO[0000] 
strict mode is on but received keys [map[task_resources:{}]] to decode with no config assigned to receive them: failed strict mode check  src="main.go:13"

Has anyone tried using

FlyteRemote

with device flow auth? I think it is not supported currently, but I wanted to check if I'm missing something.