How does Flyte handle Pod clean up? I currently have ~30-40 pods in one of my namespaces. Should I do that manually or is this configured else where?

making this a top level thread for visibility 🙂 is it possible to inject env vars into workflows at launch time? or to extend flyte context with user info?

sorry for so much spam today 😅 for context this is all around upgrading to 1.0. Getting the following issue when registering with pyflyte. see 🧵

noticed something a bit buggy in 1.0.0.

flytekit.current_context().execution_id

has a few issues • during compile time it is a string of the form

ex:local:local:local

• during execution time it is a struct of the form project, domain, name and the name portion is empty

message has been deleted

message has been deleted

Hey I am trying to pass a list of string arguments in a ContainerTask and add all of them in the

arguments

field but struggling a bit with understanding if this is possible. 🧵

is there a flyte literal parser utility to produce the json that gets surfaced in the flyte UI?

message has been deleted

message has been deleted

message has been deleted

@Han apologies for the delay, but i owe you the sample commands for specifying multiple images

I see above that

flyte-cli parse-proto

would work to print out the raw values of a proto, is there a way to get the proto from the flyte metastore directly? I’m trying to add an architecture feature to flytekit, and flyteidl and flyteadmin have both been updated, but the value isn’t making it from my register step all the way to the pod runtime, so I’m trying to debug where the breakdown is happening. I’ve copied the closure field from the tasks table, but the proto parsing isn’t outputting any information for me.

message has been deleted

Hi! 👋 We are passing the environment variable

FLYTE_SDK_WORKFLOW_PACKAGES

to

pyflyte workflows serialize

. Looks like that functionality was removed in this PR. Was this intended?

message has been deleted

message has been deleted

message has been deleted

Added a ticket requesting ability to await first task execution success before exiting pyflyte! Thanks for the great demo today!

Anyone using latest

mypy

and have this issue with

NamedTuple

(for named outputs). The example in the docs (shown below) fails mypy 0.910 Code from Flyte docs:

import typing

from flytekit import task

hello_output = typing.NamedTuple("OP", greet=str)


@task
def say_hello() -> hello_output:
    return hello_output("hello world")

Error:

mint/workflows/tmp.py:5:16: error: Unexpected arguments to namedtuple()  [misc]
    hello_output = typing.NamedTuple("OP", greet=str)
                   ^
mint/workflows/tmp.py: note: In function "say_hello":
mint/workflows/tmp.py:10:12: error: Too many arguments for "hello_output"  [call-arg]
        return hello_output("hello world")
               ^

message has been deleted

do we already have a workaround or fix for this? https://github.com/flyteorg/flyte/issues/2508

Flyte Team, I have been working on using

FlyteRemote

and now I understand how to ask my question lol. My remote workflow looks like this:

@dataclass_json
@dataclass
class WrappedList:
    s: str

@flytekit.workflow
def count_files_test_workflow(
    in: WrappedList,
):

How do I call this workflow with

FlyteRemote

? If I try to do:

remote = FlyteRemote(...)
lp = remote.fetch_launch_plan(...)
remote.execute(lp, inputs={'in': WrappedList("foo")})

Then I get

FlyteTypeException: Type error! Received: <class '__main__.WrappedList'> with value: WrappedList(s='foo'), Expected: <class 'types.WrappedlistSchema'>

. I looked into the

type_hints

argument to

remote.execute

, but I don't think that's the right thing here. What I really need to be able to do is specify

inputs={'in': x}

where x is an instance of types.WrappedListSchema.

Scratching my head here a bit... Does anyone use flytekit + flytekitplugins-data-fsspec + fast register? It seems that fsspec does not handle the the default destination

.

(dot, like current directory) well and does not download the tar.gz at all. Reproduced this with this minimal example:

import os

from flytekit.tools.fast_registration import download_distribution as _download_distribution

os.environ["AWS_ACCESS_KEY_ID"] = "key_id"
os.environ["AWS_SECRET_ACCESS_KEY"] = "secret_access_key"

destination = "."

_download_distribution(
    additional_distribution="<s3://bucket/fast-register.tar.gz>",
    destination=".")

It works if I replace

destination

with is

./

in this example. The issue is, that

.

is set by propeller (this is at least my understanding). Is there any way I can configure that?

Jeev helped me find a very interesting bug this morning when using dictionaries on Flyte. The order of the keys seems to be non-deterministic, issue is here!

message has been deleted

Hey Flyte, I am on Flyte backend system

v1.0.0

with

flytekit==0.26.0

and Spark tasks are fine. I just moved up (last night) to

flytekit==1.0.1

with

flytekitplugins-spark==1.0.1

(with the same backend system) and now Spark tasks are broken with the error

can't open file '/usr/bin/entrypoint.py': [Errno 2] No such file or directory

. I took a look and the old (good) tasks have the following set:

"mainApplicationFile": {
    "stringValue": "local:///usr/local/lib/python3.8/dist-packages/flytekit/bin/entrypoint.py"
}

and that file exists and looks fine. However, my new (broken) Spark task shows the following:

"mainApplicationFile": {
    "stringValue": "local:///usr/bin/entrypoint.py"
},
"executorPath": {
    "stringValue": "/usr/bin/python3.8"
}

but I have no

/usr/bin/entrypoint.py

in the container (which explains the error message). This seems 99% like a bug, can you take a look? I do have a

v1.0.1

Flyte backend system up and running... I'll try the same thing there.

message has been deleted

message has been deleted