Hampus Rosvall
05/02/2022, 12:30 PMDylan Wilder
05/02/2022, 3:35 PMDylan Wilder
05/02/2022, 5:16 PMDylan Wilder
05/03/2022, 10:09 PMflytekit.current_context().execution_id
has a few issues
• during compile time it is a string of the form ex:local:local:local
• during execution time it is a struct of the form project, domain, name and the name portion is emptyGitHub
05/04/2022, 12:52 AMGitHub
05/04/2022, 3:20 PMBabis Kiosidis
05/05/2022, 10:18 AMarguments
field but struggling a bit with understanding if this is possible. 🧵Dylan Wilder
05/05/2022, 8:33 PMGitHub
05/05/2022, 9:52 PMGitHub
05/06/2022, 12:28 AMGitHub
05/06/2022, 8:28 PMYee
05/06/2022, 9:26 PMRobert Everson
05/10/2022, 11:35 PMflyte-cli parse-proto
would work to print out the raw values of a proto, is there a way to get the proto from the flyte metastore directly? I’m trying to add an architecture feature to flytekit, and flyteidl and flyteadmin have both been updated, but the value isn’t making it from my register step all the way to the pod runtime, so I’m trying to debug where the breakdown is happening. I’ve copied the closure field from the tasks table, but the proto parsing isn’t outputting any information for me.GitHub
05/11/2022, 1:58 AMSonja Ericsson
05/11/2022, 1:25 PMFLYTE_SDK_WORKFLOW_PACKAGES
to pyflyte workflows serialize
. Looks like that functionality was removed in this PR. Was this intended?GitHub
05/11/2022, 3:29 PMGitHub
05/11/2022, 10:13 PMGitHub
05/13/2022, 11:37 PMGreg Gydush
05/17/2022, 6:00 PMGreg Gydush
05/17/2022, 10:56 PMmypy
and have this issue with NamedTuple
(for named outputs). The example in the docs (shown below) fails mypy 0.910
Code from Flyte docs:
import typing
from flytekit import task
hello_output = typing.NamedTuple("OP", greet=str)
@task
def say_hello() -> hello_output:
return hello_output("hello world")
Error:
mint/workflows/tmp.py:5:16: error: Unexpected arguments to namedtuple() [misc]
hello_output = typing.NamedTuple("OP", greet=str)
^
mint/workflows/tmp.py: note: In function "say_hello":
mint/workflows/tmp.py:10:12: error: Too many arguments for "hello_output" [call-arg]
return hello_output("hello world")
^
GitHub
05/18/2022, 7:15 AMjeev
05/18/2022, 9:26 PMAlex Bain
05/19/2022, 10:09 AMFlyteRemote
and now I understand how to ask my question lol. My remote workflow looks like this:
@dataclass_json
@dataclass
class WrappedList:
s: str
@flytekit.workflow
def count_files_test_workflow(
in: WrappedList,
):
How do I call this workflow with FlyteRemote
? If I try to do:
remote = FlyteRemote(...)
lp = remote.fetch_launch_plan(...)
remote.execute(lp, inputs={'in': WrappedList("foo")})
Then I get FlyteTypeException: Type error! Received: <class '__main__.WrappedList'> with value: WrappedList(s='foo'), Expected: <class 'types.WrappedlistSchema'>
. I looked into the type_hints
argument to remote.execute
, but I don't think that's the right thing here. What I really need to be able to do is specify inputs={'in': x}
where x is an instance of types.WrappedListSchema.Stephan Gref
05/19/2022, 5:15 PM.
(dot, like current directory) well and does not download the tar.gz at all.
Reproduced this with this minimal example:
import os
from flytekit.tools.fast_registration import download_distribution as _download_distribution
os.environ["AWS_ACCESS_KEY_ID"] = "key_id"
os.environ["AWS_SECRET_ACCESS_KEY"] = "secret_access_key"
destination = "."
_download_distribution(
additional_distribution="<s3://bucket/fast-register.tar.gz>",
destination=".")
It works if I replace destination
with is ./
in this example. The issue is, that .
is set by propeller (this is at least my understanding). Is there any way I can configure that?Greg Gydush
05/19/2022, 5:32 PMGitHub
05/20/2022, 10:05 PMAlex Bain
05/22/2022, 2:25 AMv1.0.0
with flytekit==0.26.0
and Spark tasks are fine. I just moved up (last night) to flytekit==1.0.1
with flytekitplugins-spark==1.0.1
(with the same backend system) and now Spark tasks are broken with the error can't open file '/usr/bin/entrypoint.py': [Errno 2] No such file or directory
. I took a look and the old (good) tasks have the following set:
"mainApplicationFile": {
"stringValue": "local:///usr/local/lib/python3.8/dist-packages/flytekit/bin/entrypoint.py"
}
and that file exists and looks fine. However, my new (broken) Spark task shows the following:
"mainApplicationFile": {
"stringValue": "local:///usr/bin/entrypoint.py"
},
"executorPath": {
"stringValue": "/usr/bin/python3.8"
}
but I have no /usr/bin/entrypoint.py
in the container (which explains the error message). This seems 99% like a bug, can you take a look? I do have a v1.0.1
Flyte backend system up and running... I'll try the same thing there.GitHub
05/24/2022, 12:58 AMGitHub
05/25/2022, 8:28 AMGitHub
05/26/2022, 2:31 AMGitHub
05/26/2022, 2:31 AM