Hi, I am running a workflow locally on my machine using

pyflyte run

or

python -m

but, I am getting this error -

2023-06-20 17:04:51,354876 ERROR    {"asctime": "2023-06-20 17:04:51,354", "name": "flytekit", "levelname":  base_task.py:587
                                    "ERROR", "message": "Failed to convert outputs of task 'read_dataset' at                 
                                    position 0:\n  [Errno 28] Error writing bytes to file. Detail: [errno                    
                                    28] No space left on device"}

Is there a virtual directory or disk flyte creates when running on a local machine which I can purge? The data for training is being pulled from s3.

[Namespaces for pods and projects] Hi, I can see that for each project, Flyte will create 3 different namespaces on k8s, corresponding to 3 domains of the project (e.g.,

flytesnacks-development

). This is normally okay. However, I’m deploying Flyte (via the

flyte-binary

Helm chart) on an on-prem k8s cluster where I only have permissions on one single namespace. As in my project, I need a private Docker image, I need to configure the

imagePullSecrets

for our service account, and I did it. However, when I tried to execute my workflows on Flyte, they fail because they couldn’t pull the private Docker image. My uneducated guess is that Flyte created seperate namespaces for the project, the service account might need to be configured with the

imagePullSecrets

for each of those namespaces as well. And since I don’t have permission on those namespace, I cannot do it. So my questions are: 1. Does my guess make sense? 2. Is there anyway to configure Flyte to not create new namespaces for project, but to use a single namespace?

Hi all, I have a question about logging. I am using flyte's helm chart to deploy flyte-binary (chart version flyte-binary-v1.10.6). I've configured logging to level 4, which in my understanding is "error". I've done this via the yaml entry: configuration.logging.level = 4. To double check, I exec'ed into flyte's pod and made sure the in /etc/flyte/config.d/000-core.yaml, the value for logger.level is 4. However, my log file is filled with "info" level messages... Am I using the incorrect level? I want to only display errors and above.

is there a way to set postgres and s3/minio passwords as secrets instead of them being in plain text in a deployment?

Hi community, when I was using the command

flytectl create project --name flytesnacks --id flytesnacks --description "flytesnacks description" --labels app=flyte

from https://docs.flyte.org/projects/flytectl/en/stable/gen/flytectl_create_project.html to create a new project, I received

INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x101d7c384]

goroutine 1 [running]:
<http://github.com/flyteorg/flytectl/cmd/core.CommandContext.AdminClient(...)|github.com/flyteorg/flytectl/cmd/core.CommandContext.AdminClient(...)>
	/home/runner/work/flytectl/flytectl/cmd/core/cmd_ctx.go:57
<http://github.com/flyteorg/flytectl/cmd/create.createProjectsCommand({0x102e26518|github.com/flyteorg/flytectl/cmd/create.createProjectsCommand({0x102e26518>, 0x1400018a010}, {0x0?, 0x0?, 0x1023ba890?}, {0x0, {0x0, 0x0}, {0x0, 0x0}, ...})
	/home/runner/work/flytectl/flytectl/cmd/create/project.go:63 +0x104
<http://github.com/flyteorg/flytectl/cmd/core.generateCommandFunc.func1(0x140007eb680|github.com/flyteorg/flytectl/cmd/core.generateCommandFunc.func1(0x140007eb680>?, {0x14000269380, 0x0, 0x8})
	/home/runner/work/flytectl/flytectl/cmd/core/cmd.go:70 +0x758
<http://github.com/spf13/cobra.(*Command).execute(0x140007eb680|github.com/spf13/cobra.(*Command).execute(0x140007eb680>, {0x14000269300, 0x8, 0x8})
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856 +0x4c4
<http://github.com/spf13/cobra.(*Command).ExecuteC(0x140006baf00)|github.com/spf13/cobra.(*Command).ExecuteC(0x140006baf00)>
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x354
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
<http://github.com/flyteorg/flytectl/cmd.ExecuteCmd()|github.com/flyteorg/flytectl/cmd.ExecuteCmd()>
	/home/runner/work/flytectl/flytectl/cmd/root.go:137 +0x20
main.main()
	/home/runner/work/flytectl/flytectl/main.go:12 +0x1c

Does anyone know what's the potential issue?

AttributeError

when fetching inputs from

node_execution

for which the optional parameter of type

FlyteFile

is

None

from attr import s
import flytekit
import flytekit.remote
import flytekit.configuration as flyte_config
from typing import List, NamedTuple, Optional


import dataclasses, dataclasses_json

@dataclasses_json.dataclass_json
@dataclasses.dataclass
class T2IO:
    a: str
    b: str
    z: Optional[flytekit.types.file.FlyteFile] = None


@flytekit.task
def task1(x: int) -> NamedTuple('op', t1=List[T2IO]):
    return [T2IO(str(t), str(-t)) for t in range(x)]


@flytekit.task
def task2(x: T2IO) -> NamedTuple('op', t2=T2IO):
    return T2IO(x.a+'0', x.b+'0')


@flytekit.task
def task3(x: List[T2IO]) -> NamedTuple('op', t3=str):
    return '-'.join([t.a for t in x]+[t.b for t in x])


@flytekit.workflow
def baby_training_wf(x: int) -> str:
    t1, = task1(x=x).with_overrides(node_name='t-1')
    t2 = flytekit.map_task(task2)(x=t1).with_overrides(node_name='t-2')
    t3, = task3(x=t2).with_overrides(node_name='t-3')
    return t3

if __name__ == '__main__':
    print(baby_training_wf(x=3))

fetched execution and tried accessing task2 inputs as

execution.node_executions['t-2'].inputs['x']

, got

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[308], line 1
----> 1 execution.node_executions['t-2'].inputs['x']

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:1691, in LiteralsResolver.__getitem__(self, key)
   1688 if key in self._native_values:
   1689     return self._native_values[key]
-> 1691 return self.get(key)

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:1720, in LiteralsResolver.get(self, attr, as_type)
   1718         else:
   1719             ValueError("as_type argument not supplied and Variable map not specified in LiteralsResolver")
-> 1720 val = TypeEngine.to_python_value(
   1721     self._ctx or FlyteContext.current_context(), self._literals[attr], cast(Type, as_type)
   1722 )
   1723 self._native_values[attr] = val
   1724 return val

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:831, in TypeEngine.to_python_value(cls, ctx, lv, expected_python_type)
    827 """
    828 Converts a Literal value with an expected python type into a python value.
    829 """
    830 transformer = cls.get_transformer(expected_python_type)
--> 831 return transformer.to_python_value(ctx, lv, expected_python_type)

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:1023, in ListTransformer.to_python_value(self, ctx, lv, expected_python_type)
   1021 else:
   1022     st = self.get_sub_type(expected_python_type)
-> 1023     return [TypeEngine.to_python_value(ctx, x, st) for x in lits]

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:1023, in <listcomp>(.0)
   1021 else:
   1022     st = self.get_sub_type(expected_python_type)
-> 1023     return [TypeEngine.to_python_value(ctx, x, st) for x in lits]

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:831, in TypeEngine.to_python_value(cls, ctx, lv, expected_python_type)
    827 """
    828 Converts a Literal value with an expected python type into a python value.
    829 """
    830 transformer = cls.get_transformer(expected_python_type)
--> 831 return transformer.to_python_value(ctx, lv, expected_python_type)

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:604, in DataclassTransformer.to_python_value(self, ctx, lv, expected_python_type)
    602 json_str = _json_format.MessageToJson(lv.scalar.generic)
    603 dc = cast(DataClassJsonMixin, expected_python_type).from_json(json_str)
--> 604 dc = self._fix_structured_dataset_type(expected_python_type, dc)
    605 return self._fix_dataclass_int(expected_python_type, self._deserialize_flyte_type(dc, expected_python_type))

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:392, in DataclassTransformer._fix_structured_dataset_type(self, python_type, python_val)
    390     for field in dataclasses.fields(python_type):
    391         val = python_val.__getattribute__(field.name)
--> 392         python_val.__setattr__(field.name, self._fix_structured_dataset_type(field.type, val))
    393 return python_val

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:391, in DataclassTransformer._fix_structured_dataset_type(self, python_type, python_val)
    389 elif dataclasses.is_dataclass(python_type):
    390     for field in dataclasses.fields(python_type):
--> 391         val = python_val.__getattribute__(field.name)
    392         python_val.__setattr__(field.name, self._fix_structured_dataset_type(field.type, val))
    393 return python_val

AttributeError: 'NoneType' object has no attribute 'path'

Still struggling to get this working. flyte-binary now deploys and I can access the console via my IdP (i.e.

userAuth

), and I have successfully submitted a workflow (embedded OAuth2 –

appAuth

using defaults, i.e. not explicitly configured), however it stalls at that point producing the following error:

E0726 20:02:10.952926       2 workers.go:102] error syncing 'flytesnacks-development/fa2b0208403a24dd2b5a': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}]
{"json":{"src":"controller.go:159"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fa2b0208403a24dd2b5a]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","routine":"worker-8","src":"handler.go:181"},"level":"info","msg":"Processing Workflow.","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"executor.go:1112","wf":"flytesnacks:development:example.training_workflow"},"level":"info","msg":"Node not yet started, will not finalize","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881067","routine":"worker","src":"token_source_provider.go:230","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"failed to get token: %!w(*oauth2.RetrieveError=\u0026{0xc23d93a120 [123 34 ... 34 125]})","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"token_source_provider.go:230","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"failed to get token: %!w(*oauth2.RetrieveError=\u0026{0xc23d93a630 [123 34 ... 34 125]})","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"workflow_event_recorder.go:69","wf":"flytesnacks:development:example.training_workflow"},"level":"info","msg":"Failed to record workflow event [execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"fa2b0208403a24dd2b5a\" \u003e producer_id:\"propeller\" phase:FAILED occurred_at:\u003cseconds:1690401730 nanos:953387521 \u003e error:\u003ccode:\"Workflow abort failed\" message:\"Workflow[flytesnacks:development:example.training_workflow] failed. RuntimeExecutionError: max number of system retry attempts [32747/10] exhausted. Last known status message: Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\\nResponse: {\\\"error\\\":\\\"invalid_client\\\",\\\"error_description\\\":\\\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\\\"}]\" kind:SYSTEM \u003e ] with err: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"executor.go:351","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]]","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"controller.go:159"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fa2b0208403a24dd2b5a]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","routine":"worker-8","src":"handler.go:367"},"level":"info","msg":"Completed processing workflow.","ts":"2023-07-26T20:02:10Z"}

The best I can work out for this is that the service (flyte itself) is trying to authenticate to something and failing. What precisely is failing and what it is attempting to authenticate to is not clear.

Hi all! Can anyone tell me the difference between a sidecar task and python task? I can’t seem to find much doc about it. Thanks! My issue is that I don’t know how they are configured

Hi, I am wondering if there is an example/tutorial on multi-node multi-gpu training. I only see the single-node multi-gpu training example.

Hello, I have defined a Flyte workflow which calls a few flyte Tasks. All but one tasks return a string, and one task,

ingest_reports

, returns a

List[Dict[str, str]]

. Furthermore in my workflow, I have

>>

operator defining the sequence of task execution. Now when I test the workflow, I have a couple of questions: 1. When I test the workflow, I want to mock the task calls. Now,

>>

operator defines the sequence of task execution, and it expects the tasks to return a Promise. Since a Promise is expected, so the mocked task should be able to return a Promise, otherwise this operation fails with an error like:

except Exception as exc:
>                   raise type(exc)(f"Error encountered while executing '{fn_name}':\n  {exc}") from exc
E                   AttributeError: Error encountered while executing 'phishing_reports_workflow':
E                     'list' object has no attribute 'ref'

2. To do so I defined the mock as below:

def mock_promise(value: any):
        magic_mock_promise = MagicMock(spec=Promise)
        mock_output = MagicMock()
        mock_output.value = value
        magic_mock_promise.output = mock_output
        return magic_mock_promise

3. This mocking of tasks to return a FlytePromise works fine as long as task return type is a String. For the task,

ingest_reports

, return type is

List[Dict[str, str]]

. When I try to mock the response of

ingest_reports

using

mock_promise

above, by passing in a list of dict of strings, it fails with exception below:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Typed List Transforms (<class 'list'>) to Flyte native
ctx = FlyteContext(file_access=<flytekit.core.data_persistence.FileAccessProvider object at 0x103772730>, level=1, flyte_cli...tackframe=<FrameSummary file /Library/Python/3.9/site-packages/flytekit/core/context_manager.py, line 877 in <module>>)
python_val = <MagicMock name='ingest_phishing_reports()' spec='Promise' id='5383586432'>
python_type = typing.List[typing.Dict[str, str]]
expected = <FlyteLiteral collection_type { map_value_type { simple: STRING } }>

    def to_literal(self, ctx: FlyteContext, python_val: T, python_type: Type[T], expected: LiteralType) -> Literal:
        if type(python_val) != list:
>           raise TypeTransformerFailedError("Expected a list")
E           flytekit.core.type_engine.TypeTransformerFailedError: Expected a list

/Library/Python/3.9/site-packages/flytekit/core/type_engine.py:1042: TypeTransformerFailedError

Any help on how to solve this?