is there a way to set postgres and s3/minio passwords as secrets instead of them being in plain text in a deployment?

AttributeError

when fetching inputs from

node_execution

for which the optional parameter of type

FlyteFile

is

None

from attr import s
import flytekit
import flytekit.remote
import flytekit.configuration as flyte_config
from typing import List, NamedTuple, Optional


import dataclasses, dataclasses_json

@dataclasses_json.dataclass_json
@dataclasses.dataclass
class T2IO:
    a: str
    b: str
    z: Optional[flytekit.types.file.FlyteFile] = None


@flytekit.task
def task1(x: int) -> NamedTuple('op', t1=List[T2IO]):
    return [T2IO(str(t), str(-t)) for t in range(x)]


@flytekit.task
def task2(x: T2IO) -> NamedTuple('op', t2=T2IO):
    return T2IO(x.a+'0', x.b+'0')


@flytekit.task
def task3(x: List[T2IO]) -> NamedTuple('op', t3=str):
    return '-'.join([t.a for t in x]+[t.b for t in x])


@flytekit.workflow
def baby_training_wf(x: int) -> str:
    t1, = task1(x=x).with_overrides(node_name='t-1')
    t2 = flytekit.map_task(task2)(x=t1).with_overrides(node_name='t-2')
    t3, = task3(x=t2).with_overrides(node_name='t-3')
    return t3

if __name__ == '__main__':
    print(baby_training_wf(x=3))

fetched execution and tried accessing task2 inputs as

execution.node_executions['t-2'].inputs['x']

, got

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[308], line 1
----> 1 execution.node_executions['t-2'].inputs['x']

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:1691, in LiteralsResolver.__getitem__(self, key)
   1688 if key in self._native_values:
   1689     return self._native_values[key]
-> 1691 return self.get(key)

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:1720, in LiteralsResolver.get(self, attr, as_type)
   1718         else:
   1719             ValueError("as_type argument not supplied and Variable map not specified in LiteralsResolver")
-> 1720 val = TypeEngine.to_python_value(
   1721     self._ctx or FlyteContext.current_context(), self._literals[attr], cast(Type, as_type)
   1722 )
   1723 self._native_values[attr] = val
   1724 return val

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:831, in TypeEngine.to_python_value(cls, ctx, lv, expected_python_type)
    827 """
    828 Converts a Literal value with an expected python type into a python value.
    829 """
    830 transformer = cls.get_transformer(expected_python_type)
--> 831 return transformer.to_python_value(ctx, lv, expected_python_type)

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:1023, in ListTransformer.to_python_value(self, ctx, lv, expected_python_type)
   1021 else:
   1022     st = self.get_sub_type(expected_python_type)
-> 1023     return [TypeEngine.to_python_value(ctx, x, st) for x in lits]

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:1023, in <listcomp>(.0)
   1021 else:
   1022     st = self.get_sub_type(expected_python_type)
-> 1023     return [TypeEngine.to_python_value(ctx, x, st) for x in lits]

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:831, in TypeEngine.to_python_value(cls, ctx, lv, expected_python_type)
    827 """
    828 Converts a Literal value with an expected python type into a python value.
    829 """
    830 transformer = cls.get_transformer(expected_python_type)
--> 831 return transformer.to_python_value(ctx, lv, expected_python_type)

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:604, in DataclassTransformer.to_python_value(self, ctx, lv, expected_python_type)
    602 json_str = _json_format.MessageToJson(lv.scalar.generic)
    603 dc = cast(DataClassJsonMixin, expected_python_type).from_json(json_str)
--> 604 dc = self._fix_structured_dataset_type(expected_python_type, dc)
    605 return self._fix_dataclass_int(expected_python_type, self._deserialize_flyte_type(dc, expected_python_type))

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:392, in DataclassTransformer._fix_structured_dataset_type(self, python_type, python_val)
    390     for field in dataclasses.fields(python_type):
    391         val = python_val.__getattribute__(field.name)
--> 392         python_val.__setattr__(field.name, self._fix_structured_dataset_type(field.type, val))
    393 return python_val

File /opt/homebrew/Caskroom/miniconda/base/envs/baby/lib/python3.10/site-packages/flytekit/core/type_engine.py:391, in DataclassTransformer._fix_structured_dataset_type(self, python_type, python_val)
    389 elif dataclasses.is_dataclass(python_type):
    390     for field in dataclasses.fields(python_type):
--> 391         val = python_val.__getattribute__(field.name)
    392         python_val.__setattr__(field.name, self._fix_structured_dataset_type(field.type, val))
    393 return python_val

AttributeError: 'NoneType' object has no attribute 'path'

Hey we ran into an interesting problem today... we have a map task that calls a C binary that uses a large reference dataset from disk to do some computations on a new smaller dataset. It keeps failing with a rather mysterious error:

[0]: code:"ResourceDeletedExternally" message:"resource not found, name [e2e-workflows-development/fb2xnzxy-n2-0-0]. reason: pods \"fb2xnzxy-n2-0-0\" not found"

We then checked control plan logs and they suggested the pod was being evicted due to memory pressure (137 = k8s OOM status code):

"containerStatuses": [
                {
                    "name": "fb2xnzxy-n2-0-0",
                    "state": {
                        "terminated": {
                            "exitCode": 137,
....

However when we look at grafana, we see that memory used is really low, way below requests/limits... however, we found that the memory cache was quite high. We then found a k8s issue about memory cache being incorrectly counted as "used" memory by kubelet when it looks at memory pressure. Note quite a flyte issue, more of a k8s issue, but the log was a bit mysterious and we're still figuring out resolution.

Still struggling to get this working. flyte-binary now deploys and I can access the console via my IdP (i.e.

userAuth

), and I have successfully submitted a workflow (embedded OAuth2 –

appAuth

using defaults, i.e. not explicitly configured), however it stalls at that point producing the following error:

E0726 20:02:10.952926       2 workers.go:102] error syncing 'flytesnacks-development/fa2b0208403a24dd2b5a': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}]
{"json":{"src":"controller.go:159"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fa2b0208403a24dd2b5a]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","routine":"worker-8","src":"handler.go:181"},"level":"info","msg":"Processing Workflow.","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"executor.go:1112","wf":"flytesnacks:development:example.training_workflow"},"level":"info","msg":"Node not yet started, will not finalize","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881067","routine":"worker","src":"token_source_provider.go:230","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"failed to get token: %!w(*oauth2.RetrieveError=\u0026{0xc23d93a120 [123 34 ... 34 125]})","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"token_source_provider.go:230","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"failed to get token: %!w(*oauth2.RetrieveError=\u0026{0xc23d93a630 [123 34 ... 34 125]})","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"workflow_event_recorder.go:69","wf":"flytesnacks:development:example.training_workflow"},"level":"info","msg":"Failed to record workflow event [execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"fa2b0208403a24dd2b5a\" \u003e producer_id:\"propeller\" phase:FAILED occurred_at:\u003cseconds:1690401730 nanos:953387521 \u003e error:\u003ccode:\"Workflow abort failed\" message:\"Workflow[flytesnacks:development:example.training_workflow] failed. RuntimeExecutionError: max number of system retry attempts [32747/10] exhausted. Last known status message: Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\\nResponse: {\\\"error\\\":\\\"invalid_client\\\",\\\"error_description\\\":\\\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\\\"}]\" kind:SYSTEM \u003e ] with err: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"executor.go:351","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]]","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"controller.go:159"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fa2b0208403a24dd2b5a]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","routine":"worker-8","src":"handler.go:367"},"level":"info","msg":"Completed processing workflow.","ts":"2023-07-26T20:02:10Z"}

The best I can work out for this is that the service (flyte itself) is trying to authenticate to something and failing. What precisely is failing and what it is attempting to authenticate to is not clear.

Hi community, when I was using the command

flytectl create project --name flytesnacks --id flytesnacks --description "flytesnacks description" --labels app=flyte

from https://docs.flyte.org/projects/flytectl/en/stable/gen/flytectl_create_project.html to create a new project, I received

INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x101d7c384]

goroutine 1 [running]:
<http://github.com/flyteorg/flytectl/cmd/core.CommandContext.AdminClient(...)|github.com/flyteorg/flytectl/cmd/core.CommandContext.AdminClient(...)>
	/home/runner/work/flytectl/flytectl/cmd/core/cmd_ctx.go:57
<http://github.com/flyteorg/flytectl/cmd/create.createProjectsCommand({0x102e26518|github.com/flyteorg/flytectl/cmd/create.createProjectsCommand({0x102e26518>, 0x1400018a010}, {0x0?, 0x0?, 0x1023ba890?}, {0x0, {0x0, 0x0}, {0x0, 0x0}, ...})
	/home/runner/work/flytectl/flytectl/cmd/create/project.go:63 +0x104
<http://github.com/flyteorg/flytectl/cmd/core.generateCommandFunc.func1(0x140007eb680|github.com/flyteorg/flytectl/cmd/core.generateCommandFunc.func1(0x140007eb680>?, {0x14000269380, 0x0, 0x8})
	/home/runner/work/flytectl/flytectl/cmd/core/cmd.go:70 +0x758
<http://github.com/spf13/cobra.(*Command).execute(0x140007eb680|github.com/spf13/cobra.(*Command).execute(0x140007eb680>, {0x14000269300, 0x8, 0x8})
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856 +0x4c4
<http://github.com/spf13/cobra.(*Command).ExecuteC(0x140006baf00)|github.com/spf13/cobra.(*Command).ExecuteC(0x140006baf00)>
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x354
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
<http://github.com/flyteorg/flytectl/cmd.ExecuteCmd()|github.com/flyteorg/flytectl/cmd.ExecuteCmd()>
	/home/runner/work/flytectl/flytectl/cmd/root.go:137 +0x20
main.main()
	/home/runner/work/flytectl/flytectl/main.go:12 +0x1c

Does anyone know what's the potential issue?

[Namespaces for pods and projects] Hi, I can see that for each project, Flyte will create 3 different namespaces on k8s, corresponding to 3 domains of the project (e.g.,

flytesnacks-development

). This is normally okay. However, I’m deploying Flyte (via the

flyte-binary

Helm chart) on an on-prem k8s cluster where I only have permissions on one single namespace. As in my project, I need a private Docker image, I need to configure the

imagePullSecrets

for our service account, and I did it. However, when I tried to execute my workflows on Flyte, they fail because they couldn’t pull the private Docker image. My uneducated guess is that Flyte created seperate namespaces for the project, the service account might need to be configured with the

imagePullSecrets

for each of those namespaces as well. And since I don’t have permission on those namespace, I cannot do it. So my questions are: 1. Does my guess make sense? 2. Is there anyway to configure Flyte to not create new namespaces for project, but to use a single namespace?

Hi, I am wondering if there is an example/tutorial on multi-node multi-gpu training. I only see the single-node multi-gpu training example.

Hi all, I have a question about logging. I am using flyte's helm chart to deploy flyte-binary (chart version flyte-binary-v1.10.6). I've configured logging to level 4, which in my understanding is "error". I've done this via the yaml entry: configuration.logging.level = 4. To double check, I exec'ed into flyte's pod and made sure the in /etc/flyte/config.d/000-core.yaml, the value for logger.level is 4. However, my log file is filled with "info" level messages... Am I using the incorrect level? I want to only display errors and above.

Hey folks we recently upgraded copilot and tasks are failing with

[1/1] currentAttempt done. Last Error: USER::[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[flyte-copilotdownloader] terminated with exit code (1). Reason [Error]. Message: 
  --storage.cache.max_size_mbs int             Maximum size of the cache where the Blob store data is cached in-memory. If not specified or set to 0,  cache is not used
      --storage.cache.target_gc_percent int        Sets the garbage collection target percentage.

I took a look at the config in the docs and looks like we need to add these. Is there anything else I’m missing and will these default values work for our deployment?

co-pilot:
  cpu: 500m
  default-input-path: /var/flyte/inputs
  default-output-path: /var/flyte/outputs
  image: <http://cr.flyte.org/flyteorg/flytecopilot:v0.0.15|cr.flyte.org/flyteorg/flytecopilot:v0.0.15>
  input-vol-name: flyte-inputs
  memory: 128Mi
  name: flyte-copilot-
  output-vol-name: flyte-outputs
  start-timeout: 1m40s
  storage: ""

Hi everyone. I am trying to run

pyflyte run --remote awesome.py main_workflow

on flyte-binary deployed on aks. I am able to access the ui. my config is the standart one, although I tried some variations:

1 admin:
  2   # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  3   endpoint: dns:///flyte.<hostname>
  4   authType: Pkce
  5   insecure: false
  6 logger:
  7   show-source: true
  8   level: 0

when I run the command, I get

Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
	details: failed to connect to all addresses; last error: UNKNOWN: ipv4:<ip>:443: Cannot check peer: missing selected ALPN property.
	Debug string UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:<ip>:443: Cannot check peer: missing selected ALPN property. {grpc_status:14, created_time:"2023-08-16T16:11:24.832905759+00:00"}

We're using azure/application-gateway as ingress. I've read through the channel and saw similar issues, but non of the solutions seemed to solve the issue. Would be glad if someone has a hint 🙂