Hello everyone! Is there any way of mounting a data volume in a ContainerTask? I need to provide a directory with an arbitrary number of files, read them, and apply some processing. The way I do it ususally is by mounting a volume when running docker run, but now I’m transforming it to a Flyte pipeline and have had some issues. What I’m trying to do is to pass it as a FlyteDirectory, but it’s not working as I expected. I’m doing it like this:

ContainerTask(
    name="my_container",
    image="localhost:30000/my_container:latest",
    input_data_dir="/input",
    output_data_dir="/output",
    inputs=kwtypes(
        config_file=FlyteFile,
        my_dir=FlyteDirectory,),
    outputs=kwtypes(out=FlyteDirectory),
    command=[
        "python app.py --input_dir=/input/my_dir --config=/input/config_file"
    ]
)

I was expecting the

/input

directory to contain a dir called

my_dir

and a file called

config_file

. However, it seems to be empty. Can someone help me figure out how to mount data? Thanks 🙌

Hey we ran into an interesting problem today... we have a map task that calls a C binary that uses a large reference dataset from disk to do some computations on a new smaller dataset. It keeps failing with a rather mysterious error:

[0]: code:"ResourceDeletedExternally" message:"resource not found, name [e2e-workflows-development/fb2xnzxy-n2-0-0]. reason: pods \"fb2xnzxy-n2-0-0\" not found"

We then checked control plan logs and they suggested the pod was being evicted due to memory pressure (137 = k8s OOM status code):

"containerStatuses": [
                {
                    "name": "fb2xnzxy-n2-0-0",
                    "state": {
                        "terminated": {
                            "exitCode": 137,
....

However when we look at grafana, we see that memory used is really low, way below requests/limits... however, we found that the memory cache was quite high. We then found a k8s issue about memory cache being incorrectly counted as "used" memory by kubelet when it looks at memory pressure. Note quite a flyte issue, more of a k8s issue, but the log was a bit mysterious and we're still figuring out resolution.

Hi community, when I was using the command

flytectl create project --name flytesnacks --id flytesnacks --description "flytesnacks description" --labels app=flyte

from https://docs.flyte.org/projects/flytectl/en/stable/gen/flytectl_create_project.html to create a new project, I received

INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags. 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x101d7c384]

goroutine 1 [running]:
<http://github.com/flyteorg/flytectl/cmd/core.CommandContext.AdminClient(...)|github.com/flyteorg/flytectl/cmd/core.CommandContext.AdminClient(...)>
	/home/runner/work/flytectl/flytectl/cmd/core/cmd_ctx.go:57
<http://github.com/flyteorg/flytectl/cmd/create.createProjectsCommand({0x102e26518|github.com/flyteorg/flytectl/cmd/create.createProjectsCommand({0x102e26518>, 0x1400018a010}, {0x0?, 0x0?, 0x1023ba890?}, {0x0, {0x0, 0x0}, {0x0, 0x0}, ...})
	/home/runner/work/flytectl/flytectl/cmd/create/project.go:63 +0x104
<http://github.com/flyteorg/flytectl/cmd/core.generateCommandFunc.func1(0x140007eb680|github.com/flyteorg/flytectl/cmd/core.generateCommandFunc.func1(0x140007eb680>?, {0x14000269380, 0x0, 0x8})
	/home/runner/work/flytectl/flytectl/cmd/core/cmd.go:70 +0x758
<http://github.com/spf13/cobra.(*Command).execute(0x140007eb680|github.com/spf13/cobra.(*Command).execute(0x140007eb680>, {0x14000269300, 0x8, 0x8})
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856 +0x4c4
<http://github.com/spf13/cobra.(*Command).ExecuteC(0x140006baf00)|github.com/spf13/cobra.(*Command).ExecuteC(0x140006baf00)>
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x354
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
	/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
<http://github.com/flyteorg/flytectl/cmd.ExecuteCmd()|github.com/flyteorg/flytectl/cmd.ExecuteCmd()>
	/home/runner/work/flytectl/flytectl/cmd/root.go:137 +0x20
main.main()
	/home/runner/work/flytectl/flytectl/main.go:12 +0x1c

Does anyone know what's the potential issue?

Hi everyone. I am trying to run

pyflyte run --remote awesome.py main_workflow

on flyte-binary deployed on aks. I am able to access the ui. my config is the standart one, although I tried some variations:

1 admin:
  2   # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  3   endpoint: dns:///flyte.<hostname>
  4   authType: Pkce
  5   insecure: false
  6 logger:
  7   show-source: true
  8   level: 0

when I run the command, I get

Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.UNAVAILABLE
	details: failed to connect to all addresses; last error: UNKNOWN: ipv4:<ip>:443: Cannot check peer: missing selected ALPN property.
	Debug string UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:<ip>:443: Cannot check peer: missing selected ALPN property. {grpc_status:14, created_time:"2023-08-16T16:11:24.832905759+00:00"}

We're using azure/application-gateway as ingress. I've read through the channel and saw similar issues, but non of the solutions seemed to solve the issue. Would be glad if someone has a hint 🙂

Is there any examples of using ContainerTasks with FlyteFile and FlyteDirectory? I’m trying to use a ContainerTask that has a FlyteFile as an input, and the output is a directory but struggling to make anything work. Any pointers?

Hi, I am wondering if there is an example/tutorial on multi-node multi-gpu training. I only see the single-node multi-gpu training example.

Hi all! Can anyone tell me the difference between a sidecar task and python task? I can’t seem to find much doc about it. Thanks! My issue is that I don’t know how they are configured

Still struggling to get this working. flyte-binary now deploys and I can access the console via my IdP (i.e.

userAuth

), and I have successfully submitted a workflow (embedded OAuth2 –

appAuth

using defaults, i.e. not explicitly configured), however it stalls at that point producing the following error:

E0726 20:02:10.952926       2 workers.go:102] error syncing 'flytesnacks-development/fa2b0208403a24dd2b5a': Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}]
{"json":{"src":"controller.go:159"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fa2b0208403a24dd2b5a]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","routine":"worker-8","src":"handler.go:181"},"level":"info","msg":"Processing Workflow.","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"executor.go:1112","wf":"flytesnacks:development:example.training_workflow"},"level":"info","msg":"Node not yet started, will not finalize","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881067","routine":"worker","src":"token_source_provider.go:230","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"failed to get token: %!w(*oauth2.RetrieveError=\u0026{0xc23d93a120 [123 34 ... 34 125]})","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"token.go:37"},"level":"info","msg":"Error occurred in NewAccessRequest: invalid_client","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"token_source_provider.go:230","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"failed to get token: %!w(*oauth2.RetrieveError=\u0026{0xc23d93a630 [123 34 ... 34 125]})","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"workflow_event_recorder.go:69","wf":"flytesnacks:development:example.training_workflow"},"level":"info","msg":"Failed to record workflow event [execution_id:\u003cproject:\"flytesnacks\" domain:\"development\" name:\"fa2b0208403a24dd2b5a\" \u003e producer_id:\"propeller\" phase:FAILED occurred_at:\u003cseconds:1690401730 nanos:953387521 \u003e error:\u003ccode:\"Workflow abort failed\" message:\"Workflow[flytesnacks:development:example.training_workflow] failed. RuntimeExecutionError: max number of system retry attempts [32747/10] exhausted. Last known status message: Workflow[] failed. ErrorRecordingError: failed to publish event, caused by: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\\nResponse: {\\\"error\\\":\\\"invalid_client\\\",\\\"error_description\\\":\\\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\\\"}]\" kind:SYSTEM \u003e ] with err: EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","res_ver":"188881069","routine":"worker-8","src":"executor.go:351","wf":"flytesnacks:development:example.training_workflow"},"level":"warning","msg":"Event recording failed. Error [EventSinkError: Error sending event, caused by [rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method).\"}]]","ts":"2023-07-26T20:02:10Z"}
{"json":{"src":"controller.go:159"},"level":"info","msg":"==\u003e Enqueueing workflow [flytesnacks-development/fa2b0208403a24dd2b5a]","ts":"2023-07-26T20:02:10Z"}
{"json":{"exec_id":"fa2b0208403a24dd2b5a","ns":"flytesnacks-development","routine":"worker-8","src":"handler.go:367"},"level":"info","msg":"Completed processing workflow.","ts":"2023-07-26T20:02:10Z"}

The best I can work out for this is that the service (flyte itself) is trying to authenticate to something and failing. What precisely is failing and what it is attempting to authenticate to is not clear.

Hi. I'm getting OOMKilled when running the diabetes demo on Kubernetes. Describing the pod I see

Limits:
      cpu:     1
      memory:  200Mi
    Requests:
      cpu:     1
      memory:  200Mi

How can I increase this? My helm yaml already has

task_resource_defaults:
    task_resources:
      defaults:
        cpu: 1000m
        memory: 1000Mi
        storage: 1000Mi
      limits:
        storage: 2000Mi

Hey folks we recently upgraded copilot and tasks are failing with

[1/1] currentAttempt done. Last Error: USER::[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[flyte-copilotdownloader] terminated with exit code (1). Reason [Error]. Message: 
  --storage.cache.max_size_mbs int             Maximum size of the cache where the Blob store data is cached in-memory. If not specified or set to 0,  cache is not used
      --storage.cache.target_gc_percent int        Sets the garbage collection target percentage.

I took a look at the config in the docs and looks like we need to add these. Is there anything else I’m missing and will these default values work for our deployment?

co-pilot:
  cpu: 500m
  default-input-path: /var/flyte/inputs
  default-output-path: /var/flyte/outputs
  image: <http://cr.flyte.org/flyteorg/flytecopilot:v0.0.15|cr.flyte.org/flyteorg/flytecopilot:v0.0.15>
  input-vol-name: flyte-inputs
  memory: 128Mi
  name: flyte-copilot-
  output-vol-name: flyte-outputs
  start-timeout: 1m40s
  storage: ""