Ok, I’m making some progress now, but I’m stuck on the pods having permissions to pull images from our GCP artifact registry. I went through the process of patching the default k8s service account, but I’m still getting this 403 when the tasks try to run.

task submitted to K8s

[ContainersNotReady|ContainerCreating]: containers with unready status: [f787e4fb7001d459f851-n0-0]|

[ContainersNotReady|ErrImagePull]: containers with unready status: [f787e4fb7001d459f851-n0-0]|rpc error: code = Unknown desc = failed to pull and unpack image "<location>-docker.pkg.dev/<project>/<repository>/<image_name>:MHon8F_9TgvC55qoS5mUtw..": failed to resolve reference "<location>-docker.pkg.dev/<project>/<repository>/<image_name>:MHon8F_9TgvC55qoS5mUtw..": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://<location>-docker.pkg.dev/v2/token?scope=repository%3A<project>%2F<repository>%2F<image_name>%3Apull&service=<location>-docker.pkg.dev: 403 Forbidden

Here are the patch steps I followed: • Created service account and downloaded the .json key file •

kubectl create secret docker-registry artifact-json-key --docker-server=<http://pkg.dev|pkg.dev> --docker-username=_json_key --docker-password=(cat artifact_auth.json | string collect) --docker-email=<email>

•

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "artifact-json-key"}]}'

I know resource based overrides work with

map_task

but the newer feature of overriding

container_image

doesn't seem to work with map_tasks. Here's a minimal example where its still using the default container image and not the overwritten one. It does work with regular tasks, just not map_task?

@task
def my_mappable_task(a: int) -> typing.Optional[str]:
    return str(a)

@workflow
def my_map_wf(x: typing.List[int]) -> typing.List[typing.Optional[str]]:
    return map_task(
        my_mappable_task,
        concurrency=10,
        min_success_ratio=0.75,
    )(a=x).with_overrides(container_image="random:image")

cc @Kevin Su since he added the original feature https://github.com/flyteorg/flytekit/commit/eafcc820303367749e63edc62190b9153fd6be5e

hello, I am trying to use spark-operator with flyte. However, I cannot pull the image.

docker pull gcr.io/spark-operator/spark-operator:v1beta2-1.2.0-3.0.0
Error response from daemon: manifest for gcr.io/spark-operator/spark-operator:v1beta2-1.2.0-3.0.0 not found: manifest unknown: Failed to fetch "v1beta2-1.2.0-3.0.0" from request "/v2/spark-operator/spark-operator/manifests/v1beta2-1.2.0-3.0.0".
(base) ➜  mlforge git:(flyte-deployment-2) ✗ docker pull gcr.io/spark-operator/spark-operator
Using default tag: latest
Error response from daemon: manifest for gcr.io/spark-operator/spark-operator:latest not found: manifest unknown: Failed to fetch "latest" from request "/v2/spark-operator/spark-operator/manifests/latest".

I got the image repo url from https://github.com/helm/charts/blob/master/incubator/sparkoperator/values.yaml, and image tag from https://googlecloudplatform.github.io/spark-on-k8s-operator/ Could anyone provide a working image tag? Thanks!

Hi guys, is it possible to trigger flyte workflow remotely from other service like a python code? I have a workflow that reads configuration files using hydra and I want to trigger the workflow from a separate python code

Hello, I am trying to run the ImageSpec example on MacBook M1 given on the following page: https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/customizing_dependencies/image_spec.html I want to push the Image to my GCP project where I have rights to push the images. If I build the images using docker and push it to my GCP project then all work perfectly without any error. At the end of building the image, I am getting the following error:

> apt-get install git:

#5 17.22 perl: warning: Please check that your locale settings:

#5 17.22 	LANGUAGE = (unset),

#5 17.22 	LC_ALL = "en_US.UTF-8",

#5 17.22 	LANG = "C.UTF-8"

#5 17.22   are supported and installed on your system.

#5 17.23 perl: warning: Falling back to a fallback locale ("C.UTF-8").

#5 17.64 debconf: delaying package configuration, since apt-utils is not installed

#5 17.71 Fetched 8904 kB in 1s (10.5 MB/s)

#5 17.76 Error while loading /usr/sbin/dpkg-split: No such file or directory

Sub-process /usr/bin/dpkg returned an error code (1)

------

Failed with Unknown Exception <class 'Exception'> Reason: failed to run command envd build --path /var/folders/01/m569sr9x04j6x8_sr_n1c03r06hms7/T/flyte-qp2ow6ji/sandbox/local_flytekit/13832f2987526bc535adaf7cc3e6fb55 --platform linux/amd64 --output type=image,name=<http://europe-west4-docker.pkg.dev/my-gcp-project/flyte/zeetesting/flytekit:L0sYlwiX5Ne76o2wlm8Y2w..,push=true|europe-west4-docker.pkg.dev/my-gcp-project/flyte/zeetesting/flytekit:L0sYlwiX5Ne76o2wlm8Y2w..,push=true> with error b'time="2023-11-29T10:20:40+01:00" level=fatal msg="failed to build the image: failed to build: failed to wait error group: failed to solve LLB: failed to solve: process \\"/dev/.buildkit_qemu_emulator bash -c apt-get update && apt-get install -y --no-install-recommends git\\" did not complete successfully: exit code: 100"\n'

Additional Information: Docker Desktop (v 4.25.0) Flyte v 1.10.0 Python 3.11 Previously it was working for me but now we have updated the fylte version and I am getting this error. Any idea, how can I solve this issue ?

Lets say you have a task that you need to run 100.000 times or more. How would you do it? We thought about MapTask after reading this blogpost but it seems like there are issues with parellelizing maptasks over very large lists, we encountered Problems when trying to parallelize on 20.000 keys already with the experimental MapTask. It seems like the non-experimental MapTask has a hard limit on 5.000 tasks. I am a bit suprised by this, as the blogpost and the documentation of MapTask seem to not mention anything in that direction. Is a dynamic workflow that schedules 100.000 tasks a better idea?

Hey everyone 🙂 I have a question about the

pyflyte run --max-parallelism

option of executions. Even when setting it to 1, I see multiple pods scheduled at the same time while I would have expected only a single one. Am I misunderstanding this option? Is the same for you as well?

hey! How are you? I'm wondering when the next flyte monorepo release will be launched. is there any releases schedule/interval? Thanks!!

Hi folks -- just watching resources in my cluster and I'm seeing that flytepropeller pods seem to be terminating / restarting due to leader election failures like:

{"json":{},"level":"fatal","msg":"Lost leader state. Shutting down.","ts":"2023-11-29T150420Z"}

Is this common... or indicative of something that could be misbehaving within the cluster? (The cluster is mostly idling)

Please could someone explain which cache the docs are referring too here

Another area of slowdown could be the size of the input-output cache that FlytePropeller maintains in-memory. This can be configured, while configuring the storage for FlytePropeller. Rule of thumb, for FlytePropeller with x memory limit, allocate x/2 to the cache

https://docs.flyte.org/en/latest/deployment/configuration/performance.html#manual-scale-out

Hello, My flyte python tasks are started fast enough in the EKS cluster. However, when the tasks are requesting more memory, e.g. requests=40Gi, it took a long time to initialize the ephemeral pod. What are the strategies to mitigate that? And is there anything wrong with my configs:

task_resources:
                defaults:
                  cpu: 2
                  memory: 1Gi
                  storage: 1Gi
                limits:
                  cpu: 20
                  memory: 500Gi
                  storage: 100Gi
                  gpu: 1
...
          cluster_resource_manager:
                customData:
                  - development:
                      - projectQuotaCpu:
                          value: "800"
                      - projectQuotaMemory:
                          value: "3200Gi"

I heard flyte agent service reduces the overhead of creating a pod for each task. Is it applicable in python task’s case, or only to specific tasks like databricks-plugin & bigquery_query_job_task? And how do I switch to using agent-service for python tasks?

Hello, I face a new issue, with the basic sample app.py I can execute

unionml train app:model -i '{"hyperparameters": {"C": 1.0, "max_iter": 1000}}'

, and the workflow execute and run OK in flyte, but when I want to predict

unionml predict app:model -f data/sample_features.json

I've got an error :

TypeError: No automatic conversion found from type <class 'sklearn.linear_model._logistic.LogisticRegression'> to FlyteFile.Supported (os.PathLike, str, Flytefile)

which is a flytekit originated error, my config is unionml 0.2.1, flyteidl 1.5.21, flytekit 1.9.1, scikit-learn 1.3.2, with flyte-binary I'm stuck with this issue, any idea on how to fix, by specifying a type somewhere ? or is it a bug due to some new version of packages ? thanks in advance for any help or hint, at least to workaround this problem, or better I use the s3 storage to store and link the features file ? Slack Conversation

Hello dear Flyte community 🙂 I have a question about data management... Is there a way to delete old/unwanted data from Flyte's bucket ? From what I can read from the discussions here, there is no feature to enable deletion of workflow executions & their artifacts. https://flyte-org.slack.com/archives/CP2HDHKE1/p1696841258825329?thread_ts=1696839023.942149&cid=CP2HDHKE1 Is that really the case ? Otherwise, how do you proceed to delete something ?

Good afternoon. Does Flyte allow the allocation of resources (gpu/cpu) per project? I'd like to be able to optimize my resource usage through different projects using a quota system or something similar to it

Hi peeps, is ther ea way to instrument workflows ?

Hi All, We are currently using

default-node-selector:

in the k8s plugin config to select nodes on our MNG for flyte workers pods. This allows us to use either spot nodes of on-demand nodes on AWS. We want to set up nodeAffinity selectors with weights to try to use spot first and if not available, go to on-demand nodes. We are using flyte-binary chart and the

deployment

section in values.yaml has a

extraPodSpec

which I believe only applies to the flyte-binary pods. Are pod templates the only way to apply nodeAffinity selection rules to worker pods? Or is there any way to apply these via the values.yaml for flyte-binary?

Quick q -- has anyone sen

pyflyte run --remote

generating an incorrect URL? I assume there's a config value set somewhere that's incorrect -- I'm seeing

console

twice in the URI like this: https://mydomain.com/console/console/projects/flytesnacks/domains/staging/executions/f3974b92e876344a3840

If I want to run a workflow using custom container with flytectl demo, is it possible to mount host volume to the container/flytectl?

Hi all, I am trying to use ktensorflow.TFJob wit a custom image. However I'm not sure what exactly gets executed. So here is a rough outline of the code:

@task(
    task_config=TfJob(
        chief=Chief(replicas=1, image=<some_image>),
        ps=PS(replicas=0, image=<some_image>),
        worker=Worker(replicas=1, image=<some_image>),
    ),
)
def func(some_arg: int):
    do somethign...

My question is, does the image <some_image> get run on each pod, if so when does

func

get run. Also if <some_image> is just run (and its entrypoint executed) how can I pass args to it?

👋 hey everyone, we're using release 1.9 and found that for flyte console, disabling a feature flag via

HORIZONTAL_LAYOUT

env var doesn't seem to work, but setting it to false via the URL parameter

horizontal-layout

works (found a related issue: 4047#issuecomment-1723528573) - wanted to check if this is a bug with the release

Hi, I'm new to Flyte and am looking to use it to replace my company's data warehousing pipeline. I have a few questions and am looking to see if this is the right place to ask them and if not, then where?

Hi, I'm exploring Flyte and I'm curious about whether or not it's possible to define Workflows in Flyte in Go code (considering Flyte is written in Go itself). I know you can define Go code that runs in Docker containers using the ContainerTask, and there are SDKs for Python and Scala, but is there a way to define userland workflow code in Go?

Hi, I am running a map_task on an input list of 2 elements, but the UI shows 4 successfull runs, with only 2 unique task ids. Does anyone know, what could cause this behavior / is it a bug in the frontend? On the cluster, only two pods are scheduled: (flytekit==1.10.1, flyte-backend==1.9.0)

I'm trying to enable AWS Secrets Manager in

flyte-binary

I've added this to my values.yaml:

configuration:
  inline:
    webhook:
      secretManagerType: "AWS"

Where do I put the AWS service account credentials? And may I specify different credentials on different project/environment combinations? I'd expect to have to put a list of credentials in somewhere, like this for each project/environment:

"AWS_ACCESS_KEY_ID": "some-value"
"AWS_SECRET_ACCESS_KEY": "some-value",
"AWS_DEFAULT_REGION": "some-value"

Thank you for taking a look.

Hello, I deployed spark-operator in EKS cluster for flyte. When I run a spark task for the first time in this newly setup flyte cluster, the spark-operator pod gave the following error. Does anyone has idea on how to debug or resolve this? I am on the latest spark-operator ghcr.io/googlecloudplatform/spark-operator:v1beta2-1.3.8-3.1.1 And the flyte version is 1.10.

Where can I find more specific error message?

Thanks in advance!

Hello, The spark task is automatically being assigned service account: spark, even if I don’t specify it in command line with:

pyflyte register .... --service-account spark

And when the task ephemeral pod is having service account = spark, as opposed to service account = default, that task pod is not having permission to access S3. Where in the flyte-core helm charts should the fixes made? Maybe the default for this section needs to be corrected?

- key: ad_spark_service_account
      value: |
        apiVersion: v1
        kind: ServiceAccount
        metadata:
          name: spark
          namespace: {{ namespace }}
    - key: ae_spark_role_binding
      value: |
        apiVersion: rbac.authorization.k8s.io/v1
        kind: RoleBinding
        metadata:
          name: spark-role-binding
          namespace: {{ namespace }}
        roleRef:
          apiGroup: rbac.authorization.k8s.io
          kind: Role
          name: spark-role
        subjects:
        - kind: ServiceAccount
          name: spark
          namespace: {{ namespace }}

I got this. The service account for spark is not having aws iam-role set, compared to the default service account.

kubectl describe sa -n flytesnacks-development
Name:                default
Namespace:           flytesnacks-development
Labels:              <none>
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::245085526351:role/flyte-role
Image pull secrets:  artifactory-da-reader-token
Mountable secrets:   default-token-x5dbz
Tokens:              <none>
Events:              <none>


Name:                spark
Namespace:           flytesnacks-development
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>