(Caching) Hey all, I rewrote one of my company’s critical workflows in Flyte and it works well. However it frequently runs with multiple repetitive sets of inputs. For example, it may run 50 times a week with (mostly the) same 2-3 sets of inputs. From my understanding, cache_version only supports one version, so both can’t be cached at the same time. If I were to run a workflow with the first set of inputs, then the second, then the third, and repeated it 10 times, caching would effectively be disabled, because the cache keys keep changing, right? Is there any workaround or other solution to this problem? I’d also like to configure cache staleness so cache can expire after a while, I tried to achieve this using lifecycle rules on my cloud storage but in case the cloud deletes the cached outputs, Flyte still tries to fetch them and errors on it. Let me know how you dealt with this issue in your cases, thanks

Hello all I'm having an issue when trying to use reference tasks... When building image, I'm getting this error:

'ReferenceTask' object has no attribute '_task_function'

... Can someone help me? I would be very appreciated

so absolutely dying trying to get this deployed in EKS -- namely on minio charts are effectively the same so I am not sure why minio all the sudden just doesnt work

flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmp030l19_7/script_mode.tar.gz to <http://localhost:30084/my-s3-bucket/a/b/PJPRTCSJDCALQRYI44IFNWNQ5M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230315%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230315T152958Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=9b449916e2b9ca077d45255d05f86fec463c3e999e14c2c38a0075928906afa0> (recursive=False).

Original exception: HTTPConnectionPool(host='localhost', port=30084): Max retries exceeded with url: /my-s3-bucket/a/b/PJPRTCSJDCALQRYI44IFNWNQ5M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230315%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230315T152958Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=9b449916e2b9ca077d45255d05f86fec463c3e999e14c2c38a0075928906afa0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f205d87edc0>: Failed to establish a new connection: [Errno 111] Connection refused'))

noticed it was using a signed url override (not sure why it's doing that all of the sudden) and pretty much anything I change/update in the charts just doesnt apply, for example:

storage:
      # -- Sets the storage type. Supported values are sandbox, s3, gcs and custom.
      type: sandbox 
      # -- bucketName defines the storage bucket flyte will use. Required for all types except for sandbox.
      bucketName: my-s3-bucket 
      # -- settings for storage type s3
      signedUrl:
        stowConfigOverride:
          endpoint: <http://minio.flyte.svc.cluster.local:9000>

anyone have any thoughts?

Hey all! I have Flyte in a K8s cluster with Prometheus scraping Kubelet/cAdvisor to get metrics such as

container_cpu_usage_seconds_total

and

kube_pod_container_resource_limits_memory_bytes

. I am trying to monitor these metrics for all pods that are generated by Flyte executions. It is kind of working but sometimes an execution is simply not scraped by Prometheus: the metric is empty. I have a workflow that runs daily and today's execution's metrics are there, but yesterday's are not. As it's the same workflow and it has not changed since yesterday I really don't know what could be happening. Can anyone help me? This is the Prometheus config file regarding kubelet:

kubelet:
  enabled: true
  namespace: kube-system

  serviceMonitor:

    interval: ""

    proxyUrl: ""

    https: true

    cAdvisor: true

    probes: true

    resource: false

    resourcePath: "/metrics/resource/v1alpha1"

    cAdvisorMetricRelabelings: []

    probesMetricRelabelings: []

    cAdvisorRelabelings:
      - sourceLabels: [__metrics_path__]
        targetLabel: metrics_path

    probesRelabelings:
      - sourceLabels: [__metrics_path__]
        targetLabel: metrics_path

    resourceRelabelings:
      - sourceLabels: [__metrics_path__]
        targetLabel: metrics_path

    metricRelabelings: []

    relabelings:
      - sourceLabels: [__metrics_path__]
        targetLabel: metrics_path

Good morning, I have been looking for a way to run backfill jobs. Typically we run daily batch jobs, which will work well with the Launch Plan. But being able to process old datasets each identified by a date is important (schemas change, things fail, etc.) I have been trying to generate lists of dates to process (something like generate_dates(start_date, num_days_back) -> List[str]). If I hardcode a for loop (for i in range(10)) and create dates that way with no particular inputs the workflow executes correctly, but it would be really nice to be able to take start date and number of days back as an input to the workflow (the blocker is that we cannot iterate over a Promise). I see some discussion on backfilling on Slack, so I am probably just missing something obvious. Is this already baked into LaunchPlan? Would love to hear how others have solved this before 🙂 This is the last major hurdle before we are ready to put Flyte into prod - it’s such an improvement to Luigi.

Hello, I'm trying to run a demo workflow with a custom image in local sandbox, but am having authentication issues puling the image from Azure container registry. I initialized the sandbox with

flytectl sandbox start

I created a secrets file

acr-secrets-flytesnacks-development.yaml

apiVersion: v1
data:
  .dockerconfigjson: ***Base64 encoded json***
kind: Secret
metadata:
  name: acr-pull-credentials
  namespace: flyte-development
type: <http://kubernetes.io/dockerconfigjson|kubernetes.io/dockerconfigjson>

I applied the secret in the project-domain namespace with

kubectl -n flytesnacks-development apply -f secrets/acr-secrets-flytesnacks-development.yaml

I patched the default service account within the project-domain namespace :

kubectl -n flytesnacks-development patch serviceaccount default -p '{"imagePullSecrets": [{"name": "acr-pull-credentials"}]}'

I run my workflow with

pyflyte run --remote --image ***.<http://azurecr.io/databricks_workflow:latest|azurecr.io/databricks_workflow:latest> databricks_wf.py databricks_workflow --sql 'select...'

The pods get stuck in pending and describe shows the following issue:

Failed to pull image "***.<http://azurecr.io/databricks_workflow:latest|azurecr.io/databricks_workflow:latest>": rpc error: code = Unknown desc = Error response from daemon: Head "https://*****.<http://azurecr.io/v2/databricks_workflow/manifests/latest|azurecr.io/v2/databricks_workflow/manifests/latest>": unauthorized: authentication required, visit <https://aka.ms/acr/authorization> for more information.

Placing the decoded secret in .docker/config.json allows

docker pull

to work locally. Thanks for a any suggestions!

Hi, I am trying to assign GPU (Nvidia MIG - Multi-Instance GPU) on flyte. However, I cannot see how I assign it on each tasks in flyte (there were no mentions in flyte docs). Does anyone have a solution to it?

Whenever I package using the flag

--fast

and then run register the actual registration takes ~1 minute compared to instant when not using fast packaging. Is this expected or is there a way to speed up this process? :)

Anyone else having dependency conflicts when pip installing from flytekit master? I get:

ERROR: Could not find a version that satisfies the requirement grpcio<2.0,>=1.45.0 (from flytekit) (from versions: 0.4.0a0, 0.4.0a1, 0.4.0a2, 0.4.0a3, 0.4.0a4, 0.4.0a5, 0.4.0a6, 0.4.0a7, 0.4.0a8, 0.4.0a13, 0.4.0a14, 0.5.0a0, 0.5.0a1, 0.5.0a2, 0.9.0a0, 0.9.0a1, 0.10.0a0, 0.11.0b0, 0.11.0b1, 0.12.0b0, 0.13.0, 0.13.1rc1, 0.13.1, 0.14.0rc1, 0.14.0, 0.15.0, 1.0.0rc1, 1.0.0rc2, 1.0.0, 1.0.1rc1, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.1.0, 1.1.3, 1.2.0, 1.2.1, 1.3.0, 1.3.5, 1.4.0, 1.6.0, 1.6.3, 1.7.0, 1.7.3, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.8.6, 1.9.0rc1, 1.9.0rc2, 1.9.0rc3, 1.9.0, 1.9.1, 1.10.0rc1, 1.10.0rc2, 1.10.0, 1.10.1rc1, 1.10.1rc2, 1.10.1, 1.11.0rc1, 1.11.0rc2, 1.11.0, 1.11.1rc1, 1.11.1, 1.12.0rc1, 1.12.0, 1.12.1, 1.13.0rc1, 1.13.0rc2, 1.13.0rc3, 1.13.0, 1.14.0rc1, 1.14.0rc2, 1.14.0, 1.14.1, 1.14.2rc1, 1.14.2, 1.15.0rc1, 1.15.0, 1.16.0rc1, 1.16.0, 1.16.1, 1.17.0, 1.17.1, 1.18.0, 1.19.0, 1.20.0rc1, 1.20.0rc2, 1.20.0rc3, 1.20.0, 1.20.1, 1.21.0rc1, 1.21.1rc1, 1.21.1, 1.22.0rc1, 1.22.0, 1.22.1, 1.23.0rc1, 1.23.0, 1.23.1, 1.24.0rc1, 1.24.0, 1.24.1, 1.24.3, 1.25.0rc1, 1.25.0, 1.26.0rc1, 1.26.0, 1.27.0rc1, 1.27.0rc2, 1.27.1, 1.27.2, 1.28.0.dev0, 1.28.0rc1, 1.28.0rc2, 1.28.0rc3, 1.28.1, 1.29.0, 1.30.0rc1, 1.30.0, 1.31.0rc1, 1.31.0rc2, 1.31.0, 1.32.0rc1, 1.32.0, 1.33.0rc1, 1.33.0rc2, 1.33.1, 1.33.2, 1.34.0rc1, 1.34.0, 1.34.1, 1.35.0rc1, 1.35.0, 1.36.0rc1, 1.36.0, 1.36.1, 1.37.0rc1, 1.37.0, 1.37.1, 1.38.0rc1, 1.38.0, 1.38.1, 1.39.0rc1, 1.39.0, 1.40.0rc1, 1.40.0, 1.41.0rc2, 1.41.0, 1.41.1, 1.42.0rc1, 1.42.0, 1.43.0rc1, 1.43.0, 1.44.0rc1, 1.44.0rc2, 1.44.0, 1.45.0rc1, 1.45.0)
ERROR: No matching distribution found for grpcio<2.0,>=1.45.0

Bump to >=1.45.0 happened four days ago Irony is 1.45.0 is listed as available 😄

Is there any declerative approach to flyte stuff like gitopsy ?