Hey guys, I'm trying to pass environment variables to my pods. I'm using flyte-binary deployment. For some reason, the task pods are getting the environment variables with their name lowercased. Other environment values passed by Flyte to the pod are not lowercased, only the added values via the k8s plugin. This is the relevant snippet from the `values.yaml`:

...
configuration
  ...
  inline:
    plugins:
      k8s:
        inject-finalizer: false
        default-env-vars:
          FLYTE_AWS_ACCESS_KEY_ID: "foo"
          FLYTE_AWS_SECRET_ACCESS_KEY: "secret-bar"
          FLYTE_AWS_ENDPOINT: "custom-endpoint"

I logged into the binary pod and located the inline config:

/etc/flyte/config.d/100-inline-config.yaml

. I can clearly see that the

default-env-vars

is not lowercased. I tried a little to locate the relevant Go code but didn't find anything that lower-case the env var names. Thanks for your help!

Hi community, I'm getting an

UNKNOWN

status on every workflow that I submit to Flyte and it just stays in that state (it never evolved to a

RUNNING

state). Some background of the Flyte installation: I have deployed Flyte on a local K8's before deploying it on our real K8's environment (sort of a POC). I have recently installed the MPI-Operator in order to be able to parallelize a ML workflow. Since I couldn't make an update of the Helm Chart because it was throwing the following error

Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: Secret "kubernetes-dashboard-csrf" in namespace "flyte" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "<http://meta.helm.sh/release-name|meta.helm.sh/release-name>" must equal "flyte": current value is "flyte-deps"

I ended up modifying the flyte-core values file, adding to the

ConfigMap

property the

enabled_plugins

property in accordance to what the documentation says. Question: What could be happening and how could I check what's going on under the hood? BTW, from time to time.. it's not weird to get a

503

error when navigating the console. Any help is greatly appreciated, thx!

How does development with Flyte compare to Dagster?

Hey everyone, I'm trying to set up auth with the flyte binary and while I've had some success (with help from you guys 🙂), I'm still running into a few issues. In particular, I'm using Azure as the OIDC provider and have everything working in the below snippet. This is one Azure application and the

oidc

section here allows me to access the UI via an ingress (defined via the chart) 🦜

auth:
    enabled: true
    oidc:
      baseUrl: <https://signin.hosted.unionai.cloud/oauth2/default>
      clientId: <IDP_CLIENT_ID>
      clientSecret: <IDP_CLIENT_SECRET>
    internal:
      clientSecret: <CC_PASSWD>
      clientSecretHash: <HASHED_CC_PASSWD>

I also have another azure application which, in the pre-flyte-binary setup that I am migrating from, was used to populate the values here:

configmap:
  adminServer:
   auth:
     ...
     appAuth:
       ...
       thirdPartyConfig:
         flyteClient:
           clientId: XXX
           redirectUri: <http://localhost:53593/callback>
           scopes:
             - XXX

In the flyte-binary chart, editing these values isn't an option but if you check out the

flyte-backend-flyte-binary-config

you can see some hardcoded values:

thirdPartyConfig:
  flyteClient:
    clientId: flytectl
    redirectUri: <http://localhost:53593/callback>
    scopes:
    - offline
    - all

Is it possible to overwrite these defaults? I tried the following in my flyte-binary chart values:

configuration:
  inline:
    auth:
      appAuth:
        thirdPartyConfig:
          flyteClient:
            clientId: XXX
            redirectUri: <http://localhost:53593/callback>
            scopes:
              - XXX

But I don't think this worked. Sorry for the long post, but any idea how I can override hard-coded

thirdPartyConfig

values in the helm chart? Thanks a lot for all the continued help 🙂

Hi, community, can anyone tell me should I SET KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT env variables ?

flyteadmin# flyte start --config flyte-single-binary-local.yaml

INFO[0000] Using config file: []
INFO[0000] [0] Couldn't find a config file []. Relying on env vars and pflags.
{"json":{},"level":"error","msg":"Failed to create controller manager. error building Kubernetes Clientset: Cannot get InCluster kubeconfig: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined","ts":"2023-07-14T22:56:12+08:00"}
{"json":{},"level":"panic","msg":"Failed to start Propeller, err: error building Kubernetes Clientset: Cannot get InCluster kubeconfig: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined","ts":"2023-07-14T22:56:12+08:00"}
panic: (*logrus.Entry) 0xc0007882a0

goroutine 117 [running]:
<http://github.com/sirupsen/logrus.(*Entry).log(0xc0007881c0|github.com/sirupsen/logrus.(*Entry).log(0xc0007881c0>, 0x0, {0xc0001aa700, 0xd2})
        /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:259 +0x45b
<http://github.com/sirupsen/logrus.(*Entry).Log(0xc0007881c0|github.com/sirupsen/logrus.(*Entry).Log(0xc0007881c0>, 0x0, {0xc00126de68?, 0x1?, 0x1?})
        /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:293 +0x4f
<http://github.com/sirupsen/logrus.(*Entry).Logf(0xc0007881c0|github.com/sirupsen/logrus.(*Entry).Logf(0xc0007881c0>, 0x0, {0x30e2999?, 0x10?}, {0xc000bb8220?, 0x10?, 0xc0000a0800?})
        /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:338 +0x85
<http://github.com/sirupsen/logrus.(*Entry).Panicf(0x3fd08e0|github.com/sirupsen/logrus.(*Entry).Panicf(0x3fd08e0>?, {0x30e2999?, 0x416667?}, {0xc000bb8220?, 0x2a09ea0?, 0x1?})
        /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:376 +0x34
<http://github.com/flyteorg/flytestdlib/logger.Panicf({0x3fd08e0|github.com/flyteorg/flytestdlib/logger.Panicf({0x3fd08e0>?, 0xc0007a8bc0?}, {0x30e2999, 0x22}, {0xc000bb8220, 0x1, 0x1})
        /root/go/pkg/mod/github.com/flyteorg/flytestdlib@v1.0.19/logger/logger.go:188 +0x64
<http://github.com/flyteorg/flyte/cmd/single.glob..func4.2()|github.com/flyteorg/flyte/cmd/single.glob..func4.2()>
        /mnt/c/code/dev/flyte/cmd/single/start.go:185 +0xbe
<http://golang.org/x/sync/errgroup.(*Group).Go.func1()|golang.org/x/sync/errgroup.(*Group).Go.func1()>
        /root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64
created by <http://golang.org/x/sync/errgroup.(*Group).Go|golang.org/x/sync/errgroup.(*Group).Go>
        /root/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa5

I am trying to run a wandb.sweep as shown in this quickstart. The idea is to run the hyperparameter sweep, on a conditional, based on a boolean flag passed to a workflow and save the best model as an artifact. The main workflow is static, which calls a dynamic sweep_setup function. The wandb.agent calls a callback function hyperparameter_sweep which is also dynamic and which calls the train_model dynamic function. Is there an example of this somewhere or any best way to do this? Because, I am facing too many errors to get this to work. The code is a bit long to share, so first wanted to check if someone has done this successfully.

Hi all, I am trying to execute a single task in a workflow (not the entire workflow) with elevated rights in my k8s cluster. For this I created an additional k8s role, service account and binding similar to the spark setup and am now trying to work out how to make Flyte launch the pod with that service account. I tried using the pod plugin and with the following task decorator:

@task(
    requests=Resources(mem="512Mi", cpu="1"),
    limits=Resources(mem="2Gi", cpu="1"),
    task_config=Pod(
        pod_spec=V1PodSpec(
            containers=[V1Container(name="primary")],
            service_account="<sa-name>",
            service_account_name="<sa-name>"),
        primary_container_name="primary",
    ),
)

However, when executing the workflow it seems that Flyte can no longer fetch the serialised Task inputs from the Flyte GCS bucket and fails with:

Error from command '['gsutil', 'cp', '<gs://flyte-store/metadata/propeller/default-development-fddb5e602ce594338828/n1/data/inputs.pb>', '/tmp/flyte-tz9k8etn/sandbox/local_flytekit/inputs.pb']':
...
raise exceptions.CommunicationError(\napitools.base.py.exceptions.CommunicationError: Could not reach metadata service: Forbidden\n

Interestingly when I put

default

as

sa-name

everything works fine and the two k8s service accounts are linked to the same gcp-service-account. I would very much appreciate any pointers towards debugging this or other ways of setting up the Task to be executed with the new service account. Cheers, Seb

Hello! I got a sandbox up and running locally and have been trying out all the examples in the cookbook. I've gone through the Spark plugin setup here and have registered the latest version of the plugin example (0.3.66) but I've run into some issues with executing them as-is and the spark context can't be initialized. I re-ran through all the steps on the plugin install page again, but I'm wondering if something needs to be changed in the configuration, restarted, or if the issue is in the container from the registered workflows. Any guidance on how to best debug would be very much appreciated!

Traceback (most recent call last):

      File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py", line 464, in dispatch_execute
        new_user_params = self.pre_execute(ctx.user_space_params)
      File "/opt/venv/lib/python3.8/site-packages/flytekitplugins/spark/task.py", line 123, in pre_execute
        self.sess = sess_builder.getOrCreate()
      File "/opt/venv/lib/python3.8/site-packages/pyspark/sql/session.py", line 228, in getOrCreate
        sc = SparkContext.getOrCreate(sparkConf)
      File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 392, in getOrCreate
        SparkContext(conf=conf or SparkConf())
      File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 146, in __init__
        self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
      File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 209, in _do_init
        self._jsc = jsc or self._initialize_context(self._conf._jconf)
      File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 329, in _initialize_context
        return self._jvm.JavaSparkContext(jconf)
      File "/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py", line 1585, in __call__
        return_value = get_return_value(
      File "/opt/venv/lib/python3.8/site-packages/py4j/protocol.py", line 334, in get_return_value
        raise Py4JError(

Message:

    An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext

SYSTEM ERROR! Contact platform administrators.

The full log from the pod:

++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sed 's/[^=]*=\(.*\)/\1/g'
+ sort -t_ -k4 -n
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ case "$1" in
+ echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
+ CMD=("$@")
+ exec /usr/bin/tini -s -- pyflyte-execute --inputs <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-nw0xm6ntwz/n0/data/inputs.pb> --output-prefix <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-nw0xm6ntwz/n0/data/1> --raw-output-data-prefix <s3://my-s3-bucket/kb/nw0xm6ntwz-n0-1> --checkpoint-path <s3://my-s3-bucket/kb/nw0xm6ntwz-n0-1/_flytecheckpoints> --prev-checkpoint <s3://my-s3-bucket/vc/nw0xm6ntwz-n0-0/_flytecheckpoints> --resolver flytekit.core.python_auto_container.default_task_resolver -- task-module k8s_spark.pyspark_pi task-name hello_spark
Non-spark-on-k8s command provided, proceeding in pass-through mode...
22/04/14 15:19:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/py4j/clientserver.py", line 480, in send_command
    raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
  File "/opt/venv/lib/python3.8/site-packages/py4j/clientserver.py", line 503, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending or receiving
{"asctime": "2022-04-14 15:20:06,729", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! Begin System Error Captured by Flyte !!"}
{"asctime": "2022-04-14 15:20:06,729", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "Traceback (most recent call last):\n\n      File \"/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py\", line 165, in system_entry_point\n        return wrapped(*args, **kwargs)\n      File \"/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py\", line 464, in dispatch_execute\n        new_user_params = self.pre_execute(ctx.user_space_params)\n      File \"/opt/venv/lib/python3.8/site-packages/flytekitplugins/spark/task.py\", line 123, in pre_execute\n        self.sess = sess_builder.getOrCreate()\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/sql/session.py\", line 228, in getOrCreate\n        sc = SparkContext.getOrCreate(sparkConf)\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 392, in getOrCreate\n        SparkContext(conf=conf or SparkConf())\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 146, in __init__\n        self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 209, in _do_init\n        self._jsc = jsc or self._initialize_context(self._conf._jconf)\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 329, in _initialize_context\n        return self._jvm.JavaSparkContext(jconf)\n      File \"/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py\", line 1585, in __call__\n        return_value = get_return_value(\n      File \"/opt/venv/lib/python3.8/site-packages/py4j/protocol.py\", line 334, in get_return_value\n        raise Py4JError(\n\nMessage:\n\n    An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext\n\nSYSTEM ERROR! Contact platform administrators."}
{"asctime": "2022-04-14 15:20:06,730", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! End Error Captured by Flyte !!"}

Hello, Is Flyte capable of MANUALLY resuming the workflow from the failed task forward? By manually I mean either from the UI or CLI. I see a rerun button on the UI to the right of the tasks. Will clicking the rerun button resume the workflow or just rerun the failed task? What about manually marking a failed task as SUCCEEDED and resume the rest of the tasks forward?

Hey everyone! I'm working to deploy Flyte in our GKE cluster via the standalone cluster documentation. We have requirements around leveraging Minio (already deployed on the cluster). I am able to get Flyte stood up and access it via console, however, when I "pyflyte run --remote" a workflow I get errors regarding connection to Minio based on the conneciton details I passed into the Flyte deployment via the helm chart. Is the pyflyte cli trying to connect directly to Minio and causing this error? I've tried adjusting the config.yaml with a storage config to point to the port-forwarded minio service but it still appears to try to connect to minio via teh config passed to the deployment.

Original exception: HTTPConnectionPool(host='signals-minio', port=9000): Max retries exceeded with url: /flyte/flytesnacks/development/YK2ZGX4HTNQPDYSEHLOIX7K2UM%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio_access_key%2F20230210%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230210T152959Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=6f79fdccd56b3bc8ff016e3db2c94234384c2c14e96aa38bce2ca66679748a9e (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x11d5884c0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))