hey all - I'm having an issue launching a task execution on my bare metal k8s cluster - when flyteadmin attempts to create a new FlyteWorkflow resource, it fails because the default namespace ({{project}}-{{domain}}) doesn't exist. Have I misconfigured something? I'm wondering how it works in the sandbox

how do you handle a task failure in a workflow? can you just use try-catch?

Does the sandbox cluster support running in a podman rootless cluster?

Hi all. I originally had flyte working, but had to delete the cluster and reinstall. I passed the deployment stage and set it using this console (https://birdejo.live). Unfortunately the console doesnt seem to work. Any ideas on how to diagnose what the issue is?

@Niels Bantilan / @Eduardo Apolinario (eapolinario) can you point @Khuyen Tran to an example of flytekit remote

So I created a sandbox using:

flytectl sandbox start --source .

. After doing that, I can access to Flyte UI. However, after stopping the container, I tried to run

flytectl sandbox start --source .

to resume the existing sandbox cluster, but I can't access to Flyte UI. Here is the specific outputs I got:

➜ flytectl sandbox start                                       
🧑‍🏭 Bootstrapping a brand new flyte cluster... 🔨 🔧
delete existing sandbox cluster [y/n]: n
Existing details of your sandbox👨‍💻 Flyte is ready! Flyte UI is available at <http://localhost:30081/console> 🚀 🚀 🎉 
Add KUBECONFIG and FLYTECTL_CONFIG to your environment variable 
export KUBECONFIG=$KUBECONFIG:/home/khuyen/.kube/config:/home/khuyen/.flyte/k3s/k3s.yaml 
export FLYTECTL_CONFIG=/home/khuyen/.flyte/config-sandbox.yaml

I wonder what is the best way to resume the existing cluster?

I would like to run sandbox locally on an offline machine. I'm encountering the issue above preventing me from restarting the sandbox. Ultimately I want to be able to run the sandbox offline. Is this supported out of the box or will I need to modify?

I've deployed flyte using kubeadm along with spark-k8s-operator which run successful this example, did try run the example from flytesnacks and got some issues (logs in 🧵). Someone know about

Non-spark-on-k8s command provided

?

I’m currently testing out flyte and still have a few issues. In the sandbox UI I can not see any workflows. When I try to register my own workflow with

python deploy.py

I get the following error message:

Serializing Flyte workflows
Usage: pyflyte [OPTIONS] COMMAND [ARGS]...
Try 'pyflyte --help' for help.

Error: Invalid value for '-k' / '--pkgs': Illegal package value booster-team for parameter: <Option pkgs>. Expected for the form [a.b.c]
Registering Flyte workflows
Error: open flyte-package.tgz: no such file or directory

What is the best way to diagnose what’s going wrong?

Hi, I'm trying to understand concurrency using Flyte. I've implemented this example, • https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/subworkflows.html#example-2 Under which or what circumstances does concurrency occur?

Heyy! I am learning and getting hands-on in flyte. following https://docs.flyte.org/en/latest/getting_started/deploy.html running $flytectl sandbox start --source . Don't know exactly why the services of the sandbox here are remaining in pending state

Hey, everyone! I need some help understanding what I'm doing wrong. I'm trying out Flyte using the sandbox and flytectl. I'm following the workflow: build the container, serialize tasks, register the tasks. I'm avoiding fast serialization because I'm using a custom Dockerfile that sets the user as non-root, and apparently that breaks the fast registration system (by the way, is this expected? Will this also happen in a remote cluster?). Now, I just created a new task to test Google Cloud integration, called

test_google_cloud

, but the system won't find it: if I try to run it, I'm met with the error

AttributeError: module 'src.flyte.tasks' has no attribute 'test_google_cloud'

. I ran

docker build

again, serialized the tasks again, registered them again, and still nothing. The task is registered, since I can see it on the UI, but the worker can't find it. What is happening?

Hello everyone! I'm testing out flyte and running it locally in sandbox mode. I have created a simple task that loads a 2 GB file in memory and prints some useful information about the file. At this point, I'm just loading the file in memory and not doing anything with it. As I try to run the task from the console, my pod gets killed with OOMKilled. How can I increase the resources available for pod created by Flyte? Thank you!

Hi, I’m new to flyte but I found this project interesting. i was building a training pipeline that uses multiple k8s cluster (one is public aws eks cluster and another is private on-premise k8s cluster). The training dataset (petastorm format) needs to be generated by aws eks and stores in s3 path and copied over to local premise. And then local premise cluster will kick off distributed horovod training consuming the generated dataset (if the dataset already exists (already synced across cluster), no copy is needed). In order to achieve that, what’s the best practice in Flyte ? How many workflows is needed ?

Hey! I'm trying out flyte but I'm running into some issues with authentication. I've set up a new GKE cluster following https://docs.flyte.org/en/latest/deployment/gcp/manual.html#deployment-gcp-manual and then added Oauth2 authentication using https://docs.flyte.org/en/latest/deployment/cluster_config/auth_setup.html. I can see that it is working on the browser

https://${MY_DOMAIN}/console

however I can't seem to get it to work on the command line, when running a command I get:

Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for ingress.local, not ${MY_DOMAIN}"

My config looks something like (i've tried a few variations):

admin:
  endpoint: dns:///${MY_DOMAIN}
  authType: Pkce
  insecure: false
  clientId: ${MY_ID}
logger:
  show-source: true
  level: 0

Any hints on how to get around this?

Hello,

Hello! Does Min.io deployment inside the single container Flyte sandbox need additional setup to get the web interface working? I try to access it through localhost:30084 but it just redirects me to localhost:9001 and shows "can't be reached" error.

Hello, everyone! I have some questions regarding launch plans and scheduling. 1. I'm not sure how I'm supposed to create new launch plans for a project. Do they have to be defined in the same file as the workflows? Can I create a separate file, and Flyte will find/run them? 2. To set a cron schedule to a launch plan, do I necessarily need to define an input on the workflow that will be solely used as the kickoff time? Can't I just define a CronSchedule of monthy execution (for instance), so that the scheduler will run the launch plan a month from now? 3. Is it possible to define scheduled launch plans as YAML files, like it's possible for executions? If so, what fields do I need to define? I feel like saving the launch plans as YAML files will result in a more organized code.

Hi All - I've been doing a little testing with a sandbox install, and would like to try installing something closer to 'production', but on an on-prem k8s cluster. I'm a little confused as to the recommended approach here - would I start with the sandbox helm charts, and modify those for my environment? (The README in flyte/charts/flyte seems to indicate that these are sandbox deployments) Thanks!

Hi, everyone! Another question: is it possible to start the sandbox without using flytectl? A member of my team is using Windows and wants to try Flyte. I was trying to accomplish this by starting the sandbox with Docker Compose, and having another container to communicate with it. I was able to run simple commands such as

flytectl create

and

flytectl get

from the container, but I can't build images, and I think the problem is on the sandbox side, since I can't do that even on my host machine. I get the error:

OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "docker": executable file not found in $PATH: unknown

This is the configuration for the sandbox. What am I missing?

services:
  flyte-sandbox:
    image: <http://cr.flyte.org/flyteorg/flyte-sandbox:latest|cr.flyte.org/flyteorg/flyte-sandbox:latest>
    container_name: flyte-sandbox
    privileged: true
    volumes:
      - ..:/usr/src           # The docker-compose file is inside a sandbox/ dir
    ports:
      - "30081:30081"         # Console at <http://localhost:30081/console>
      - "30084:30084"
      - "30088:30088"

Hi guys, I am trying to deploy Flyte to our Kubernetes stack but it seems like our FlyteScheduler deployment does not seem to want to start. I keep noticing this error in the Flyte Scheduler logs.

Error: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: oauth2: cannot fetch token: 401 Unauthorized

Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}

This is the log from FlyteAdmin logs:

{
  "json": {
    "src": "token.go:37"
  },
  "level": "info",
  "msg": "Error occurred in NewAccessRequest: invalid_client",
  "ts": "2022-03-30T19:38:33Z"
}

My Flyte Scheduler configmap is listed below. Am I perhaps missing a step?

data:
  admin.yaml: |
    admin:
      clientId: ExternalODICclientID
      clientSecretLocation: /etc/secrets/flyte-secret-auth
      endpoint: flyteadmin.flyte.svc:81
      insecure: true
    event:
      capacity: 1000
      rate: 500
      type: admin
  db.yaml: |
    database:
      dbname: postgres
      host: 'postgres-postgresql.flyte.svc'
      passwordPath: /etc/db/pass.txt
      port: 5432
      username: xxxx
  logger.yaml: |
    logger:
      level: 4
      show-source: true
  server.yaml: |
    scheduler:
      metricsScope: 'flyte:'
      profilerPort: 10254

This is our Flyte Admin configmap:

auth:
  appAuth:
    openId:
      baseUrl: <https://ExternalODICbaseurl.com>
      clientId: OurODICClientID
      scopes:
      - profile
      - openid
      - email
    thirdPartyConfig:
      flyteClient:
        clientId: ExternalODICclientID
        redirectUri: <http://localhost:53593/callback>
        scopes:
        - offline
        - all
  authorizedUris:
  - <https://our.domain.com>
  userAuth:
    openId:
      baseUrl: <https://ExternalODICbaseurl.com>
      clientId: ExternalODICclientID
      scopes:
      - profile
      - openid
      - email
  authorizedUris:
  - <https://our.domain.com>

Hey, everyone! I was wondering: what is the best approach to deal with persistent files on Flyte tasks/workflows? For instance, what if I train a model and I want to save this model on my Flyte server to be used every time a certain task is called, is it possible? In this example on the User Guide, the data returned is a saved file, but will it be persistent? And can it have a human-friendly name, or will it be random letters and numbers, like how it is with DataFrames outputs?

Hi, everyone. I have a naive question: How do I load a file from a GCS bucket inside a spark_task? Is there some idiomatic ways to do it in Flyte? (Normally I would add some maven coordinates to my spark config, but I noticed that this does not work for Flyte in a cluster where spark images has to be pre-built.)

HI all! After playing around with the sandbox I'm trying to set up flyte for my team so they can try it out as well. We're on EKS so I followed the AWS guide and got everything working without auth. Then I followed the auth setup guide and got the console behind Google IdP OAuth (still using self-signed certs at this stage). However, I can't seem to get

flytectl

to work now - I get redirected to

<http://localhost:53593/callback>

with the error message:

Flyte Authentication
Couldn't get access token due to error: Post "<https://flyte>.<mydomain.com>:443/oauth2/token": x509: "flyte.<mydomain.com>" certificate is not standards compliant

I also set

insecureSkipVerify: true

in

~/.flyte/config.yaml

but to no avail.

Hello! I got a sandbox up and running locally and have been trying out all the examples in the cookbook. I've gone through the Spark plugin setup here and have registered the latest version of the plugin example (0.3.66) but I've run into some issues with executing them as-is and the spark context can't be initialized. I re-ran through all the steps on the plugin install page again, but I'm wondering if something needs to be changed in the configuration, restarted, or if the issue is in the container from the registered workflows. Any guidance on how to best debug would be very much appreciated!

Traceback (most recent call last):

      File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 165, in system_entry_point
        return wrapped(*args, **kwargs)
      File "/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py", line 464, in dispatch_execute
        new_user_params = self.pre_execute(ctx.user_space_params)
      File "/opt/venv/lib/python3.8/site-packages/flytekitplugins/spark/task.py", line 123, in pre_execute
        self.sess = sess_builder.getOrCreate()
      File "/opt/venv/lib/python3.8/site-packages/pyspark/sql/session.py", line 228, in getOrCreate
        sc = SparkContext.getOrCreate(sparkConf)
      File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 392, in getOrCreate
        SparkContext(conf=conf or SparkConf())
      File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 146, in __init__
        self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
      File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 209, in _do_init
        self._jsc = jsc or self._initialize_context(self._conf._jconf)
      File "/opt/venv/lib/python3.8/site-packages/pyspark/context.py", line 329, in _initialize_context
        return self._jvm.JavaSparkContext(jconf)
      File "/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py", line 1585, in __call__
        return_value = get_return_value(
      File "/opt/venv/lib/python3.8/site-packages/py4j/protocol.py", line 334, in get_return_value
        raise Py4JError(

Message:

    An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext

SYSTEM ERROR! Contact platform administrators.

The full log from the pod:

++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sed 's/[^=]*=\(.*\)/\1/g'
+ sort -t_ -k4 -n
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ case "$1" in
+ echo 'Non-spark-on-k8s command provided, proceeding in pass-through mode...'
+ CMD=("$@")
+ exec /usr/bin/tini -s -- pyflyte-execute --inputs <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-nw0xm6ntwz/n0/data/inputs.pb> --output-prefix <s3://my-s3-bucket/metadata/propeller/flytesnacks-development-nw0xm6ntwz/n0/data/1> --raw-output-data-prefix <s3://my-s3-bucket/kb/nw0xm6ntwz-n0-1> --checkpoint-path <s3://my-s3-bucket/kb/nw0xm6ntwz-n0-1/_flytecheckpoints> --prev-checkpoint <s3://my-s3-bucket/vc/nw0xm6ntwz-n0-0/_flytecheckpoints> --resolver flytekit.core.python_auto_container.default_task_resolver -- task-module k8s_spark.pyspark_pi task-name hello_spark
Non-spark-on-k8s command provided, proceeding in pass-through mode...
22/04/14 15:19:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/py4j/clientserver.py", line 480, in send_command
    raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
  File "/opt/venv/lib/python3.8/site-packages/py4j/clientserver.py", line 503, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending or receiving
{"asctime": "2022-04-14 15:20:06,729", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! Begin System Error Captured by Flyte !!"}
{"asctime": "2022-04-14 15:20:06,729", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "Traceback (most recent call last):\n\n      File \"/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py\", line 165, in system_entry_point\n        return wrapped(*args, **kwargs)\n      File \"/opt/venv/lib/python3.8/site-packages/flytekit/core/base_task.py\", line 464, in dispatch_execute\n        new_user_params = self.pre_execute(ctx.user_space_params)\n      File \"/opt/venv/lib/python3.8/site-packages/flytekitplugins/spark/task.py\", line 123, in pre_execute\n        self.sess = sess_builder.getOrCreate()\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/sql/session.py\", line 228, in getOrCreate\n        sc = SparkContext.getOrCreate(sparkConf)\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 392, in getOrCreate\n        SparkContext(conf=conf or SparkConf())\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 146, in __init__\n        self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 209, in _do_init\n        self._jsc = jsc or self._initialize_context(self._conf._jconf)\n      File \"/opt/venv/lib/python3.8/site-packages/pyspark/context.py\", line 329, in _initialize_context\n        return self._jvm.JavaSparkContext(jconf)\n      File \"/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py\", line 1585, in __call__\n        return_value = get_return_value(\n      File \"/opt/venv/lib/python3.8/site-packages/py4j/protocol.py\", line 334, in get_return_value\n        raise Py4JError(\n\nMessage:\n\n    An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext\n\nSYSTEM ERROR! Contact platform administrators."}
{"asctime": "2022-04-14 15:20:06,730", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! End Error Captured by Flyte !!"}

@jeev when you get a chance - mind giving the new getting started guide a whirl? would like to see what you think ofit

Hi! I'm porting some stuff to Flyte to see how it compares to our current solution, but I get stuck at a very simple thing. I need to do datetime arithmetic in numpy, so how to I pass datetype parameters to a workflow? np.datetime64 is not supported and trying to parse a string errors. Thanks!

import numpy as np
from flytekit import workflow

@workflow
def wf(date: str = "2022-01-01"):
    date = np.datetime64(date) # this errors

You should be able to use a python datetime as input type: https://docs.flyte.org/projects/cookbook/en/latest/auto/core/type_system/flyte_python_types.html Also the actual processing needs to be done inside a task. Workflows are basically a DSL to compose tasks. I.e. something like this (not tested):

from datetime import datetime
@task
def t(date: datetime = datetime(2022,1,1)):
    date = np.datetime64(date)
    ...

@workflows
def wf(date: datetime):
    t(date=date)

Searching for

flyte run workflow

on Google, the first result is a 404 page on the documentation site.

Hello! I'm starting with flyte, I'm trying to understand how flyte isolates resources, especially how to define the domains and their resourceQuotas, can I change it on the fly? For now, I believe this is defined in the helm values and cannot be changed without changing the helm, is this true? or is there another way?