Hi all, just wanted to cross-post this PSA to help us improve flytekit error messages: https://flyte-org.slack.com/archives/C04NJPLRWUX/p1678467143394579

In tutorials, dataclasses are used as workflow arguments (e.g. Hyperparameters) - how do you enter this when you use

pyflyte run

for local testing?

Hi all, has anyone else run into issue when trying to render workflows that go past a depth of 5 in the UI? The UI can’t seem to render anything past the 5th nested node. Attached is a screenshot of what I’m seeing and the code for how to reproduce it. I would expect to see

wf_6

and

wf_7

also rendered in the UI bug I don’t see in the “Nodes” tab nor the “Timeline” tab. The “Graph” tab only shows a single node which is another problem but I believe people are aware and working on that issue.

@dynamic
def wf_1(n: int) -> int:
    return wf_2(n=n+1)

@dynamic
def wf_2(n: int) -> int:
    return wf_3(n=n+1)

@dynamic
def wf_3(n: int) -> int:
    return wf_4(n=n+1)

@dynamic
def wf_4(n: int) -> int:
    return wf_5(n=n+1)

@dynamic
def wf_5(n: int) -> int:
    return wf_6(n=n+1)

@dynamic
def wf_6(n: int) -> int:
    return wf_7(n=n+1)

@dynamic
def wf_7(n: int) -> int:
    return n+1

@workflow
def run_wf(n: int) -> int:
    return wf_1(n=n)

Hi there, how do I read a csv file from S3 bucket using StructuredDataset and convert to pandas DataFrame?

Hello, I need to create secrets and access them in flyte. I am following Creating Secrets with a Secrets Manager https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/use_secrets.html My question is how to create secrets for different environments? Current we have development and beta, and soon will have production envs. Thanks!

Hey all, I’m seeing some weird behavior with some flyte tasks where they seem to be stuck in a “Running” state. The “Running” status has the additional information “[ContainersNotReady|ContainerCreating]: containers with unready status…“. From looking at the k8s pod for the task, the task seems to have completed and the execution shows as succeeded in the flyte ui. Has anyone seen something like this?

I’m also seeing several workflow executions abort with the error

Workflow execution not found in flyteadmin.

which I don’t fully understand.

Hi everyone.. is there a way to get the full stacktrace of an error? I’m using pyflyte run locally and just getting this as the error:`Failed with Unknown Exception <class 'elasticsearch.exceptions.SerializationError'> Reason: ({'query': {'term': {'entity_id': Promise(node:.entity_id)}}}, TypeError("Unable to serialize Promise(node:.entity_id) (type: <class 'flytekit.core.promise.Promise'>)"))`

Hi Flyters, A n00b question: We have written 10+ Spark applications written in Scala and currently deploying them with some other orchastration tool. We want to use Flyte for orchastration purpose. Some people told us that to use Flyte we have to REWRITE all these 10+ applications using PySpark. Is it really true that Flyte does not support Spark applications written in Scala?

Hi, I an trying to do scheduling workflows by following this https://docs.flyte.org/projects/cookbook/en/latest/auto/core/scheduled_workflows/lp_schedules.html From ,

kubectl get po -n flyte

shall show scheduler pod, but for me I cannot find such pods. So I get stacked now. Could you give me some advice?

$ kubectl get po -n flyte 
NAME                                                  READY   STATUS    RESTARTS      AGE
flyte-sandbox-proxy-d95874857-m8hlt                   1/1     Running   0             46m
flyte-sandbox-kubernetes-dashboard-6757db879c-867nz   1/1     Running   0             46m
flyte-sandbox-docker-registry-78686dccc-vd666         1/1     Running   0             46m
flyte-sandbox-postgresql-0                            1/1     Running   0             46m
flyte-sandbox-c744b94c5-5mp68                         1/1     Running   1 (45m ago)   46m
flyte-sandbox-minio-645c8ddf7c-4vwxr                  1/1     Running   0             46m

Excuse me, I have another question. I wanna create my own docker image. Ati first, I tried install dependency by requirements.text and pushed GCR but had some issue. Then I found the Dockerfile but it does not seem to install xgboost as in tutorial. How was the Docker image in the tutorial created?https://github.com/flyteorg/flyte/blob/master/docker/sandbox/Dockerfile

set the channel topic: Welcome! 👋 - Please follow up on any questions in the appropriate 🧵 - Please don't tag people in any case — we'll get back to you ASAP. - If you encounter any issues, please file them at https://github.com/flyteorg/flyte/issues.

Hi, 🙂 I am trying to configure a private container registry. What worked for me was to add the imagePullSecret to the service account as described here: https://docs.flyte.org/projects/cookbook/en/latest/auto/core/containerization/private_images.html It works when I add the imagePullSecret to the project-specific service account. However, I want to use a global (flyte-wide) service account instead so that I do not have to patch the service account on the cluster for every new Flyte project. I tried using the --k8sServiceAccount parameter with flytectl register. However, it always preprends the project-specific service account. E.g. if I set '--k8sServiceAccount flyte-backend-flyte-binary', then Flyte reports ''[1/1] currentAttempt done. Last Error: USER::pods "apqwf5zfxrj8sbpnlhtv-n0-0" is forbidden: error looking up service account flyte-poc-development/flyte-backend-flyte-binary: serviceaccount "flyte-backend-flyte-binary" not found" i.e., Flyte preprends the project-specific service account to my custom servicer account and, consequently, cannot find it. Is there a way to get this done with --k8sServiceAccount? Or will I have to use custom pod templates?

what would be the best way to call 2 tasks depending how another one ends? Currently I’m using conditional but then can trigger just 1 task

Is there a way to access a property of the workflow arguments in the workflow code? When I do something like this, it throws

TypeError: 'Promise' object is not subscriptable

since flyte creates promises during packaging/deployment:

@workflow
def wf(wf_args) -> int:
    print(wf_args["x"])
    ...

We’re seeing an error where each task in a map task get’s two lines in Flyteconsole and the numbering of the task does not match up. E.g in the screenshot in the 🧵 only one task is running, but two are shown. Clicking on the second link will lead to the second screenshot. In case there would be more tasks, the second link would be labeled the same but lead to the details of the second task. We believe that this might be because we’re showing two links per task (Stackdriver + custom datadog). Is this known or fixed in a newer version? Flyteconsole version is 1.4.1

I’m trying to run multiple flyte deployments in one k8s cluster. Besides having dedicated resources, such as db, s3… Do I need to set anything else other than configmap.core.propeller.limit-namespace and configmap.core.propeller.leader-election.lock-config-map.namespace?

I've been thinking about more monitoring capabilities from Jupyter notebooks. Has anyone thought about using ipywidgets to mimic some of the functionality in FlyteRemote/FlyteConsole?

If I understand correctly,

@dynamic

is a workflow whose DAG is created during runtime (unlike

@workflow

whose DAG is created during compile time), which makes it more flexible. So my question is, what are the downsides of

@dynamic

compared to a static

@workflow

? Why aren’t all flyte workflows

@dynamic

(kind of like how all PyTorch computational graphs are by default dynamic)? Are there meaningful performance penalties to using

@dynamic

?

hey everyone! just trying out the Flyte tutorial for running locally, and something seems off with the Flytekit biz:

$ pyflyte run --remote example.py training_workflow \
                                        --hyperparameters '{"C": 0.1}'
{"asctime": "2023-03-13 16:41:33,286", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
Go to <http://localhost:30080/console/projects/flytesnacks/domains/development/executions/f3f06e76414d540c0994> to see execution in the console.

On the webiste:

[1/1] currentAttempt done. Last Error: USER::containers with unready status: [f3f06e76414d540c0994-n0-0]|Back-off pulling image "<http://cr.flyte.org/flyteorg/flytekit:py3.9-1.4.2|cr.flyte.org/flyteorg/flytekit:py3.9-1.4.2>"

Anyone seen this before?

Hi Flyte community,

pyflyte register

was failing because a module's path was not included in syspath. To workaround this failure and allow the registration command to complete, I added the path to

PYTHONPATH

. However, when I try to run the workflow that's successfully registered,

pyflyte-execute

fails in the pod that Flyte launches because that same module whose path I had included in

PYTHONPATH

was not packaged into the tar file that contains the workflow logic. Does

pyflyte register

support a workaround that allows me to pass in specific module paths that aren't included in the default registration logic (and package those module paths)? I would prefer not to change my project layout if possible. Project layout: root |__ module_with_workflow_and_tasks |__ build |__ generated |__source |__.. |__ _module_not_loaded_ Update: I've temporarily worked around the issue by adding init.py to every subdirectory (build, generated, source) and adding the path to module_not_loaded as an additional argument to

pyflyte register

flyte register only serialize the flyte code. you need to build a docker image with your code (and libraries) to run the workflow

I'm getting an error from flytepropeller claiming I have a wrong image:

{
  "json": {
    "exec_id": "a98zvdbl4hzdwn95f7xv",
    "ns": "development",
    "res_ver": "381075629",
    "routine": "worker29",
    "wf": "flyteexamples:development:container_task_demo.workflows.mmseqs_demo"
  },
  "level": "error",
  "msg": "Error when trying to reconcile workflow. Error [failed at Node[n1]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [] failed to create resource, caused by: admission webhook \"mutate.kyverno.svc-fail\" denied the request: failed to add image information to the policy rule context: invalid image {{.image.container_task_demo_default.fqn}}:{{.image.container_task_demo_default.version}}]. Error Type[*errors.NodeErrorWithCause]",
  "ts": "2023-03-14T09:47:08Z"
}

I registered with the image, here are the logs:

Running pyflyte register from [...] with images ImageConfig(default_image=Image(name='default', fqn='<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>', tag='py3.9-1.4.1'), images=[Image(name='default', fqn='<http://cr.flyte.org/flyteorg/flytekit|cr.flyte.org/flyteorg/flytekit>', tag='py3.9-1.4.1'), Image(name='container_task_demo_default', fqn='location-docker.pkg.dev/[path to image]', tag='6747b4e2db0d5145e23a00044bdd547d4f276c01')]) and image destination folder /root on 1 package(s) 
Registering against flyte.[domain]
Detected Root [...], using this to create deployable package...
Loading packages ['container_task_demo'] under source root [...]
Successfully serialized 1 flyte objects
[✔] Registration container_task_demo.workflows.[task_name] type TASK successful with version 6747b4e2db0d5145e23a00044bdd547d4f276c01
Successfully registered 1 entities
flyte_task

The UI reports the correct image and the task before it runs with that image without issue. I'm not sure what could be causing this.

Hello. I got error in flyteadmin anybody help me with issue?

panic: [AUTH_CONTEXT_SETUP_FAILED] Error creating oidc provider w/ issuer [], caused by: Get "/.well-known/openid-configuration": unsupported protocol scheme ""

I have hidden my values for security but I am using azure ad for authentication is there something wrong with my configuration?

server.yaml: |

server:

grpcPort: 8089

httpPort: 8088

security:

useAuth: true

allowCors: true

allowedHeaders:

- Content-Type

- flyte-authorization

allowedOrigins:

- '*'

secure: false

auth:

authorizedUris:

- <https://test.com>

- <http://flyteadmin:80>

- <http://flyteadmin.flyte.svc.cluster.local:80>

userAuth:

openId:

baseUrl: <https://login.microsoftonline.com/{tenent}/v2.0>

clientId: app-id

clientSecretLocation: /etc/secrets/client_secret

scopes:

- profile

- openid

- offline_access

appAuth:

thirdPartyConfig:

flyteClient:

clientId: app-id

redirectUri: https://{url}/callback

scopes:

- offline

- all

flyteadmin:

eventVersion: 2

metadataStoragePrefix:

- metadata

- admin

metricsScope: 'flyte:'

profilerPort: 10254

testing:

host: <http://flyteadmin:80>

Is there an easy way to specify a that a task should recieve the serialized version of an output from a previous task? Like say I have a train task that outputs a

nn.Module

with the pytorch plugin, but I want a downstream task to receive the filepath that the

nn.Module

is saved to. Is there a way to do that?

Whats the best way to set the default service account for Flyte tasks in the flyte-binary helm chart?

Cross-posting this from the #contribute channel: https://flyte-org.slack.com/archives/C04NJPLRWUX/p1678802736953579

Excuse me, could you tell me the default resource size? When I see the document, those are None but if I did not set any for them, then what size is allocated for each task? https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.Resources.html

Hey! We have this workflow that even though a task failed, a subworkflow node looks like it is still running in flyteconsole 🧵 Anyone seen this before?

Hello! I'm having issues with the SQLAlchemyTask. It appears that for sqlalchemy >= 2.0 the task is broken because that library fails to execute string querys and only works with sqlalchemy.text(querys). On https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-sqlalchemy/flytekitplugins/sqlalchemy/task.py#L135

df = pd.read_sql_query(interpolated_query, connection)

becomes

df = pd.read_sql_query(sqlalchemy.text(interpolated_query), connection)

I can verify locally sqlalchemy 1.4.x works with either a text object or string, but 2.0.x only works with text. I can submit that as an issue against flytekit, but I'm wondering if a) if there is a way to verify what version of the module the task is using and b) if there is a way to force using a specific sqlalchemy version.